Publishing a Customized Docker Image with Apache Hop, Workflows, and Pipelines

Introduction

In my previous blog on Docker container customization
(https://saikatsrecentworks.blogspot.com/2026/03/custom-hop-docker.html), I discussed making the customized environment easily accessible. This post completes that effort by sharing the fully prepared Docker image.

Docker Image Contents

This Docker image is built on Alpine Linux and comes pre-installed with Java and Apache Hop. In addition, it includes all components required for my project: automated information ingestion from Microsoft Office 365 using Microsoft Graph APIs ( https://saikatsrecentworks.blogspot.com/2026/03/apache-hop-office365.html), powered by Python, Selenium, and Apache Hop.

The purpose and implementation details of these components are explained in my earlier articles.

Included components:

  • Python 3
  • pip (Py3-pip)
  • wget
  • dpkg
  • Chromium & Chromium Chromedriver
  • Git
  • Selenium and BeautifulSoup (bs4) for Python

Accessing the Docker Image

The complete image is available via GitHub Container Registry-

https://github.com/users/basusk/packages/container/package/hop_customized_docker

You can pull and run the image using:

        docker pull ghcr.io/basusk/hop_customized_docker:latest

    

After pulling, start the container using the standard docker run command.

If you encounter any issues, feel free to reach out at basusk@comcat.net

Apache Hop Setup

Apache Hop (version 2.13) is installed inside the container at: 

    /opt/hop

Note: This is an older version. You can upgrade it in-place if needed.

All workflows and pipelines are located at: 

    /home/hop/EmailHandling

Hop Workflows and Pipelines

The main (parent) workflow is: 

    office365-workflow-main.hwf 

(https://github.com/basusk/hop/blob/main/office365_workflow_main.hwf)

All related child workflows, pipelines, shell scripts, and Python scripts are available inside docker container as well as in this repository:

👉 https://github.com/basusk/hop

Implementation Overview

As described in my earlier blog
(https://saikatsrecentworks.blogspot.com/2026/03/apache-hop-office365.html), the solution automates Office 365 data ingestion using a combination of Hop workflows, pipelines and supporting scripts.

The implementation is structured in multiple stages.

Some workflows also include server-specific versions, adapted for execution in Linux servers or Docker environments.

Getting Started

To use this setup:

  1. Pull and run the Docker image
  2. Login and execute main workflow inside docker via Hop Command line.
  3. Otherwise, clone the workflows repository locally ( https://github.com/basusk/hop )
  4. Configure required Azure and Office 365 credentials
  5. Substitute the correct values in email-attachment-download.properties file
  6. Execute the main workflows to test the end-to-end job

Final Notes

This setup is designed to reduce manual effort and enable end-to-end automation of Office 365 data ingestion workflows. Feedback and suggestions are always welcome.

Location: Foster City, CA 94404, USA

Comments

Post a Comment

Popular posts from this blog

Customizing an Apache Hop Docker Container

  Cu stomizing an Apache Hop Docker Container for Automated Office 365 Authentication Introduction In my previous blog, Apache Hop implementation for selectively downloading email-attachments from MS Office365 Cloud , I focused on explaining Azure Entra Oath2 implementation using Apache hop and python. We chose to use Apache hop docker container for execution of the automated task via scheduler, following a ‘wake-up, do task, shutdown’ approach, as explained in hop documentation as a short-lived execution pattern. In this article, I describe the steps for customizing docker container, which was necessary to support the execution of Bash and Python scripts integrated within Hop pipelines and workflows Hop Container with External Dependencies  Apache hop container required following external packages - Python3 Py3-pip Wget Dpkg Chromium-chromedriver Chromium Git Selenium (and bs4) for python3 Making Hop Container Ready The Apache Hop Docker image is built on a lightweight Alpin...
Read more

Secured Information Ingestion following Azure Entra OAuth2 from Office365

Image
Information ingestion from MS Office365 via MS Graph APIs using Python Selenium and Apache Hop Introduction Requirement for this task was to download attachments from a shared email address selectively  based on attachment’s file type in an automated and scheduled manner. And then parse and  extract information from those files to stage on a database platform. In this document we will  discuss only the attachments downloading part by designing and implementing via Python Selenium library, Apache Hop Pipelines, Workflows. To accomplish the task, we had taken the approach of low-coding, preferably no-coding. As this  approach offers much more agility and flexibility for this implementation. I will describe all the  details of implementation in the following sections. To accomplish the requirement of automation and scheduling of the tasks, we decided to use  Apache Hop Docker container hosting Hop as a server, which would wake-up to execute the  flow and ...
Read more

Automating Kyligence Index Recommendation Feature After Analysing Pushdown Queries

Image
Extending Kyligence Recommendation System for Ease of Use Introduction In this article we will discuss existing Kyligence feature of AI Augmented Recommendation System by analyzing query patterns and making suggestions for further model tuning by adding/altering indexes inside a model for better cost optimized query execution, reducing latency and hitting appropriate index by it’s intelligent query router. Recommendation system is also configurable for significant, heavy-weight query user and less priority use cases accordingly. After a brief description of Recommendation system, we will explain existing gap for making this completely automated with no human intervention. Finally, we will describe supplementing utility programs to bridge this gap and making Kyligence a step forward towards more user friendliness and complete automated handling of new queries starting from model creation and index tuning both, followed by data loading into models. Kyligence Recommendation System Existin...
Read more