Airflow Seminar

Deployment

Requirements

In order to deploy this tutorial the following hardware and software requirements must be met.

Hardware

• Memory: 16 GB RAM
• Processor: Intel(R) Core(TM) i5-6500 or greater / AMD Ryzen 5 3600 or greater
• Storage: 15 GB available space

Software

• Docker version >= 27.4.0
• Docker-compose version >= 2.29.2

Installation

To install the project, run the following steps:

1. Run database migrations and create the first user account
   docker compose up airflow-init

2. Initiate Airflow with:
   docker compose --profile flower up -d

After the compose builds the project, the tutorial's services will be available at:

Service	URL
Airflow Web Interface	http://localhost:8080
MlFlow	http://localhost:5000
Flower	http://localhost:5555
Spark Master	http://localhost:8090
Mongo Express	http://localhost:28081

Setting Up

1. Navigate to Airflow's web interface
2. Log in with username: airflow and password: airflow
3. Navigate into the connections section in admin dropdown.
4. Click the add new record option
5. Set Connection Id to mongo_default
6. Select MongoDB in Connection type dropdown menu
7. Provide Host, Username, Password and Port in accordance with the .env file.
8. Save connection

An example connection configuration is depicted below:

Technologies used

• Docker
• Docker-compose
• Airflow
• Apache Spark
• Flower
• MongoDB
• Mongo Express UI