Unable to execute a command in Ubuntu container, from Airflow container using DockerOperator

I am very new to Docker + Airflow. Below is what I am trying to accomplish. I have 4 services as shown in the below compose file. 3 are relevant to Airflow and one as a test Ubuntu instance. The...

Airflow2 With AWS RDS Postgres as metastore error

Hello Developers i have been struggling to figure out how i can leverage AWS RDS as airflow Meta Store i am creating the issue after going through several Post and blogs References...

Airflow: trigger Spark in different Docker container

I have both Airflow 2 (the official image) and Apache Spark running in a docker-compose pipeline. I would like to execute a DAG triggering a Spark script by means of the SparkSubmitOperator...

Cannot install additional requirements to apache airflow

I am using the following docker-compose image, I got this image from: https://github.com/apache/airflow/blob/main/docs/apache-airflow/start/docker-compose.yaml version: "3" x-airflow-common:...

How to set connections and variables in airflow using docker-compose file

I'm creating a dev environment to use airflow for testing. I'm using the docker-compose.yaml file available on Airflow website. I would like to know if it is possible to set my connections and...

apache-airflow-providers-google in MWAA

I tried to install apache-airflow-providers-google in MWAA but it seems that Airflow does not pick the library: Broken DAG: [/usr/local/airflow/dags/mi-framework/jobs/google_analytics/main.py]...