Airflow webserver doesnt start except in debug mode

Airflow webserver start only in debug mode: airflow webserver -p 8051 And the traceback I get: [2016-12-07 03:38:48,056] {__init__.py:36} INFO - Using executor CeleryExecutor [2016-12-07...

How to use local docker images with Minikube?

I have several docker images that I want to use with minikube. I don't want to first have to upload and then download the same image instead of just using the local image directly. How do I do...

Airflow: Can't connect to ('0.0.0.0', 8080)

I am on Ubuntu 16.04,I have installed Airflow with pip. Next step airflow initdb [2017-07-29 12:20:23,483] {__init__.py:57} INFO - Using executor SequentialExecutor DB:...

airflow webserver starting - gunicorn workers shutting down

I am running airflow 1.8 on centos7 on docker and my webserver is not getting to the browser. I installed airflow via pip2.7. Flower ui is displaying fine, initdb ran connecting to a postgres...

Trying to run apache airflow on ubuntu server with systemd

I'm trying to run airflow on an ubuntu server with systemd. I have followed quick start guide and the tutorial from the airflow documentation and I have managed to install airflow and successfully...

Airflow hide_paused_dags_by_default on airflow.cfg is not working

I want to hide all my Paused DAGs on UI but the configuration on airflow.cfg does not seem to work. # By default, the webserver shows paused DAGs. Flip this to hide paused # DAGs by...

Apache - Airflow 1.10.1 don't start a job

I have a problem with Airflow, The first job in a DAG always starts and ends successfully but the second job never starts automatically. I try to clear the job in the UI but it doesn't starts, if...

Apache Airflow: After pointing airflow.cfg to postgres, it still tries to run on MySQL

I'm using Apache-airflow2. My dags were running on LocalExecutor up until now smoothly. Now i want to scale it up and use CeleryExecutor (I'm still doing it on my Local Mac) I've configured it to...

Airflow signals SIGTERM to subprocesses unexpectedly

I am using the PythonOperator to call a function that parallelizes data engineering process as an Airflow task. This is done simply by wrapping a simple function with a callable wrapper function...

Specify Beam Version for Dataflow Operator on Cloud Composer

We have written a Beam pipeline for version 2.11 but when we try to run it on Cloud Composer using the DataflowOperator it uses SDK version 2.5. Is there anywhere to specify that 2.11 should be...

Is there a way to pause an airflow DagRun?

Is there a way to pause a specific DagRun within Airflow? I want to be able to have multiple, simultaneous executing runs of a single DAG, and I want to be able to pause those runs individually at...

How to control the parallelism or concurrency of an Airflow installation?

In some of my Apache Airflow installations, DAGs or tasks that are scheduled to run do not run even when the scheduler doesn't appear to be fully loaded. How can I increase the number of DAGs or...

what is the best Airflow architecture for AWS EMR clusters?

I have an AWS EMR cluster with 1 master node, 30 core nodes and some auto-scaled task nodes. now, hundreds of Hive and mysql jobs are running by Oozie on the cluster. I'm going to change some jobs...

airflow sending sigterms to tasks randomly

I was running into an issue with airflow 1.10.1. Some of the tasks in the dags are getting SIGTERM from helpers.py, from what I understood this is to perform shutdown for the workers and terminate...

Airflow sla_miss_callback function not triggering

I have been trying to get a slack message callback to trigger on SLA misses. I've noticed that: SLA misses get registered successfully in the Airflow web UI at slamiss/list/ on_failure_callback...

Airflow DockerOperator: connect sock.connect(self.unix_socket) FileNotFoundError: [Errno 2] No such file or directory

I am trying to get DockerOperator work with Airflow on my Mac. I am running Airflow based on Puckel with small modifications. Dockerfile build as puckel-airflow-with-docker-inside: FROM...

Google Cloud Composer (Apache Airflow) cannot access log files

I'm running a DAG in Google Cloud Composer (hosted Airflow) which runs fine in Airflow locally. All it does is print "Hello World". However, when I run it through Cloud Composer I receive the...

Where to put airflow_local_settings.py in Composer?

In composer (airflow 1.10.10), is it possible to create an airflow_local_settings.py file? And if so where should it be stored? I need this as I need an initContainer for my pod. Add a...

Pod Launching failed: Pod took too long to start, Failed to run KubernetesPodOperator secret

I'm running the quickstart for KubernetesPodOperator secret using the link below : https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator Code used below : from airflow...

Unable to attach or mount volumes: unmounted volumes

i deplyed my application on kubernetes but have been getting this error: **MountVolume.SetUp failed for volume "airflow-volume" : mount failed: mount failed: exit status 32 Mounting command:...

Airflow on_failure_callback

Hello hope you all are doing fine i would like to ask one question recently i have been trying airlfow and playing with it here is the situation everything works fine i have two...

NameError: name '_mysql' is not defined -- On airflow start in MacOSX

There are numbers of articles on the titled question but none of them worked for me. The detailed error is as follows: Traceback (most recent call last): File...

apache airflow 2.0.2 slow web UI

I have installed apache-airflow 2.0 on a new EC2 instance (r5a.xlarge) 4 CPU and 32 RAM and SSD disk. Airflow webserver and scheduler are running in a stable way, BUT the web response is slooooow...

Airflow Celery executor start failing tasks

I need help in order to solve the problem about Celery executor fails. Below my architecture: Airflow 1.10.7 Airflow Scheduler, Webserver and Workers running on Docker over AWS EC2...

Airflow - KubernetesPodOperator - Role binding a service account

I am currently using the KubernetesPodOperator to run a Pod on a Kubernetes cluster. I am getting the below error: kubernetes.client.rest.ApiException: (403) Reason: Forbidden HTTP response...

How to add new user to docker image when running distributed airflow architecture using docker-compose

(THE ORIGINAL QUESTION WAS EDITED TO MAKE IT MORE CLEAR) SOLUTION AT THE END OF THE QUESTION ANOTHER SOLUTION IN THE ANSWER The goal and the setup The main goal is to run container based...

ValueError: unsupported pickle protocol: 5 while running jobs in Airflow

We have installed Airflow 2.1.3 version in Linux server, worker is also available in the same server and while we are trying to run the job it says Error: ValueError: unsupported pickle protocol:...

Airflow CeleryExecutor - 'int' object has no attribute 'startswith' in Celery

Airflow 2.0 is queuing but not launching tasks in my dev environment. DAG and Pool settings are valid, but all tasks in each dag are queued when I trigger them, and are never running. When typing...

Google Cloud Composer v2 health-check seems to be false negative/flaky

We created a Composer v2 environment to migrate from Google Cloud Composer v1. All DAG code was adjusted and we are using the to this date newest available image...

Scaling Airflow with a Celery cluster using Docker swarm

As the title says, i want to setup Airflow that would run on a cluster (1 master, 2 nodes) using Docker swarm. Current setup: Right now i have Airflow setup that uses the CeleryExecutor that is...