what's the difference between airflow's 'parallelism' and 'dag_concurrency'

The other answer is only partially correct:

dag_concurrency does not explicitly control tasks per worker. dag_concurrency is the number of tasks running simultaneously per dag_run. So if your DAG has a place where 10 tasks could be running simultaneously but you want to limit the traffic to the workers you would set dag_concurrency lower.

The queues and pools setting also have an effect on the number of tasks per worker.

These setting are very important as you start to build large libraries of simultaneously running DAGs.

parallelism is the maximum number of tasks across all the workers and DAGs.

Tags:

Python

Airflow