u/WhatASave83

I manage a bunch of Airflow Instances for my organization, and have been educating people on writing better DAGs which don't over load the DB, while making improvements to bring stability to all the instances.

I have one instance in particular where around 100 DAGs run at the same time, and some of these DAGs run tasks for hours. Is that a good use of Airflow, or should I be breaking these tasks down to finish up and quit faster and break down into batches of tasks?

reddit.com
u/WhatASave83 — 10 days ago