Hello folks !
I am migrating from airflow 2.9.0 to 3.1.8
All dags related changes are done and configuration related also.
So in current airflow prof we have deployed it on EC2 with ECS. So all of containers ( webserver , Postgres’s , redis , scheduler, celery worker) is working fine in M6a.large instance type.
But when we do test deployment with airflow 3.1.8 api server and celery worker is killed by OOM when more then 10 dags are scheduled together and even ideal state api-server is using around 1.8 gb memory. Any one facing same issues ? What is work around for this ? Any suggestions how to scale it ? How all other using which architecture ?
Any suggestions are appreciated! Thanks