英文:
DAG changes not reflected on Airflow UI
问题
Running MWAA medium. It works fine most of the time.
Sometimes after making changes to a DAG, I don't see the updated version of the DAG on the UI. As a workaround, I change the DAG ID.
I have also seen a DAG disappear from UI despite no changes. Had to change the DAG ID to make it reappear.
Have over 100 DAGs and parse time has been degrading according to cloudwatch (currently at 30 seconds).
Want to know what might be causing this behavior. Do I need to move to large MWAA configuration?
英文:
Running MWAA medium. It works fine most of the time.
Sometimes after making changes to a DAG, I don't see the updated version of the DAG on the UI. As a workaround, I change the DAG ID.
I have also seen a DAG disappear from UI despite no changes. Had to change the DAG ID to make it reappear.
Have over 100 DAGs and parse time has been degrading according to cloudwatch (currently at 30 seconds).
Want to know what might be causing this behaviour. Do I need to move to large MWAA configuration?
答案1
得分: 1
使用动态DAG生成还是单个Python文件定义每个DAG对象？如果在单独的文件中定义DAG，超过100个DAG会需要更多资源和DAG处理器的时间。这应该反映在TotalParseTime指标上。理想情况下，这个指标应该在2-3秒左右。较高的值可能会对DAG处理器和调度器施加压力，并可能导致DAG解析和任务出现问题。推荐的方法是尽量保持总解析时间尽可能低。利用动态DAG、.airflowignore和调整DAG解析相关属性是一些推荐的方法。
您可以查看调度器和DAG处理日志，以获取有关时间持续时间以及是否出现任何错误的更多信息。
英文:
Are you using dynamic DAG generation or a single Python file defining each DAG object? In case of defining DAGs in separate files, having over 100 DAGs would require more resources and time for DAG Processor. This should be reflected on the TotalParseTime metric. Ideally, this metric should be around 2-3 seconds. Higher values can lead to pressure on the DAG Processor and Scheduler, and can cause issues with DAG parsing and tasks. The recommended approach is to keep the total parsing time as low as possible. Utilizing Dynamic DAGs, .airflowignore, and tuning DAG parsing related properties would be some of the recommended approaches.
You can review the Scheduler and DAG Processing logs for more insight into the time duration and if any error is being raised.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论