在Airflow中特定的DAGs收到许多警告。

huangapple go评论62阅读模式
英文:

getting many warnings for particular dags in airflow

问题

我刚刚最近安装了Airflow,每当我执行一个任务时,都会收到关于不同DAG的警告:

[2023-03-01 06:25:35,691] {taskmixin.py:205} WARNING - 依赖<Task(BashOperator): create_entry_group>, delete_entry_group已经在DAG: example_complex中注册
[2023-03-01 06:25:35,691] {taskmixin.py:205} WARNING - 依赖<Task(BashOperator): delete_entry_group>, create_entry_group已经在DAG: example_complex中注册
[2023-03-01 06:25:35,691] {taskmixin.py:205} WARNING - 依赖<Task(BashOperator): create_entry_gcs>, delete_entry已经在DAG: example_complex中注册
[2023-03-01 06:25:35,692] {taskmixin.py:205} WARNING - 依赖<Task(BashOperator): delete_entry>, create_entry_gcs已经在DAG: example_complex中注册
[2023-03-01 06:25:35,692] {taskmixin.py:205} WARNING - 依赖<Task(BashOperator): create_tag>, delete_tag已经在DAG: example_complex中注册
[2023-03-01 06:25:35,692] {taskmixin.py:205} WARNING - 依赖<Task(BashOperator): delete_tag>, create_tag已经在DAG: example_complex中注册
[2023-03-01 06:25:35,759] {taskmixin.py:205} WARNING - 依赖<Task(_PythonDecoratedOperator): prepare_email>, send_email已经在DAG: example_dag_decorator中注册
[2023-03-01 06:25:35,759] {taskmixin.py:205} WARNING - 依赖<Task(EmailOperator): send_email>, prepare_email已经在DAG: example_dag_decorator中注册
[2023-03-01 06:25:35,769] {example_kubernetes_executor.py:41} WARNING - example_kubernetes_executor示例DAG需要Kubernetes提供程序。请使用pip install apache-***[cncf.kubernetes]进行安装
[2023-03-01 06:25:35,772] {example_local_kubernetes_executor.py:39} WARNING - 无法在example_local_kubernetes_executor.py中导入DAGs
导入失败,最近的调用如下:
  File "/home/d5291029/venv/lib/python3.10/site-packages/airflow/example_dags/example_local_kubernetes_executor.py", line 37, in <module>
    from kubernetes.client import models as k8s
ModuleNotFoundError: 没有找到模块“kubernetes”
[2023-03-01 06:25:35,773] {example_local_kubernetes_executor.py:40} WARNING - 使用pip install apache-***[cncf.kubernetes]安装Kubernetes依赖
[2023-03-01 06:25:35,781] {taskmixin.py:205} WARNING - 依赖<Task(_PythonDecoratedOperator): print_the_context>, log_sql_query已经在DAG: example_python_operator中注册
[2023-03-01 06:25:35,781] {taskmixin.py:205} WARNING - 依赖<Task(_PythonDecoratedOperator): log_sql_query>, print_the_context已经在DAG: example_python_operator中注册
[2023-03-01 06:25:35,782] {taskmixin.py:205} WARNING - 依赖<Task(_PythonDecoratedOperator): print_the_context>, log_sql_query已经在DAG: example_python_operator中注册
[2023-03-01 06:25:35,782] {taskmixin.py:205} WARNING - 依赖<Task(_PythonDecoratedOperator): log_sql_query>, print_the_context已经在DAG: example_python_operator中注册
[2023-03-01 06:25:35,782] {taskmixin.py:205} WARNING - 依赖<Task(_PythonDecoratedOperator): print_the_context>, log_sql_query已经在DAG: example_python_operator中注册
[2023-03-01 06:25:35,782] {taskmixin.py:205} WARNING - 依赖<Task(_PythonDecoratedOperator): log_sql_query>, print_the_context已经在DAG: example_python_operator中注册
[2023-03-01 06:25:35,783] {taskmixin.py:205} WARNING - 依赖<Task(_PythonDecoratedOperator): print_the_context>, log_sql_query已经在DAG: example_python_operator中注册
[2023-03-01 06:25:35,783] {taskmixin.py:205} WARNING - 依赖<Task(_PythonDecoratedOperator): log_sql_query>, print_the_context已经在DAG: example_python_operator中注册
/home/d5291029/venv/lib/python3.10/site-packages/airflow/cli/commands/task_command.py:159 RemovedInAirflow3Warning: 调用“DAG.create_dagrun()”时没有显式数据间隔已被弃用

如何消除这些警告?

英文:

I just recently installed airflow and whenever I execute a task, I get warning about different dags:

[2023-03-01 06:25:35,691] {taskmixin.py:205} WARNING - Dependency <Task(BashOperator): create_entry_group>, delete_entry_group already registered for DAG: example_complex
[2023-03-01 06:25:35,691] {taskmixin.py:205} WARNING - Dependency <Task(BashOperator): delete_entry_group>, create_entry_group already registered for DAG: example_complex
[2023-03-01 06:25:35,691] {taskmixin.py:205} WARNING - Dependency <Task(BashOperator): create_entry_gcs>, delete_entry already registered for DAG: example_complex
[2023-03-01 06:25:35,692] {taskmixin.py:205} WARNING - Dependency <Task(BashOperator): delete_entry>, create_entry_gcs already registered for DAG: example_complex
[2023-03-01 06:25:35,692] {taskmixin.py:205} WARNING - Dependency <Task(BashOperator): create_tag>, delete_tag already registered for DAG:
 example_complex
[2023-03-01 06:25:35,692] {taskmixin.py:205} WARNING - Dependency <Task(BashOperator): delete_tag>, create_tag already registered for DAG:
 example_complex
[2023-03-01 06:25:35,759] {taskmixin.py:205} WARNING - Dependency <Task(_PythonDecoratedOperator): prepare_email>, send_email already registered for DAG: example_dag_decorator
[2023-03-01 06:25:35,759] {taskmixin.py:205} WARNING - Dependency <Task(EmailOperator): send_email>, prepare_email already registered for DAG: example_dag_decorator
[2023-03-01 06:25:35,769] {example_kubernetes_executor.py:41} WARNING - The example_kubernetes_executor example DAG requires the kubernetes provider. Please install it with: pip install apache-***[cncf.kubernetes]
[2023-03-01 06:25:35,772] {example_local_kubernetes_executor.py:39} WARNING - Could not import DAGs in example_local_kubernetes_executor.py
Traceback (most recent call last):
  File "/home/d5291029/venv/lib/python3.10/site-packages/airflow/example_dags/example_local_kubernetes_executor.py", line 37, in <module>
    from kubernetes.client import models as k8s
ModuleNotFoundError: No module named 'kubernetes'
[2023-03-01 06:25:35,773] {example_local_kubernetes_executor.py:40} WARNING - Install Kubernetes dependencies with: pip install apache-***[cncf.kubernetes]
[2023-03-01 06:25:35,781] {taskmixin.py:205} WARNING - Dependency <Task(_PythonDecoratedOperator): print_the_context>, log_sql_query already registered for DAG: example_python_operator
[2023-03-01 06:25:35,781] {taskmixin.py:205} WARNING - Dependency <Task(_PythonDecoratedOperator): log_sql_query>, print_the_context already registered for DAG: example_python_operator
[2023-03-01 06:25:35,782] {taskmixin.py:205} WARNING - Dependency <Task(_PythonDecoratedOperator): print_the_context>, log_sql_query already registered for DAG: example_python_operator
[2023-03-01 06:25:35,782] {taskmixin.py:205} WARNING - Dependency <Task(_PythonDecoratedOperator): log_sql_query>, print_the_context already registered for DAG: example_python_operator
[2023-03-01 06:25:35,782] {taskmixin.py:205} WARNING - Dependency <Task(_PythonDecoratedOperator): print_the_context>, log_sql_query already registered for DAG: example_python_operator
[2023-03-01 06:25:35,782] {taskmixin.py:205} WARNING - Dependency <Task(_PythonDecoratedOperator): log_sql_query>, print_the_context already registered for DAG: example_python_operator
[2023-03-01 06:25:35,783] {taskmixin.py:205} WARNING - Dependency <Task(_PythonDecoratedOperator): print_the_context>, log_sql_query already registered for DAG: example_python_operator
[2023-03-01 06:25:35,783] {taskmixin.py:205} WARNING - Dependency <Task(_PythonDecoratedOperator): log_sql_query>, print_the_context already registered for DAG: example_python_operator
/home/d5291029/venv/lib/python3.10/site-packages/airflow/cli/commands/task_command.py:159 RemovedInAirflow3Warning: Calling `DAG.create_dagrun()` without an explicit data interval is deprecated

How do i get rid of these warnings ?

答案1

得分: 1

很不幸,这已经是众所周知的行为已经有一段时间了。如果您想跟踪,可以查看GitHub讨论主题

除非您不混合使用TaskFlow任务和经典操作器,否则这个警告将继续存在。它有点干扰,但不影响功能。

英文:

Unfortunately, this is known behavior for quite some time. There is a GitHub Discussion thread about this if you'd like to follow along.

Unless you don't mix TaskFlow tasks and classic operators, this warning will continue to persist. It's somewhat intrusive, but it doesn't impact functionality.

huangapple
  • 本文由 发表于 2023年3月1日 14:27:22
  • 转载请务必保留本文链接:https://go.coder-hub.com/75600219.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定