英文:
DAG run marked as success while task failed
问题
def my_main(**kwargs):
try:
....
except:
print("错误：", pprint.pformat(traceback.format_exc(), indent=1))
dag = DAG(
'my_main',
description='my_main',
start_date=days_ago(0),
schedule_interval='@daily',
catchup=False
)
task_checker = PythonOperator(task_id='task_checker_my_main', provide_context=True, python_callable=my_main, dag=dag)
英文:
I have some PythonOperator in my DAG which time to time throws error therefore simply it goes into except statement and entire process is stopped due to that. What i do not understand is my dag run is marked as success. Can anyone explains that to me please as i thought if code inside task throws an error the dag should be marked as failed, not? Why dag is marked success?
DAG (cutted):
import time
import datetime
import json
import pprint
import requests
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.python_operator import PythonOperator
from airflow.operators.python_operator import BranchPythonOperator
def my_main(**kwargs):
try:
....
except:
print("Error:", pprint.pformat(traceback.format_exc(), indent=1))
dag = DAG(
'my_main',
description='my_main',
start_date=days_ago(0),
schedule_interval='@daily',
catchup=False
)
task_checker = PythonOperator(task_id='task_checker_my_main', provide_context=True, python_callable=my_main, dag=dag)
答案1
得分: 2
你需要触发异常。你“吞掉”了真正的异常，所以函数以无错误的方式结束。在打印后直接触发异常。
def my_main(**kwargs):
try:
....
except:
print("错误:", pprint.pformat(traceback.format_exc(), indent=1))
raise AirflowException("你的消息")
英文:
You need to raise Exception. you "swallow" the real exception so the function ends with no error . just raise exception after the print.
def my_main(**kwargs):
try:
....
except:
print("Error:", pprint.pformat(traceback.format_exc(), indent=1))
raise AirflowException("your message")
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论