英文:
Accessing value of {{ts_nodash}} in airflow
问题
我需要访问{{ts_nodash}}的值，以将其格式化为日期时间并减去一小时。但这在Dag上给了我一个错误，说：“ValueError: time data '{{ ts_nodash }}' does not match format '%Y%m%dT%H%M%S'”
我的方法如下所示：
ts_nodash = datetime.strptime("{{ ts_nodash }}", "%Y%m%dT%H%M%S")get_numbers = gke_wrapper.execute_gke_operator(task_id=f"get_numbers_{ctype_name}",labels=get_cost_labels(pod=Pod.pod,service="service",id="mmoe",environment="dev",component="airflow",),cmd=["sh"],args=["get_numbers.sh",output_table_name,dump_table_name,model_path,model_path,ctype_name,config.variants[ctype_name],(ts_nodash - timedelta(hours=config.delay_interval)).strftime("%Y%m%dT%H%M%S"),],nodepool=_NODE_POOL,image=_IMAGE,resource=_RESOURCES,)
有人能提到如何访问模板变量的值吗？
我尝试了许多不同的方法来在args=[]中传递{{ts_nodash}}，但显然在模板值可用之前会执行日期时间解析操作。任何帮助都将不胜感激。
英文:
I need to access the value of {{ts_nodash}} to format it as a datetime and subtract an hour from it. But this gives me an error on dag saying: ValueError: time data '{{ ts_nodash }}' does not match format '%Y%m%dT%H%M%S'
This is how my method looks like:
ts_nodash = datetime.strptime("{{ ts_nodash }}", "%Y%m%dT%H%M%S")get_numbers = gke_wrapper.execute_gke_operator(task_id=f"get_numbers_{ctype_name}",labels=get_cost_labels(pod=Pod.pod,service="service",id="mmoe",environment="dev",component="airflow",),cmd=["sh"],args=["get_numbers.sh",output_table_name,dump_table_name,model_path,model_path,ctype_name,config.variants[ctype_name],(ts_nodash - timedelta(hours=config.delay_interval)).strftime("%Y%m%dT%H%M%S"),],nodepool=_NODE_POOL,image=_IMAGE,resource=_RESOURCES,)
Can someone mention how can I access the template variable value?
I tried many different ways of passing {{ts_nodash}} within args=[] but apparently the datetime parsing operation occurs before the template value is made available. Any help will be much appreciated.
答案1
得分: 1
模板在Dag运行期间评估（即在操作员的execute方法内部）。看起来你正试图在Dag的运行时之外获取ts_nodash。
以下是使用模板的示例：
op = BashOperator(task_id="pre-validation",bash_command="echo 'this is ts_nodash {{ ts_nodash }}'\n""echo 'this is dag_run.logical_date {{ dag_run.logical_date }}'",dag=dag,)
因此，你需要像下面这样将模板本身传递给操作员，它将在运行时进行评估：
...args = ["get_numbers.sh",output_table_name,dump_table_name,model_path,model_path,ctype_name,config.variants[ctype_name],"{{ ts_nodash }}",],...
英文:
Templates evaluated during the Dag run (i.e. inside the execute method of the operator). Looks like you are trying to fetch ts_nodash outside the run time of the dag.
Below is an example of using templates:
op = BashOperator(task_id="pre-validation",bash_command="echo 'this is ts_nodash {{ ts_nodash }}'\n""echo 'this is dag_run.logical_date {{ dag_run.logical_date }}'",dag=dag,)
So you will have to pass the template itself to the operator like below and it will be evaluated during the run time:
...args = ["get_numbers.sh",output_table_name,dump_table_name,model_path,model_path,ctype_name,config.variants[ctype_name],"{{ ts_nodash }}"],...
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论