英文:
Accessing value of {{ts_nodash}} in airflow
问题
我需要访问{{ts_nodash}}的值，以将其格式化为日期时间并减去一小时。但这在Dag上给了我一个错误，说：“ValueError: time data '{{ ts_nodash }}' does not match format '%Y%m%dT%H%M%S'”
我的方法如下所示：
ts_nodash = datetime.strptime("{{ ts_nodash }}", "%Y%m%dT%H%M%S")
get_numbers = gke_wrapper.execute_gke_operator(
task_id=f"get_numbers_{ctype_name}",
labels=get_cost_labels(
pod=Pod.pod,
service="service",
id="mmoe",
environment="dev",
component="airflow",
),
cmd=["sh"],
args=[
"get_numbers.sh",
output_table_name,
dump_table_name,
model_path,
model_path,
ctype_name,
config.variants[ctype_name],
(ts_nodash - timedelta(hours=config.delay_interval)).strftime(
"%Y%m%dT%H%M%S"
),
],
nodepool=_NODE_POOL,
image=_IMAGE,
resource=_RESOURCES,
)
有人能提到如何访问模板变量的值吗？
我尝试了许多不同的方法来在args=[]中传递{{ts_nodash}}，但显然在模板值可用之前会执行日期时间解析操作。任何帮助都将不胜感激。
英文:
I need to access the value of {{ts_nodash}} to format it as a datetime and subtract an hour from it. But this gives me an error on dag saying: ValueError: time data '{{ ts_nodash }}' does not match format '%Y%m%dT%H%M%S'
This is how my method looks like:
ts_nodash = datetime.strptime("{{ ts_nodash }}", "%Y%m%dT%H%M%S")
get_numbers = gke_wrapper.execute_gke_operator(
task_id=f"get_numbers_{ctype_name}",
labels=get_cost_labels(
pod=Pod.pod,
service="service",
id="mmoe",
environment="dev",
component="airflow",
),
cmd=["sh"],
args=[
"get_numbers.sh",
output_table_name,
dump_table_name,
model_path,
model_path,
ctype_name,
config.variants[ctype_name],
(ts_nodash - timedelta(hours=config.delay_interval)).strftime(
"%Y%m%dT%H%M%S"
),
],
nodepool=_NODE_POOL,
image=_IMAGE,
resource=_RESOURCES,
)
Can someone mention how can I access the template variable value?
I tried many different ways of passing {{ts_nodash}} within args=[] but apparently the datetime parsing operation occurs before the template value is made available. Any help will be much appreciated.
答案1
得分: 1
模板在Dag运行期间评估（即在操作员的execute方法内部）。看起来你正试图在Dag的运行时之外获取ts_nodash。
以下是使用模板的示例：
op = BashOperator(
task_id="pre-validation",
bash_command="echo 'this is ts_nodash {{ ts_nodash }}'\n"
"echo 'this is dag_run.logical_date {{ dag_run.logical_date }}'",
dag=dag,
)
因此，你需要像下面这样将模板本身传递给操作员，它将在运行时进行评估：
...
args = [
"get_numbers.sh",
output_table_name,
dump_table_name,
model_path,
model_path,
ctype_name,
config.variants[ctype_name],
"{{ ts_nodash }}",
],
...
英文:
Templates evaluated during the Dag run (i.e. inside the execute method of the operator). Looks like you are trying to fetch ts_nodash outside the run time of the dag.
Below is an example of using templates:
op = BashOperator(
task_id="pre-validation",
bash_command="echo 'this is ts_nodash {{ ts_nodash }}'\n"
"echo 'this is dag_run.logical_date {{ dag_run.logical_date }}'",
dag=dag,
)
So you will have to pass the template itself to the operator like below and it will be evaluated during the run time:
...
args = [
"get_numbers.sh",
output_table_name,
dump_table_name,
model_path,
model_path,
ctype_name,
config.variants[ctype_name],
"{{ ts_nodash }}"
],
...
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论