英文:
env_var function in dbt project is not behaving as expected
问题
在我的 dbt_project.yml
中:
target-path: "{{ env_var('DBT_RUNTIME_DIR', '') }}target" # 存储编译后 SQL 文件的目录
log-path: "{{ env_var('DBT_RUNTIME_DIR', '') }}logs" # 存储编译后 SQL 文件的目录
我使用了 env_var()
函数来使用环境变量。如果没有设置 DBT_RUNTIME_DIR
,那么路径应该只是 "logs"(因为从 env_var
输出应该是空字符串)。
不幸的是,它创建了一个名为 {{ env_var('DBT_RUNTIME_DIR', '') }}logs
的目录,这意味着函数的工作方式与预期不符。
如果我更改为不存在的环境变量,那么它将不会呈现:
log-path: "{{ env_var('non-existing-env-var') }}logs" # 存储编译后 SQL 文件的目录
11:17:06 使用 dbt=1.5.1 运行
11:17:07 遇到错误:
解析错误
需要环境变量但未提供:'non-existing-env-var'
这意味着函数是有效的,但不按预期工作。
如果我添加任何内容到 "{{ env_var('DBT_RUNTIME_DIR') }}"
中,它只会更改目录的名称。
如果我更改为已存在的变量:
export existing_variable=test
echo $existing_variable
>>> test
log-path: "{{ env_var('existing_variable') }}" # 存储编译后 SQL 文件的目录
这是 dbt 中的一个错误还是我漏掉了什么?
英文:
in my dbt_project.yml
target-path: "{{ env_var('DBT_RUNTIME_DIR', '') }}target" # directory which will store compiled SQL files
log-path: "{{ env_var('DBT_RUNTIME_DIR', '') }}logs" # directory which will store compiled SQL files
I used env_var() function to use environment variables. If there is no DBT_RUNTIME_DIR
set then the path should be just logs (because output from env_var should be empty string).
Unfortunately, it creates directory which is named
{{ env_var('DBT_RUNTIME_DIR', '') }}logs
which means function is not working as expected.
If I change it to non-existing env-var, then it will not render:
log-path: "{{ env_var('non-existing-env-var') }}logs" # directory which will store compiled SQL files
11:17:06 Running with dbt=1.5.1
11:17:07 Encountered an error:
Parsing Error
Env var required but not provided: 'non-existing-env-var'
so it means function is working, but not as supposed to.
If I add antything to "{{ env_var('DBT_RUNTIME_DIR') }}"
it will just change the name of directory.
If I change to existing variable
export existing_variable=test
echo $existing_variable
>>> test
log-path: "{{ env_var('existing_variable') }}" # directory which will store compiled SQL files
Is there a bug in dbt or I missed something?
答案1
得分: 2
dbt_project.yml
中可用于使用的 Jinja 上下文非常有限。请参阅 https://docs.getdbt.com/reference/dbt-jinja-functions/dbt-project-yml-context。从该页面,您应该注意以下内容:
这适用于 dbt_project.yml 文件中的 models:、seeds: 和 snapshots: 键。
在您的 yml 中的这些节点中,您也受到可以调用的 Jinja 方法的限制。类似地,在 on-run-start
、on-run-end
(https://docs.getdbt.com/reference/dbt-jinja-functions/on-run-end-context)以及您的 profiles.yml
配置中也提供了一些 Jinja(https://docs.getdbt.com/reference/dbt-jinja-functions/profiles-yml-context)。
您看到的行为原因是在运行 DBT 时 dbt_project.yml
被多次使用。首先,它用于路径配置,因此当您有类似 target-path: "{{ env_var('DBT_RUNTIME_DIR', '') }}target"
的内容时,它将只使用字面字符串。然后,它将再次解析项目配置文件中允许的 Jinja 部分。但是,在此阶段它会解析整个文件,这就是为什么没有默认值的版本会失败的原因。
英文:
The Jinja context available to use inside dbt_project.yml
is very limited. See https://docs.getdbt.com/reference/dbt-jinja-functions/dbt-project-yml-context. From that page, you should note the following:
> This applies to the models:, seeds:, and snapshots: keys in the dbt_project.yml file.
Within those nodes in your yml, you're also limited to what Jinja methods you can call. Similarly, there is some Jinja available to you in on-run-start
, on-run-end
(https://docs.getdbt.com/reference/dbt-jinja-functions/on-run-end-context) and within your profiles.yml
configuration as well (https://docs.getdbt.com/reference/dbt-jinja-functions/profiles-yml-context).
The reason for the behavior you're seeing is because dbt_project.yml
is used multiple times when running DBT. It's used for the path configurations at first, so when you have something like target-path: "{{ env_var('DBT_RUNTIME_DIR', '') }}target"
, it will just use the literal string. Then, it will do another pass to actually parse the Jinja in the project config file for the areas where it is allowed. However, it does parse the entire file during this phase which is why your version without the default values is failing.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论