在dbt项目中,`env_var`函数的行为不如预期。

huangapple go评论110阅读模式
英文:

env_var function in dbt project is not behaving as expected

问题

在我的 dbt_project.yml 中:

target-path: "{{ env_var('DBT_RUNTIME_DIR', '') }}target" # 存储编译后 SQL 文件的目录
log-path: "{{ env_var('DBT_RUNTIME_DIR', '') }}logs" # 存储编译后 SQL 文件的目录

我使用了 env_var() 函数来使用环境变量。如果没有设置 DBT_RUNTIME_DIR,那么路径应该只是 "logs"(因为从 env_var 输出应该是空字符串)。

不幸的是,它创建了一个名为 {{ env_var('DBT_RUNTIME_DIR', '') }}logs 的目录,这意味着函数的工作方式与预期不符。

如果我更改为不存在的环境变量,那么它将不会呈现:

log-path: "{{ env_var('non-existing-env-var') }}logs" # 存储编译后 SQL 文件的目录
11:17:06  使用 dbt=1.5.1 运行
11:17:07  遇到错误:
解析错误
  需要环境变量但未提供:'non-existing-env-var'

这意味着函数是有效的,但不按预期工作。

如果我添加任何内容到 "{{ env_var('DBT_RUNTIME_DIR') }}" 中,它只会更改目录的名称。

如果我更改为已存在的变量:

export existing_variable=test
echo $existing_variable
>>> test
log-path: "{{ env_var('existing_variable') }}" # 存储编译后 SQL 文件的目录

这是 dbt 中的一个错误还是我漏掉了什么?

英文:

in my dbt_project.yml

target-path: "{{ env_var('DBT_RUNTIME_DIR', '') }}target" # directory which will store compiled SQL files
log-path: "{{ env_var('DBT_RUNTIME_DIR', '') }}logs" # directory which will store compiled SQL files

I used env_var() function to use environment variables. If there is no DBT_RUNTIME_DIR set then the path should be just logs (because output from env_var should be empty string).

Unfortunately, it creates directory which is named
{{ env_var('DBT_RUNTIME_DIR', '') }}logs which means function is not working as expected.

在dbt项目中,`env_var`函数的行为不如预期。

If I change it to non-existing env-var, then it will not render:

log-path: "{{ env_var('non-existing-env-var') }}logs" # directory which will store compiled SQL files
11:17:06  Running with dbt=1.5.1
11:17:07  Encountered an error:
Parsing Error
  Env var required but not provided: 'non-existing-env-var'

so it means function is working, but not as supposed to.

If I add antything to "{{ env_var('DBT_RUNTIME_DIR') }}" it will just change the name of directory.

If I change to existing variable

export existing_variable=test
echo $existing_variable
>>> test
log-path: "{{ env_var('existing_variable') }}" # directory which will store compiled SQL files

在dbt项目中,`env_var`函数的行为不如预期。

Is there a bug in dbt or I missed something?

答案1

得分: 2

dbt_project.yml 中可用于使用的 Jinja 上下文非常有限。请参阅 https://docs.getdbt.com/reference/dbt-jinja-functions/dbt-project-yml-context。从该页面,您应该注意以下内容:

这适用于 dbt_project.yml 文件中的 models:、seeds: 和 snapshots: 键。

在您的 yml 中的这些节点中,您也受到可以调用的 Jinja 方法的限制。类似地,在 on-run-starton-run-end(https://docs.getdbt.com/reference/dbt-jinja-functions/on-run-end-context)以及您的 profiles.yml 配置中也提供了一些 Jinja(https://docs.getdbt.com/reference/dbt-jinja-functions/profiles-yml-context)。

您看到的行为原因是在运行 DBT 时 dbt_project.yml 被多次使用。首先,它用于路径配置,因此当您有类似 target-path: "{{ env_var('DBT_RUNTIME_DIR', '') }}target" 的内容时,它将只使用字面字符串。然后,它将再次解析项目配置文件中允许的 Jinja 部分。但是,在此阶段它会解析整个文件,这就是为什么没有默认值的版本会失败的原因。

英文:

The Jinja context available to use inside dbt_project.yml is very limited. See https://docs.getdbt.com/reference/dbt-jinja-functions/dbt-project-yml-context. From that page, you should note the following:

> This applies to the models:, seeds:, and snapshots: keys in the dbt_project.yml file.

Within those nodes in your yml, you're also limited to what Jinja methods you can call. Similarly, there is some Jinja available to you in on-run-start, on-run-end (https://docs.getdbt.com/reference/dbt-jinja-functions/on-run-end-context) and within your profiles.yml configuration as well (https://docs.getdbt.com/reference/dbt-jinja-functions/profiles-yml-context).

The reason for the behavior you're seeing is because dbt_project.yml is used multiple times when running DBT. It's used for the path configurations at first, so when you have something like target-path: "{{ env_var('DBT_RUNTIME_DIR', '') }}target", it will just use the literal string. Then, it will do another pass to actually parse the Jinja in the project config file for the areas where it is allowed. However, it does parse the entire file during this phase which is why your version without the default values is failing.

huangapple
  • 本文由 发表于 2023年7月10日 19:25:04
  • 转载请务必保留本文链接:https://go.coder-hub.com/76653253.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定