英文:
DBT - get a ref model config variables inside a macro
问题
我正在使用宏来自动化和创建通用的DBT增量过程。
问题：
我尝试从一个我在另一个模型中引用的模型中获取一些配置变量。对于我的每个模型，我在配置中都保留了一个timestamp_column。
我希望能够在需要时从配置中获取timestamp_column的值。我没有找到合适的方法来实现这一点。
代码示例：
例如，在我的_models.yml中：
- name: L0__game_runs
config:
materialized: incremental
time_interval: hour
timestamp_column: time_started
- name: L1__drafts
config:
materialized: incremental
time_interval: hour
timestamp_column: created_at
L1__drafts模型的查询引用了L0__game_runs，如下所示：
with
{% if is_incremental() %}
{% set self_max_timestamp = get_table_max_timestamp(this) %}
drafts as {{ (generate_source_query('L0__game_runs', self_max_timestamp)) }}
{% else %}
drafts as (select * from {{ ref('L0__game_runs') }}),
{% endif %}
<SQL的其余部分在这里...>
generate_source_query.sql宏：
{% macro generate_source_query(source_model, self_max_timestamp) %}
{%- set source_timestamp_column = source_model.get('timestamp_column') -%}
<宏的其余部分在这里...>
{% endmacro %}
但是source_timestamp_column返回空字符串结果。我没有找到其他解决方法。有什么想法吗？
英文:
I am using macros to automate and make generic DBT incremental procedures.
Problem:
I try to get some config-variables from a model i use as a ref in another model. for each of my models i keep a timestamp_column in config.
I want to be able to get the timestamp_column value from config whenever i need. I didn't find any proper way to do so.
Code Example:
for example in my _models.yml:
- name: L0__game_runs
config:
materialized: incremental
time_interval: hour
timestamp_column: time_started
- name: L1__drafts
config:
materialized: incremental
time_interval: hour
timestamp_column: created_at
TheL1__drafts model query is referring L0__game_runs like:
with
{% if is_incremental() %}
{% set self_max_timestamp = get_table_max_timestamp(this) %}
drafts as {{ (generate_source_query(ref('L0__game_runs'), self_max_timestamp)) }}
{% else %}
drafts as (select * from {{ ref('L0__game_runs') }}),
{% endif %}
<REST OF SQL COMES HERE...>
generate_source_query.sql macro:
{% macro generate_source_query(source_model, self_max_timestamp) %}
{%- set source_timestamp_column = source_model.get('timestamp_column') -%}
<REST OF MACRO COMES HERE...>
{% endmacro %}
but source_timestamp_columnreturn empty string result. didn't find any other way to fix it.
Any ideas?
答案1
得分: 0
使用DBT图上下文的方式，经过大量研究，我找到了解决方法，可以获取所有DBT模型配置（以及更多信息）。
我编写了一个宏来获取引用模型配置的值。虽然它有点简单，但对我来说现在足够好了：
{% macro get_model_config_values(model_ref) %}
{%- set table_name = model_ref.identifier -%}
{% for node in graph.nodes.values() %}
{%- set model_name = node.unique_id.split('.').pop() -%}
{%- if table_name == model_name -%}
{%- set model_config = node.config -%}
{{ return(model_config) }}
{%- endif -%}
{% endfor %}
{% endmacro %}
<details>
<summary>英文:</summary>
After a lot of digging, i found a way to solve it - using the [DBT graph context][1] which get all DBT models configs (and more).
I wrote a macro to retrieve a ref model config values. its a bit naive but it good enough for me right now:
{% macro get_model_config_values(model_ref) %}
{%- set table_name = model_ref.identifier -%}
{% for node in graph.nodes.values() %}
{%- set model_name = node.unique_id.split('.')[-1] -%}
{%- if table_name == model_name -%}
{%- set model_config = node.config -%}
{{ return(model_config) }}
{%- endif -%}
{% endfor %}
{% endmacro %}
[1]: https://docs.getdbt.com/reference/dbt-jinja-functions/graph
</details>
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论