连接查询本地 MongoDB 数据库从 DAG Airflow

huangapple go评论88阅读模式
英文:

Connect query local mongodb database from dag airflow

问题

这是你要翻译的内容:

  1. Airflow初学者的问题:
  2. 我无法从DAG airflow中的任务中查询我的本地mongodb数据库。
  3. DAGAirflow中被正确导入,但当我启动它时,任务因超时服务器而失败:
  4. [在此输入图像描述](https://i.stack.imgur.com/U7850.png)
  5. 这是我在Airflow中设置的连接:
  6. [在此输入图像描述](https://i.stack.imgur.com/XGrXI.png)
  7. 这是我用于初始化代码的Python函数:
  8. ```python
  9. from airflow import DAG
  10. from airflow.utils.dates import days_ago
  11. from airflow.operators.python import PythonOperator
  12. from airflow.providers.mongo.hooks.mongo import MongoHook
  13. from pymongo.mongo_client import MongoClient
  14. def connecter():
  15. hook = MongoHook(mongo_conn_id='mongo_default')
  16. client = hook.get_conn()
  17. my_db = client['Immo']
  18. return my_db
  19. mydag = DAG(
  20. dag_id='my_dag',
  21. default_args={
  22. 'owner': 'airflow',
  23. 'start_date': days_ago(0),
  24. }
  25. )
  26. def get_one(c: str):
  27. a = connecter().find({"id_transaction": c})[0]['_id_transaction']
  28. print(a)
  29. return a
  30. task_1_0 = PythonOperator(
  31. task_id='test_connection_my_bd',
  32. python_callable=db.get_one,
  33. dag=mydag,
  34. op_kwargs={'c': 107332}
  35. )

我在容器中运行Airflow,使用社区Airflow构建的docker-compose:
https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml

当我在DAG airflow之外运行我的函数get_one(c:str),并使用connecter_local函数时,它可以正常工作:

  1. def connecter_local():
  2. uri = "mongodb://localhost:27017"
  3. client = MongoClient(uri)
  4. my_db = client['Immo']
  5. return my_db
  6. def get_one_local(c: str):
  7. a = connecter_local().find({"id_transaction": c})[0]['_id_transaction']
  8. print(a)
  9. return a
  10. print(get_one_local('1'))

提前感谢。

  1. <details>
  2. <summary>英文:</summary>
  3. question from beginner in Airflow:
  4. I can&#39;t query my local mongodb database from a task in a DAG airflow.
  5. the dag is correctly imported in Airflow but when i start it the task fail because of timeout servor:
  6. [enter image description here](https://i.stack.imgur.com/U7850.png)
  7. Here the connection I set into Airflow:
  8. [enter image description here](https://i.stack.imgur.com/XGrXI.png)
  9. Here my pyhton function to initialize code:

from airflow import DAG
from airflow.utils.dates import days_ago
from airflow.operators.python import PythonOperator
from airflow.providers.mongo.hooks.mongo import MongoHook
from pymongo.mongo_client import MongoClient

def connecter():
hook = MongoHook(mongo_conn_id='mongo_default')
client = hook.get_conn()
my_db = client['Immo']
return my_db

mydag = DAG(
dag_id='my_dag',
default_args={
'owner': 'airflow',
'start_date': days_ago(0),
}
)

def get_one(c: str):

  1. a=connecter().find({&quot;id_transaction&quot;: c})[0][&#39;_id_transaction&#39;]
  2. print(a)
  3. return a

task_1_0 = PythonOperator(
task_id='test_connection_my_bd',
python_callable=db.get_one,
dag=mydag,
op_kwargs={'c': 107332}
)

  1. I run airflow inside containers using the docker-compose built by the community airflow:
  2. https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml
  3. When I run my function get_one(c:str) outside of dag airflow with the function connecter_local it works fine:

def connecter_local():
uri= "mongodb://localhost:27017"
client = MongoClient(uri)
my_db = client['Immo']
return my_db

def get_one_local(c: str):

  1. a=connecter_local().find({&quot;id_transaction&quot;: c})[0][&#39;_id_transaction&#39;]
  2. print(a)
  3. return a

print(get_one_local('1'))

  1. Thanks in advance.
  2. </details>
  3. # 答案1
  4. **得分**: 0
  5. 您的Airflow正在Docker上运行,而MongoDB在您的计算机上(不在Airflow Docker内部)。当您配置host=127.0.0.1时,它位于Docker内部网络,而不是主机IP。您应该将host更改为您的计算机名称或IP地址。
  6. <details>
  7. <summary>英文:</summary>
  8. Your airflow is running on a docker and the mongo-db on your machine (not inside the airflow docker).
  9. when you configure host=127.0.0.1 its inside the docker (internal network) and its not the host ip. you should change host to your computer-name or ip
  10. </details>

huangapple
  • 本文由 发表于 2023年5月29日 23:16:25
  • 转载请务必保留本文链接:https://go.coder-hub.com/76358480.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定