英文:
How do you write Data from a Spark Data Frame to Azure Application Insights from data bricks using python?
问题
以下是您要翻译的代码部分：
try:
# 检查对账数据框的记录数，并且如果记录数大于 0，则将其发送到 Azure Application Insights，表示存在一个或多个对账错误。
if reconciliation_DF.count() > 0:
reconciliation_DF_json = reconciliation_DF.toJSON().collect() # 将数据框转换为 JSON
reconciliation_DF_json = json.dumps(reconciliation_DF_json) # 将 JSON 列表转换为用于日志记录的字符串
# 在这里编写代码将 "reconciliation_DF_json" 数据写入 Azure Application Insights？？？
# 这是我卡住的地方。
else:
print("未找到对账错误")
except Exception as e:
print("发生错误：{}".format(e))
raise
希望这有所帮助。如果您有其他问题或需要进一步的帮助，请随时提出。
英文:
I have a spark data frame that has some reconciliation data and this data frame only gets created if reconciliation fails.
When this spark data frame does get created in databricks, I need to send the data in it to azure application insights from databricks.
try:
# Check count of reconciliation Data Frame and send to Application Insights if count > 0, meaning there's one or more reconciliation error(s).
if reconciliation_DF.count() > 0:
reconciliation_DF_json = reconciliation_DF.toJSON().collect() # Convert Data Frame to JSON
reconciliation_DF_json = json.dumps(reconciliation_DF_json) # Convert list of JSONs to string for logging
# Some code here to write this "reconciliation_DF_json" data to Azure Application Insights???
# This is where I am stuck.
else:
print("No reconciliation errors found")
except Exception as e:
print("An error occurred: {}".format(e))
raise
There seems to be a library, “Open Telemetry”, but the Microsoft documentation isn’t clear on how to use that for my scenario.
A python code example of how to do this with a data frame named "reconciliation_DF" would be much appreciated.
答案1
得分: 1
我在 Azure Monitor 的 Python OpenTelemetry 解决方案上工作。让我知道这是否有所帮助。使用 azure-monitor-opentelemetry 库，您可以通过单个 configure_azure_monitor() 调用来对常见库和 Python 日志进行仪表化。然后，您可以使用原生的 Python 日志库将跟踪发送到 Application Insights：
configure_azure_monitor(connection_string=CONN_STR)
logger.setLevel(logging.INFO)
try:
# 检查协调数据框的计数，如果计数大于 0，则表示存在一个或多个协调错误。
if reconciliation_DF.count() > 0:
reconciliation_DF_json = reconciliation_DF.toJSON().collect() # 将数据框转换为 JSON
reconciliation_DF_json = json.dumps(reconciliation_DF_json) # 将 JSON 列表转换为字符串以进行日志记录
logger.info(reconciliation_DF_json)
else:
print("未找到协调错误")
except Exception as e:
print("发生错误: {}".format(e))
raise
请注意，您也可以使用环境变量设置连接字符串。可能有几种方法可以实现您尝试的操作。请让我知道这是否符合您的用例。
英文:
I work on Azure Monitor's Python OpenTelemetry solutions. Let me know if this helps. Using the azure-monitor-opentelemetry library, you can instrument common libraries and python logs via a single configure_azure_monitor() call. You can then use the native python logging library to sent traces to Application Insights:
configure_azure_monitor(connection_string=CONN_STR)
logger.setLevel(logging.INFO)
try:
# Check count of reconciliation Data Frame and send to Application Insights if count > 0, meaning there's one or more reconciliation error(s).
if reconciliation_DF.count() > 0:
reconciliation_DF_json = reconciliation_DF.toJSON().collect() # Convert Data Frame to JSON
reconciliation_DF_json = json.dumps(reconciliation_DF_json) # Convert list of JSONs to string for logging
logger.info(reconciliation_DF_json)
else:
print("No reconciliation errors found")
except Exception as e:
print("An error occurred: {}".format(e))
raise
Note that you can also set the connection string with an environment variable. There may be a few ways to do what you are trying to. Please, let me know if this fits your use case.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论