英文:
Alternative to 30+ `| rex field=path mode=sed...` in order to replace path parameters in urls
问题
我正在编写一个Splunk查询，用于总结AWS ELB访问日志。目的是收集所有需要的信息，以便进行模拟生产流量的负载测试。
我们的许多URL包括路径参数。例如，目录组件可以是GUID、用户帐户ID等。为了将所有对特定端点的调用折叠成单个字符串，我需要查找并替换这些路径参数为表示参数的字符串。例如：/users/3h5l2h5fn1 变成 /users/{accountId}。这很容易，我只需使用一系列 | rex field=path mode=sed 命令。
我遇到的问题是，当我达到大约30个这样的命令时，查询开始失败。没有明显的原因，作业只是被终止。
我的问题是，是否有另一种在路径字段上执行50多次查找和替换的方法？一种不会破坏Splunk的方法？
英文:
I am composing a Splunk query to summarize AWS ELB access logs. The purpose is to gather all the information needed to inform a load test that approximates production traffic.
Many of our URLs include path parameters. For example, a directory component may be a GUID, or a user's account ID, etc. In order to collapse all calls to a particular endpoint into a single string, I need to find and replace those path parameters with a string that represent the parameter. For example: /users/3h5l2h5fn1 becomes /users/{accountId}. This is easy enough, I just use a sequence of | rex field=path mode=sed commands.
The problem I'm encountering is that about the time I hit 30 such commands, the query starts failing. There's no clear indication why, the job just gets killed.
My question for the Stack Overflow experts is: Is there another way of doing a series of 50+ find-and-replace on the path field? One that won't break Splunk?
答案1
得分: 0
这听起来像是 transforms.conf 的任务，或者可能只需使用 props.conf 正确提取所有这些字段。
如果您等到搜索时才掩盖数据，那么您仍然在 存储 您尝试掩盖的所有内容。
最后，根据我的经验，eval myfield=replace(my_field, "正则表达式", "文字字符串") 几乎总是比 rex mode=sed 更快。
有关 eval、rex 的 Splunk 文档参考链接：
- https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/eval
- https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/rex
英文:
this sounds like a job for transforms.conf or maybe just to properly extract all those fields with props.conf.
If you're waiting until search time to mask data, you're still storing all of what you're trying to mask
Lastly, eval myfield=replace(my_field,"regex","literal string") is almost always faster, in my experience, than rex mode=sed
Doc.Splunk references for eval, rex:
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论