Filebeat 如何检查文件中的新内容?

huangapple go评论113阅读模式
英文:

How filebeat checks for new content in a file?

问题

Filebeat使用tail -f命令来检查文件中的新内容,并将其刷新到所需的输出吗?还是它还有其他检查文件中新内容的方式?

英文:

Does filebeat uses tail -f to check for new contents in a file and then flushes it over to the desired output? Or is there any other way it checks for new contents in a file?

答案1

得分: 1

由于Filebeat是开源的,你可以随时自己查看

以下是来自上述链接文件的Go代码,用于检查文件是否已更新。

我已经大大简化了这段代码,任何你看到 ... 的地方都是不相关的代码块,我鼓励任何阅读此文的人去查看整个文件,它是一些非常优秀的Go代码。

  1. // Scan starts a scanGlob for each provided path/glob
  2. func (p *ProspectorLog) scan() {
  3. newlastscan := time.Now()
  4. // Now let's do one quick scan to pick up new files
  5. for _, path := range p.config.Paths {
  6. p.scanGlob(path)
  7. }
  8. p.lastscan = newlastscan
  9. }

上述函数在每个指定的时间块 n 被调用一次,其中 n 在配置中指定。调用了 scanGlob,如下所示。

  1. // Scans the specific path which can be a glob (/**/**/*.log)
  2. // For all found files it is checked if a harvester should be started
  3. func (p *ProspectorLog) scanGlob(glob string) {
  4. ...
  5. // Evaluate the path as a wildcards/shell glob
  6. matches, err := filepath.Glob(glob)
  7. ...
  8. // Check any matched files to see if we need to start a harvester
  9. for _, file := range matches {
  10. ...

对于所有匹配的文件,使用特定于操作系统的调用检查文件的统计信息,对于Linux来说,这将是 stat <file>

  1. // Stat the file, following any symlinks.
  2. fileinfo, err := os.Stat(file)
  3. ...

根据 stat 调用的结果,决定是否需要启动一个 harvester,harvester 是这个Go应用程序中读取文件的部分。

  1. // Conditions for starting a new harvester:
  2. // - file path hasn't been seen before
  3. // - the file's inode or device changed
  4. if !isKnown {
  5. p.checkNewFile(h)
  6. } else {
  7. h.Stat.Continue(&lastinfo)
  8. p.checkExistingFile(h, &newFile, &oldFile)
  9. }
  10. // Track the stat data for this file for later comparison to check for
  11. // rotation/etc
  12. p.prospectorList[h.Path] = *h.Stat
  13. }

简而言之,Filebeat使用操作系统报告的文件统计信息来判断文件是否自上次收集文件以来已更新。

英文:

Since filebeat is open source, you can always go look yourself

Here's the go code from the above linked file which checks if a file has been updated.

I've heavily abridged this code, anywhere you see ... is a code block that was not exactly relevant, I encourage anybody reading this too go look at the entire file, its some pretty well written go.

  1. // Scan starts a scanGlob for each provided path/glob
  2. func (p *ProspectorLog) scan() {
  3. newlastscan := time.Now()
  4. // Now let&#39;s do one quick scan to pick up new files
  5. for _, path := range p.config.Paths {
  6. p.scanGlob(path)
  7. }
  8. p.lastscan = newlastscan
  9. }

The above function gets called every n-length time block where n is specified in the configuration. ScanGlob gets called, and is shown below.

  1. // Scans the specific path which can be a glob (/**/**/*.log)
  2. // For all found files it is checked if a harvester should be started
  3. func (p *ProspectorLog) scanGlob(glob string) {
  4. ...
  5. // Evaluate the path as a wildcards/shell glob
  6. matches, err := filepath.Glob(glob)
  7. ...
  8. // Check any matched files to see if we need to start a harvester
  9. for _, file := range matches {
  10. ...

For all files which matched the glob, check the statistics on the file using the OS specific call, for linux this would be stat &lt;file&gt;

  1. // Stat the file, following any symlinks.
  2. fileinfo, err := os.Stat(file)
  3. ...

Based on the stat call, it is decided if a harvester, the part of this go application which reads the files, needs to be started.

  1. // Conditions for starting a new harvester:
  2. // - file path hasn&#39;t been seen before
  3. // - the file&#39;s inode or device changed
  4. if !isKnown {
  5. p.checkNewFile(h)
  6. } else {
  7. h.Stat.Continue(&amp;lastinfo)
  8. p.checkExistingFile(h, &amp;newFile, &amp;oldFile)
  9. }
  10. // Track the stat data for this file for later comparison to check for
  11. // rotation/etc
  12. p.prospectorList[h.Path] = *h.Stat
  13. }
  14. }

TL;DR: Filebeat used the files statistics reported by the OS to see if the file has been updated since the last time it harvested the file.

huangapple
  • 本文由 发表于 2016年4月18日 06:01:15
  • 转载请务必保留本文链接:https://go.coder-hub.com/36682714.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定