英文:
When does O_SYNC have an effect?
问题
我正在研究我在Mender中遇到的问题，即安装进度（关于在块设备上复制文件）没有正确报告。
我觉得这可能与内核页面缓存有关：进度条在代码读取整个镜像后显示为“100%”，但这并不意味着内核已经完成写入。
具体来说，Mender调用n, err := io.Copy(dev, image)，该函数在内核完成写入后返回。但是进度条与“image”Reader相关联，该Reader在io.Copy返回之前就已经完全读取了数十秒。
因为文件在这里使用了标志这里，我天真地认为我只需要设置flag |= O_SYNC，这样io.Copy(dev, image)就不会比写入dev更快地读取image。
但是设置O_SYNC没有任何区别。
我不清楚O_SYNC是否只是一个提示（所以我不能依赖它），是否可能是我在设备上漏掉了某些东西（比如说，我在树莓派上忘记了一个内核选项，因此O_SYNC是无用的），还是我对O_SYNC的理解有误？
编辑：我还尝试设置O_SYNC | O_DIRECT（尽管Go中显然没有暴露O_DIRECT，所以我使用了O_SYNC | 0o40000），但是在打开块设备时出现了以下错误：
Opening device: /dev/mmcblk0p2 for writing with flag: 1069058
Failed to open the device: "/dev/mmcblk0p2": open /dev/mmcblk0p2: not a directory
英文:
I was looking into an issue I had with Mender, where the installation progress (which is about copying a file on a block device) is not reported correctly.
My feeling is that it's about the kernel page cache: the progress bar says "100%" when the code has read the whole image, but that does not mean that the kernel is done writing it.
More specifically, Mender calls n, err := io.Copy(dev, image), which returns after the kernel is done writing. But the progress bar is linked to the "image" Reader, which is fully read tens of seconds before io.Copy returns.
Because the file is opened with flags here, I naively thought that I just had to set flag |= O_SYNC, so that io.Copy(dev, image) would not read image faster than it writes to dev.
But setting O_SYNC does not make a difference.
It is not clear to me if O_SYNC is merely a hint (so I cannot count on it), if it could be that I am missing something on my device (say, I forgot a kernel option on my Raspberry Pi and therefore O_SYNC is useless), or if I just misunderstood what O_SYNC does?
EDIT: I also tried to set O_SYNC | O_DIRECT (though O_DIRECT is apparently not exposed in Go and so I did O_SYNC | 0o40000), but I got the following error when opening the block device:
Opening device: /dev/mmcblk0p2 for writing with flag: 1069058
Failed to open the device: "/dev/mmcblk0p2": open /dev/mmcblk0p2: not a directory
答案1
得分: 1
总结评论：
-
主要问题是进度条装饰的是读者（正如Yotam Salmon指出的），而不是写者；延迟出现在写者一侧。
-
在大多数Linux系统上，
O_DIRECT确实是0o40000，但在ARM（包括树莓派）上是0o200000，而0o40000是O_DIRECTORY。这解释了“不是目录”的错误。 -
O_SYNC实际上是你想要的位，或者你可以简单地发出一个fsync系统调用（如果适用，使用Flush，然后使用Sync，如https://stackoverflow.com/q/10862375/1256452中所述）。O_SYNC位意味着每个write系统调用都会包含一个fsync系统调用。
完全同步的I/O有点棘手：有些设备会说谎，它们是否将数据写入非易失性存储。然而，O_SYNC或fsync是你在这里能得到的最大保证。O_DIRECT可能与你直接访问设备分区的/dev文件无关。O_SYNC或fsync 可能会传递给设备驱动程序，该驱动程序可能会对其进行处理，从而可能使设备写入非易失性存储。关于这方面的更多信息，请参阅https://stackoverflow.com/q/41257656/1256452。
英文:
Summarizing the comments:
-
The main issue is that the progress bar is decorating the reader (as Yotam Salmon noted), not the writer; the delay is on the side of the writer.
-
On most Linux systems,
O_DIRECTis indeed0o40000, but on ARM (including Raspberry Pi) it is0o200000, with0o40000beingO_DIRECTORY. This explains the "not a directory" error. -
O_SYNCis in fact the bit you want, or you can simply issue anfsyncsystem call (useFlushif appropriate, and thenSync, as noted in https://stackoverflow.com/q/10862375/1256452). TheO_SYNCbit implies anfsyncsystem call as part of eachwritesystem call.
Fully synchronous I/O is a bit of a minefield: some devices lie about whether they've written data to nonvolatile storage. However, O_SYNC or fsync is the most guarantee you'll get here. O_DIRECT is likely irrelevant since you're going directly to a device partition /dev file. O_SYNC or fsync may be passed through to the device driver, which may do something with it, which may get the device to write to nonvolatile storage. There's more about this in https://stackoverflow.com/q/41257656/1256452
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论