How do I skip the filesystem cache when reading a file in Golang?

Assume that the contents of the file Foo.txt are as follows.

Foo Bar Bar Foo

Consider the following short program.

package main

import "syscall"
import "fmt"


func main() {
    fd, err := syscall.Open("Foo.txt", syscall.O_RDONLY, 0)
    if err != nil {
        fmt.Println("Failed on open: ", err)
    }
    data := make([]byte, 100)
    _, err = syscall.Read(fd, data)
    if err != nil {
        fmt.Println("Failed on read: ", err)
    }
    syscall.Close(fd)
}

When we run the program above, we get no errors, which is correct behavior.

Now, I modify the syscall.Open line to be the following.

fd, err := syscall.Open("Foo.txt", syscall.O_RDONLY | syscall.O_SYNC | syscall.O_DIRECT, 0)

When I run the program again, I get the following (undesirable) output.

Failed on read:  invalid argument

How can I correctly pass the flags syscall.O_SYNC and syscall.O_DIRECT as specified by the the open man page for skipping the filesystem cache?

Note that I am using the syscall file interface directly instead of the os file interface because I could not find a way to pass those flags into the functions provided by os, but I am open to solutions that use os provided that they work correctly to disable the filesystem cache on reads.

Note also that I am running on Ubuntu 14.04 with ext4 as my filesystem.

Update: I tried to use @Nick Craig-Wood's package in the code below.

package main

import "io"
import "github.com/ncw/directio" 
import "os"
import "fmt"


func main() {
    in, err := directio.OpenFile("Foo.txt", os.O_RDONLY, 0666)
    if err != nil {
        fmt.Println("Error on open: ", err)
    }

    block := directio.AlignedBlock(directio.BlockSize)
    _, err = io.ReadFull(in, block)
    if err != nil {
        fmt.Println("Error on read: ", err)
    }
}

The output is the following

Error on read:  unexpected EOF

You may enjoy my directio package which I made for exactly this purpose.

From the site

This is library for the Go language to enable use of Direct IO under all supported OSes of Go (except openbsd and plan9).

Direct IO does IO to and from disk without buffering data in the OS. It is useful when you are reading or writing lots of data you don't want to fill the OS cache up with.

See here for package docs

http://go.pkgdoc.org/github.com/ncw/directio

From the open man page, under NOTES:

> The O_DIRECT flag may impose alignment restrictions on the length and address of user-space buffers and the file offset of I/Os. In Linux alignment restrictions vary by file system and kernel version and might be absent entirely.

So you could have alignment issues, of either the memory or the file offset, or your buffer size could be "wrong". What the alignments and sizes should be is not obvious. The man page continues:

> However there is currently no file system-independent interface for an application to discover these restrictions for a given file or file system.

And even Linus weighs in, in his usual understated manner:

> "The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances." —Linus

Good luck!

p.s. Stab in the dark: why not read 512 bytes?

you can try to use fadvice and madvice, but there is no guarantee. both will work more probably with larger files/data, because:

> Partial pages are deliberately preserved on the expectation that it is better to preserve needed memory than to discard unneeded memory.

see the linux source code, what will do something and what not. POSIX_FADV_NOREUSE for example doesn't do anything.

http://lxr.free-electrons.com/source/mm/fadvise.c#L62

http://lxr.free-electrons.com/source/mm/madvise.c

package main

import "fmt"
import "os"
import "syscall"

import "golang.org/x/sys/unix"

func main() {
    advise := false
    if len(os.Args) > 1 && os.Args[1] == "-x" {
        fmt.Println("setting file advise")
        advise =true
    }

    data := make([]byte, 100)
    handler, err := os.Open("Foo.txt")
    if err != nil {
        fmt.Println("Failed on open: ", err)
    }; defer handler.Close()

    if advise {
        unix.Fadvise(int(handler.Fd()), 0, 0, 4) // 4 == POSIX_FADV_DONTNEED
    }

    read, err := handler.Read(data)
    if err != nil {
        fmt.Println("Failed on read: ", err)
        os.Exit(1)
    }

    if advise {
        syscall.Madvise(data, 4) // 4 == MADV_DONTNEED
    }

    fmt.Printf("read %v bytes\n", read)
}

/usr/bin/time -v ./direct -x

Command being timed: "./direct -x"
User time (seconds): 0.00
System time (seconds): 0.00
Percent of CPU this job got: 0%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.03
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 1832
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 2
Minor (reclaiming a frame) page faults: 149
Voluntary context switches: 2
Involuntary context switches: 2
Swaps: 0
File system inputs: 200
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0