Go: Best way to handle excessive memory application? Mmap, memory or caching?

I have a Go application which requires around 600GB of memory. The machine on which is will run has 128GB of RAM. I'm trying to decide how best to handle this.

The options are:

1. Just load everything into the memory (pretend like I have 600GB RAM) and let the OS page out the infrequently accessed part of the memory into virtual memory. I like this idea because I don't have to do anything special in the code, the OS will just handle everything. However, I'm not sure this is a good idea.

2. Have the data stored on disk and use mmap (memory mapped file) which I guess is similar to the above but will require a lot more coding. Also it appears to mean that the data will have to be stored as []byte and then parsed every time I need to use it, rather that being already in whatever type I need it for the actual calculations.

3. Build a caching system in which the data is kept on HDD and then loaded it when it's needed, with the most frequently accessed data being held in memory and the least frequently accessed data being purged whenever the memory limited is exceeded.

What are the advantages and disadvantages with these? I'd prefer to go with (1) if possible due to its simplicity... is there anything wrong with that?

It all depends on the nature of the data access. Will the accesses to those 600GB be uniformly distributed? If that's not the case then a solution where you cache part of your content in memory and keep the rest of it on the HDD will likely be sufficient since you have enough RAM to cache more than 20% of your data. Keeping everything in virtual memory space may come with surprising drawbacks such as the need for a huge swap partition.

To cache the data on disk you could use a DB engine as Dave suggests since they usually do a good job of caching the most frequently accessed content. You could also use memcached, a library and client for caching stuff in memory.

The bottom line is that optimizing performance without knowing the exact usage patterns is hard. Luckily, with Go you don't have to guess. You can test and measure.

You can define an interface similar to

type Index interface{
    Lookup(query string) Result
}

And then try all of your solutions, starting with the easiest to implement.

type inMemoryIndex struct {...}

func (*inMemoryIndex) Lookup(query string) Result {...}

type memcachedIndex struct {...}

type dbIndex struct {...}

Then you can use Go's builtin benchmarking tools to benchmark your application and see if it lives up to your standards. You can even benchmark on that machine, using real data and mocked user queries.

You're correct to assume that mmap would require more coding so I would have saved that until I had tried all other options.