How does Neo4j work with large databases?

Imagine there is a large database (>50GB) and because of its size it cannot be loaded entirely in the main memory. So how does Neo4j work with such databases? Is there some kind of paging? If so how does it work, if not - how does it load parts of the database in memory? I want to understand how it works at a low level.

Here are a couple of old knowledge base articles that provide some internal information on how neo4j uses:

• memory (this may also be helpful), and
• storage.

The details have probably changed over time, but the above should at least give you a feel for some of the neo4j internals.

Different answer for differing use cases: loading, storage or retrieving data?

For loading you can use CALL apoc.periodic.iterate. You can set the size of each iteration of a file which is not in memory. This also has a recovery capability if the connection is interrupted.

The schema is particularly important in storing large data sets. The initial load can be followed by analytics which are memorialized in the knowledge graph (read up on that subject). This involves adding new nodes, relationships or properties which enhance performance. While it involved some redundancy and extra storage, the improved efficiency justifies this.

For data retrieval, consider subqueries using unwind-call features which iterate through the initial query result to pull out tranches of data sequentially in the subquery. You can write the results to a file and not use memory for the results.