Unable to create/save/load very large array on disk

For learning purposes I want to create ,save and then load(into tensorflow.keras) a very large int array of lengths in the order of 10^10.

initially tried numpy but failed in the creation stage.

Example code:

x=np.ones((274576,200,200,1),dtype='int')

Then i used dask.array,succeeded in creating the array but saving it in hdf5 format itself takes lots of time and large memory .

Is there any alternates with better speed and space efficiency?

Example code:

import dask.array as da
x=da.ones((274576,200,200,1),dtype='float')#create
da.to_hdf5('x.hdf5',{'x':x})#save
import h5py
y=h5py.File('x.hdf5','r')['x'][:]#read
print(y)

Also in this dask array format fitting them directly into model.fit gives out a warning.

Is there any conversion step that needs to be done to avoid this?

Warning message:

WARNING:tensorflow:Keras is training/fitting/evaluating on array-like data. Keras may not be optimized for this format, so if your input data format is supported by TensorFlow I/O (https://github.com/tensorflow/io) we recommend using that to load a Dataset instead.

You really have 2 questions: 1) about HDF5 storage and 2) about reading the file with tensorflow model.fit(). My answer addresses the 1st question.

When creating large datasets with HDF5, it's important to use chunked storage and set the chunks parameter appropriately. It's not clear from your post what values dask sets when you use da.to_hdf5().
I was able to reduce file creation time to 6 mins for your example using h5py and chunked storage. Code at the end.

By default, HDF5 dataset storage is contiguous. When you enable chunked storage, data is stored and accessed in fixed-size chunks. Chunking has performance implications. The recommended chunk size should be between 10 KiB and 1 MiB (larger for larger datasets). Also, when any element in a chunk is accessed, the entire chunk is read from disk. So, the"best" chuck shape depends on how you will access the data.

Note: this line y=h5py.File('x.hdf5','r')['x'][:] will read the entire dataset into a NumPy array (all 42 Gib of it!). I'm surprised it works. It's better to create a h5py dataset object like this: x_ds=h5py.File('x.hdf5','r')['x']. For many operations, the object behaves like an array. When you need an array object, you can use NumPy slice notation to read the desired data (for example: x_ds[0:5,:,:,:]).

Here is a method to create a HDF5 file with a dataset of shape=(274_576,200,200,1). Note that it uses chunked storage, and data is added in blocks that match the chunk size. The appropriate chunks parameter depends on how you plan access the data. I set it assuming you will read "rows" of data (on axis 0). Modify my values to benchmark how the resulting file behaves.

chunksize = (c0,c1,c2,c3) = (4,200,200,1)
nrows = 274_576
start = time.time()

with h5py.File('SO_76299271.h5','w') as h5f:
    h5f.create_dataset('x',shape=(nrows,200,200,1),dtype='int', chunks=chunksize) # creates empty dataset of ints
    
    for i in range(nrows//c0):
        arr = np.random.randint(0, 255, size=chunksize)
        h5f['x'][i*c0:(i*c0+c0)] = arr

with h5py.File('SO_76299271.h5','r') as h5f:
    x_ds = h5f['x'] # creates a h5py dataset object