Python unable to write 3 million records to .xpt sas dataset

i have a pandas dataframe with 3 million records but not able to write large dataframe to .xpt files. Although the same script works for other data frames having 2 million records without modifying anything.

While writing I am using df.from_dataframe() function and while writing to the xpt file suddenly after couple of minutes i get "KILLED" message on the console.

• System - RHEL Linux
• Python - 3.7
• Pandas version - 1.2
• Using xport module with .v56 subpackage while writing.

Need guidance in below areas

• Is this a case of memory leak
• any chances of bad data in dataframe

What is the correct approach in debugging a single built-in function?

Not sure about the functionality of writing pandas dataframe to sas datasets in .xpt format.

A related error had been reported on the GitHub page of xport and got resolved in a version of xport which is, unfortunately, not compatible with pandas==1.2.4

I ran the code below with pandas==1.2.4 & xport==3.2.1 (highest version of xport compatible with this pandas version), which gave me the following error: NotImplementedError: Can't copy SAS variable metadata to dataframe

Once upgrading to pandas==1.3.5 & xport==3.6.1, the code worked like a charm. Here's the good news: pandas==1.3.5 still works on Python 3.7.1 and above according to its official doc

Code (example code from xport documentation with some adaptations):

import pandas as pd
import xport
import xport.v56

# create list with 3 million items each
alpha_list = [10, 20, 30] * 1_000_000
beta_list = ['x', 'y', 'z'] * 1_000_000

df = pd.DataFrame({
    'alpha': alpha_list,
    'beta': beta_list
})


ds = xport.Dataset(df, name='DATA', label='Wonderful data')

# Variable names are limited to 8 characters in SAS transport
# files (.xpt). In SAS datasets, variable names may be up to 
# 32 characters  As with Pandas dataframes, you must change the
# name on the dataset rather than the column directly.
ds = ds.rename(columns={k: k.upper()[:8] for k in ds})

# Other SAS metadata can be set on the columns themselves.
for k, v in ds.items():
    v.label = k.title()
    if v.dtype == 'object':
        v.format = '$CHAR20.'
    else:
        v.format = '10.2'

# Libraries can have multiple datasets.
library = xport.Library({'DATA': ds})

with open('example.xpt', 'wb') as f:
    xport.v56.dump(library, f)
    
# load exported file to check if everything's working
df = pd.read_sas('example.xpt')

print(df.head(n=10).to_markdown(index=False))

prints:

|   ALPHA | BETA   |
|--------:|:-------|
|      10 | x      |
|      20 | y      |
|      30 | z      |
|      10 | x      |
|      20 | y      |
|      30 | z      |
|      10 | x      |
|      20 | y      |
|      30 | z      |
|      10 | x      |