How to encrypt pandas Dataframe with pyarrow and parquet

I would like to encrypt pandas dataframe as parquet file using the modular encryption. I tought the best way to do that, is to transform the dataframe to the pyarrow format and then save it to parquet with a ModularEncryption option. Something like this:

import pandas as pd
d = {'col1': [1, 2], 'col2': [3, 4]}
df = pd.DataFrame(data=d)
import pyarrow as pa
schema = pa.Schema.from_pandas(df)
pa.parquet.write_table(df,"test.parquet",encryption_properties=enc_prop)

My problem is, that I'm stuck with the encrypton_properties creation.
Has anyone a idea how to create them?

Big Thanks,
Seb

There is an example python file in Apache Arrow repo with

> An example for writing an encrypted parquet and reading an encrypted
> parquet using master keys managed by Hashicorp Vault KMS.

More info:

• Parquet Modular Encryption (Columnar Encryption) general docs: https://arrow.apache.org/docs/python/parquet.html#parquet-modular-encryption-columnar-encryption
• A test for writing encrypted file: https://github.com/apache/arrow/blob/45918a90a6ca1cf3fd67c256a7d6a240249e555a/python/pyarrow/tests/parquet/test_encryption.py#L68-L98

Hope that helps.

I am able to encrypt and decrypt parquet files as

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.parquet.encryption as pe
from pyarrow.tests.parquet.encryption import InMemoryKmsClient

# pandas dataframe

d = {'col1': [1, 2], 'col2': [3, 4], 'col3': [5, 6]}
df = pd.DataFrame(data=d)
table = pa.Table.from_pandas(df)

# -------------------------------------------------
# encryption

# secret 128-bit AES key
private_key = b"1234567890123450"

encryption_config = pe.EncryptionConfiguration(
    footer_key="footer",
    column_keys={
       "columns": df.columns.tolist(),
    },
    encryption_algorithm="AES_GCM_V1",
    data_key_length_bits=128)

kms_connection_config = pe.KmsConnectionConfig(
    custom_kms_conf={
        "footer": private_key.decode("UTF-8"),
        "columns": private_key.decode("UTF-8"),
    }
)

def kms_factory(kms_connection_configuration):
    return InMemoryKmsClient(kms_connection_configuration)

crypto_factory = pe.CryptoFactory(kms_factory)

encryption_properties = (
    crypto_factory.file_encryption_properties(
        kms_connection_config,
        encryption_config
    )
)

with pq.ParquetWriter(
    'encrypted_table.parquet',
    table.schema,
    encryption_properties=encryption_properties
) as writer:
    writer.write_table(table)

# -------------------------------------------------
# decryption

decryption_properties = (
    crypto_factory.file_decryption_properties(
        kms_connection_config
    )
)

parquet_file = pq.ParquetFile(
    'encrypted_table.parquet',
    decryption_properties=decryption_properties
)

print(parquet_file.read().to_pandas())

• This is going to encrypt all columns. If you want to exclude some, just mention you want to include in encryption_config.column_keys["columns"]
• Different keys can be used to encrypt dataframe body and footer. I just same private key for them both.
• Both 128-bit and 256-bit AES are working.