How to read parquet files from Azure Blobs into Pandas DataFrame with columns projection on server-side?

Following this question: https://stackoverflow.com/questions/63351478/how-to-read-parquet-files-from-azure-blobs-into-pandas-dataframe

Is it possible to perform a column projection on the parquet file at server level before downloading it to be more efficient? I.e. I would like to filter only desidered columns before downloading the file.

At the moment I am connecting to Azure services only by a connection string if that helps and using the Python Client library.

> Is it possible to perform a column projection on the parquet file at server level before downloading it to be more efficient? I.e. I would like to filter only desired columns before downloading the file.

To download the desired column from the parquet file in Azure blob storage, you can use the below Python code:

Code:

import pyarrow.parquet as pq
from azure.storage.blob import BlobServiceClient
import pandas as pd 

# Set up the BlobServiceClient
blob_service_client = BlobServiceClient.from_connection_string('your connection string')

# Get a reference to the Parquet file in Azure Blob Storage
blob_container_client = blob_service_client.get_container_client('test1')
blob_client = blob_container_client.get_blob_client('samplepar.parquet')

# Define the list of columns to read from the Parquet file
columns = ['title', 'salary', 'birthdate', 'id1', 'id2']
columns_query = ", ".join([f"[{column}]" for column in columns])
query = f"SELECT {columns_query} FROM BlobStorage"
with open("sample.parquet1", "wb") as file:
    blob_client.download_blob().download_to_stream(file)

table = pq.read_table("sample1.parquet")
available_columns = [column for column in columns if column in table.column_names]
print(available_columns)
if available_columns:
    table = table.select(available_columns)
    df = table.to_pandas()
    print(df)
else:
    print("Error: None of the specified columns are present in the Parquet file.")

Output:

['title', 'salary', 'birthdate']
                      title     salary  birthdate
0          Internal Auditor   49756.53   3/8/1971
1             Accountant IV  150280.17  1/16/1968
2       Structural Engineer  144972.51   2/1/1960
3    Senior Cost Accountant   90263.05   4/8/1997

Downloaded File: