How to register pandas dataframe as parquet or csv dataset in the container and in the Data at the same time?

I'm using Azure ML. As a part of the project I want to register output pandas dataframe as a file and dataset in the Data section. Preferably I want it to exist in the container which is created for AML (in csv or parquet extension) as well as being available in the Data section in one go by executing function.

So far, code for my function looks the following:

def register_future_predictions(forecast_val, ws):

    global last_date
    last_date = forecast_val['ds'].iloc[-1]
    global future_dates
    future_dates = pd.date_range(start=last_date, periods=182, freq='W')
    global future_df
    future_df = pd.DataFrame({'ds': future_dates})

    for col in X_val.columns:
        if col != 'ds':
            future_df[col] = 0

    global future_predictions   
    future_predictions = results.predict(future_df)

    reg_data_future = pd.concat([X_train, X_test, X_val, future_predictions[['ds', 'yhat']].rename({'ds': 'Created on day', 'yhat': 'target_col'})])
    target_datastore = Datastore.get(ws, 'container-where-i-keep-datasets')
    ds = Dataset.Tabular.register_pandas_dataframe(dataframe=reg_data_future, name='full_data-TEST', target=target_datastore)

    return future_predictions, future_df

Obviously this doesn't work. Maybe you have some propositions or working solution to refer?

Thank you in advance.

Based on the scenario I have taken a sample data X_train, X_test, X_val and results.

To register a Pandas dataframe as a Parquet or CSV dataset in the container and in the Data section of Azure ML, you can use the Dataset.Tabular.from_delimited_files function:

import  pandas  as  pd
from  azureml.core  import  Workspace, Datastore, Dataset

# Sample test data
X_train = pd.DataFrame({'ds': ['2023-01-01', '2023-01-02', '2023-01-03'], 'y': [1, 2, 3]})
X_test = pd.DataFrame({'ds': ['2023-01-04', '2023-01-05'], 'y': [4, 5]})
X_val = pd.DataFrame({'ds': ['2023-01-06', '2021-03-07'], 'y': [6, 7]})
results = pd.DataFrame({'ds': ['2023-01-08', '2023-01-09'], 'yhat': [8, 9]})

Below function execute both csv and parquet file. You can modify this according to your need.

def  register_future_predictions(forecast_val, ws):


    last_date = forecast_val['ds'].iloc[-1]
    future_dates = pd.date_range(start=last_date, periods=182, freq='W')
    future_df = pd.DataFrame({'ds': future_dates})

    for  col  in  X_val.columns:
        if  col != 'ds':
            future_df[col] = 0

    future_predictions = results
    reg_data_future = pd.concat([X_train, X_test, X_val, 
    future_predictions[['ds', 'yhat']].rename(columns={'ds': 'Created on day', 'yhat': 'target_col'})])

    # Save the dataframe as a file, e.g., CSV or Parquet
    reg_data_future.to_csv('reg_data_future.csv', index=False)
    reg_data_future.to_parquet('reg_data_future.parquet', index=False)

    # Get the target datastore
    target_datastore = Datastore.get(ws, 'workspaceblobstore')

    # Upload the files to the datastore's container
    target_datastore.upload_files(files=['reg_data_future.csv', 'reg_data_future.parquet'], target_path='data')

    # Register the files as datasets
    csv_dataset = Dataset.Tabular.from_delimited_files(path=[(target_datastore, 'data/reg_data_future.csv')])

    csv_dataset.register(workspace=ws, name='full_data_csv', create_new_version=True)

    parquet_dataset = Dataset.Tabular.from_parquet_files(path=[(target_datastore, 'data/reg_data_future.parquet')])

    parquet_dataset.register(workspace=ws, name='full_data_parquet', create_new_version=True)

    return  future_predictions, future_df

# Create or load Azure ML workspace
ws = Workspace.from_config()

# Test the function
sample_forecast_val = pd.DataFrame({'ds': ['2024-01-01', '2024-01-02', '2024-01-03']})

predictions, df = register_future_predictions(sample_forecast_val, ws)

With the updated function will successfully execute.

.

Create the csv or parquet file in container.

And register the dataset as data asset.

Please refer to this documentation for more details and examples.