Why toPandas is bad?

According to the documentation, PySpark's toPandas method is not designed to be used with large datasets. Even with Arrow enabled.

> This method should only be used if the resulting Pandas pandas.DataFrame is expected to be small, as all the data is loaded into the driver’s memory.

Nevertheless, many of us use it to transfer relatively large dataframes (I personally transferred 5Gb once).

1. Why toPandas is actually bad for large data?

2. After what threshold can it be considered that a dataframe is small enough to be transferred via toPandas

3. What is the most suitable approach to transfer a dataframe to the local python? df.repartition(1).write.csv followed by hdfs dfs -get?

Upd. I consider datasets that can comfortly fit to driver's node memory

1. Because it will occupy the entire data in your RAM. If RAM gets nearly full, the data will be off-loaded to the swap memory on the hard drive, which will slow down the operation significantly. This is especially the case, if the CPU need to access data that was moved to the swap previously, because it needs to put it back in the RAM moving something else into the swap memory. If RAM and swap limit will both be exceeded, you will face an out of memory error and the operation will cancel.

2. The Threshold really depends on the available RAM on the host machine, but it is generally recommend to stay way below possible limits to ensure a fast operation when the server is in high load.

3. It's generally better to stream large data to avoid this issue. Often, you can't know how much RAM and swap will be available on the host machine. Streaming is safer and faster for large data. This article has four excellent strategies and I'd like to quote two of them here:

> Sampling: The most simple option is sampling your dataset. This approach can be especially powerful during the exploration phase: how does the data look like? What features can I create? In other words, what works and what does not. Often a random sample of 10% of such a large dataset will already contain a lot of information. That raises the first question, do you actually need to process your entire dataset to train an adequate model?
>
> py
> import pandas
> import random
>
> filename = "data.csv"
> n = sum(1 for line in open(filename))-1 # Calculate number of rows > in file
> s = n//10 # sample size of 10%
> skip = sorted(random.sample(range(1, n+1), n-s)) # n+1 to compensate for header
> df = pandas.read_csv(filename, skiprows=skip)
>
>
> Chunking: If you do need to process all data, you can choose to split the data into a number of chunks (which in itself do fit in memory) and perform your data cleaning and feature engineering on each individual chunk. Moreover, depending on the type of model you want to use, you have two options:
>
> - If the model of your choosing allows for partial fitting, you can incrementally train a model on the data of each chunk;
> - Train a model on each individual chunk. Subsequently, to score new unseen data, make a prediction with each model and take the average or majority vote as the final prediction.
>
> py
> import pandas
> from sklearn.linear_model import LogisticRegression
> datafile = "data.csv"
> chunksize = 100000
> models = []
> for chunk in pd.read_csv(datafile, chunksize=chunksize):
> chunk = pre_process_and_feature_engineer(chunk)
> # A function to clean my data and create my features
> model = LogisticRegression()
> model.fit(chunk[features], chunk['label'])
> models.append(model)
> df = pd.read_csv("data_to_score.csv")
> df = pre_process_and_feature_engineer(df)
> predictions = mean([model.predict(df[features]) for model in models], axis=0)
>