Dataframe: Permutations after grouping column data into a dictionary/list with recurring elements

I'm struggling with manipulating large sets of data into condensed rows that will allow me to run permutations on and get all possible combinations.
Basically this is the structure:

	PLOT	SEED	SUB	EDGE	PT	0	1	2	3
1	117	1	1	1	2	8	None	None	None
2	117	1	1	1	3	8	None	None	None
3	117	1	1	2	1	8	None	None	None
4	117	1	1	2	2	6	7	8	None
5	117	1	2	1	1	8	None	None	None

I'm not sure if I should use lists or dictionaries, as this data should have thousands to even millions of rows in the future.

The idea is to add the ['SUB','EDGE','PT'] values to each value in the [0:] columns (it won't necessarily end at 3), so that for example row number 4 will look like:

	PLOT	SEED	SUB	TYPES
4	117	1	1	[1,2,2,6],[1,2,2,7],[1,2,2,8]

OR dictionaries:

	PLOT	SEED	SUB	TYPES
4	117	1	1	{'SUB':1, 'EDGE':2, 'PT':2, 'TYPE':6},{'SUB':1, 'EDGE':2, 'PT':2, 'TYPE':7},{'SUB':1, 'EDGE':2, 'PT':2, 'TYPE':8}

And then run permutations on each ['PLOT','SEED'] using the ['SUB'] column.
That means that if Plot 117, Seed 1 has 2 Subs, it will run all possible combinations between all the options in Sub1 and all the options in Sub1, while keeping the ['PLOT','SEED'] data. So in this case (building on the original 5 row table up top):

# 1st Sub:
[1,1,2,8],[1,1,3,8],[1,2,1,8],[1,2,2,6],[1,2,2,7],[1,2,2,8]
# 2nd Sub:
[2,1,1,8]

And What I should get is:

#Plot 117 Seed 1 Permutations:
[[1,1,2,8],[2,1,1,8]], [[1,1,3,8],[2,1,1,8]], [[1,2,1,8],[2,1,1,8]], ...
Or with Dictionaries:
[{'SUB':1, 'EDGE':2, 'PT':2, 'TYPE':6},{'SUB':2, 'EDGE':1, 'PT':1, 'TYPE':8}], ...

Obviously this needs to iterate over the entire dataset for each ['PLOT','SEED'], of which there will be many, and also in many cases there will be more than 2 'SUB' per PLOT+SEED.

I tried many things like 'groupby', for example:

df_grouped_edge = df_joined.groupby(['PLOT','SEED','SUB','EDGE'], as_index=False).agg(np.array)
df_grouped_edge

For which I get:

	PLOT	SEED	SUB	EDGE	PT	0	1	2	3
0	117	1	1	1	[2, 3]	[8, 8]	[None, None]	[None, None]	[None, None]
1	117	1	1	2	[1, 2, 3]	[8, 6, 8]	[None, 7, None]	[None, 8, None]	[None, None, None]

This is not what I'm looking for, or at least I don't think it is and I don't know how to work with that output. Tried many other grouping attempts the last two days but I can't seem to get it work, and I understand that looping over large datasets per item/column is a very wrong way of working (and I should vectorize it, I just don't understand how yet).

I hope I explained myself well, I know this is a big question but I'm really at a loss.
Starting to learn SQL very soon but I'm not there yet, and I have to get this done.

Thank you very much!
Lev

First, group your data by the columns ['PLOT', 'SEED']. This will allow you to iterate over each unique combination of ['PLOT', 'SEED'] in your dataset.

grouped_data = df.groupby(['PLOT', 'SEED'])

Define a function that takes a group and performs the necessary operations to generate the combinations. This function will be applied to each group using apply().

import itertools

def generate_combinations(group):
    sub_values = group['SUB'].unique()
    combinations = list(itertools.product(*sub_values))
    
    # Create a new DataFrame to store the combinations
    result_df = pd.DataFrame(columns=df.columns)
    
    for combination in combinations:
        # Create a new row with the combination values
        new_row = group.iloc[0].copy()
        new_row[['SUB', 'EDGE', 'PT']] = combination
        
        # Append the row to the result DataFrame
        result_df = result_df.append(new_row, ignore_index=True)
    
    return result_df

Apply the generate_combinations function to each group in the grouped data and concatenate the results.

result = pd.concat([generate_combinations(group) for _, group in grouped_data], ignore_index=True)

Now, the result should contain the DataFrame with all the possible combinations for each unique ['PLOT', 'SEED'] combination, based on the values in the 'SUB' column.

I ended up solving it using .melt():

df.melt(id_vars=["PLOT", "SEED", "SUB", "EDGE", "PT"], 
        var_name="DROP", 
        value_name="TYPO",
        ignore_index=True) #False here leaves duplicates of the original indexes
df=df.drop('DROP', axis=1)
# Drop Scenarios with no typos matching
df=df.dropna()

After getting each typo value as a row, I used .groupby(['PLOT', 'SEED'] into dictionaries, then iterated over each entry in the dict and used groupby('SUB'... everything in place.

Then in each dict in a large list I only used the key that has the list of lists (with all the options) and I used itertools.product to get all the combinations between said lists.

Obviously this required a lot of work on the data structures, adding back the corresponding data into each dict, probably not the cleanest work, but it did the job.

My problem now is that .product stopped working due to MemoryError when the dataset became too big:
https://stackoverflow.com/questions/76559405/using-np-meshgrid-on-list-of-lists-with-different-lengths-in-a-large-dataset-in