Efficient ways of handling dataframes with columns that are mutually exclusive?

As an example imagine I have a csv below

id | a1 | a2 |....| aN | b1 | b2 |....| bM |
___________________________________________
0  |data|data|....|data|None|None|....|None|
1  |data|data|....|data|None|None|....|None|
2  |data|data|....|data|None|None|....|None|
3  |None|None|....|None|data|data|....|data|
4  |None|None|....|None|data|data|....|data|
....

I have N a columns and M b columns, and the a columns and b columns are mutually exclusive i.e. if I have data in a then I wouldn't have anything in b. data in this case is mostly string or float values.

It feels inefficient that I will have M or N elements for each row that don't contain anything.

I can split the above into two different dataframes i.e.

df_a
id | a1 | a2 |....| aN |
________________________
0  |data|data|....|data|
1  |data|data|....|data|
2  |data|data|....|data|


df_b
id | b1 | b2 |....| bM |
________________________
3  |data|data|....|data|
4  |data|data|....|data|

But then I have two dataframes that I have to keep track of instead of one dataframe.

What is the most efficient way of keeping the data together without creating a bloated dataframe? Would the solution work if I have c and d columns as well?

One thing I can do is make the csv into an excel sheet and put the b columns in a different sheet. But it's still a bit clunky for me.

A possible solution is to use Sparse data structures (it does not store None values):

txt = """
id    a1    a2    a3     b1     b2     b3
0  10.0  20.0  30.0   None   None   None
1  40.0  50.0  60.0   None   None   None
2  70.0  80.0  90.0   None   None   None
3   None   None   None  100.0  200.0  300.0
4   None   None   None  400.0  500.0  600.0
"""

df = pd.read_csv(StringIO(txt), sep='\s+')

df = df.astype(pd.SparseDtype("float", None))

print(df.memory_usage())

Output:

Index    132
id        60
a1        36
a2        36
a3        36
b1        24
b2        24
b3        24
dtype: int64

@kkawabat, how about this, first separate a,b then assign both dataframes, a new row called, 'TYPE', for df of a fill TYPE with 'a' and TYPE 'b' for other and then merge both data frame, in such case u don't need to make df columns like a1, a2, b1, b2, you can just do col1,col2,.... or also u can set that TYPE to boolean and refer True as a and False as b.

It's a bit hacky but what I ended up doing was:

create a new dataframe with the columns id, type, value_dict
where type is either a or b
and the values of value_dict is a dictionary where the keys are all the col names with values in them. So for example,

id | a1 | a2 |....| aN | b1 | b2 |....| bM |
___________________________________________
0  |data|data|....|data|None|None|....|None|
1  |data|data|....|data|None|None|....|None|
2  |data|data|....|data|None|None|....|None|
3  |None|None|....|None|data|data|....|data|
4  |None|None|....|None|data|data|....|data|
....

will be:

id | type | value_dict |
___________________________________________
0  | "a"  | {"a1": data, "a2": data, ... "aN": data}|
1  | "a"  | {"a1": data, "a2": data, ... "aN": data}|
2  | "a"  | {"a1": data, "a2": data, ... "aN": data}|
3  | "b"  | {"b1": data, "b2": data, ... "bM": data}|
4  | "b"  | {"b1": data, "b2": data, ... "bM": data}|
....

It's kind of weird that I'm using dictionaries as elements but this seems the most intuitive solution that I could come up with.