Grouping a list of a dictionaries (with common values) from 3 lists of dictionaries

list1 = [
  {'a': 1, 'fruit': 'apple', 'thing': 'aeroplane'}, 
  {'b': 2, 'fruit': 'banana', 'thing': 'bicycle'}, 
  {'c': 3, 'fruit': 'cherry', 'thing': 'chair'}
]

list2 = [
  {'fruit': 'apple', 'color': 'green'}, 
  {'fruit': 'banana', 'color': 'yellow'}, 
  {'fruit': 'cherry', 'color': 'red'}
]

list3 = [
  {'thing': 'aeroplane', 'capacity': 100}, 
  {'thing': 'bicycle', 'capacity': 2}, 
  {'thing': 'chair', 'capacity': 1}
]

what_i_want = [
  [
    {'a': 1, 'fruit': 'apple', 'thing': 'aeroplane'}, 
    {'fruit': 'apple', 'color':'green'}, 
    {'thing': 'aeroplane', 'capacity': 100}
  ],
  [
    {'b': 2, 'fruit': 'banana', 'thing': 'bicycle'}, 
    {'fruit': 'banana', 'color':'yellow'}, 
    {'thing': 'bicycle', 'capacity': 2}
  ],
  [
    {'c': 3, 'fruit': 'cherry', 'thing': 'chair'}, 
    {'fruit': 'cherry', 'color': 'red'}, 
    {'thing': 'chair', 'capacity': 1}
  ]
]

The grouping should be done by the order of list1. The lists I am working on have more than 100 dict objects in no particular order. Also the common values are computer generated ids and not the alphabetic values given in the example (just in case anybody was thinking of sorting by alphabets). I want to do this in the most pythonic way possible with minimum iterations.

I have looked at similar questions. They suggested using defaultdict, groupby and itemgetter but I'm not sure if these will work in my use case since the other questions were dealing with a single list while I am dealing with 3-4 lists.

We can solve this with a list comprehension with some nested list comprehensions to select matching items from list2 and list3.

[
  [
    x, 
    *[y for y in list2 if x['fruit'] == y['fruit']], 
    *[z for z in list3 if x['thing'] == z['thing']]
  ]
  for x in list1
]

Result:

[
  [
    {'a': 1, 'fruit': 'apple', 'thing': 'aeroplane'}, 
    {'fruit': 'apple', 'color': 'green'}, 
    {'thing': 'aeroplane', 'capacity': 100}
  ], 
  [
    {'b': 2, 'fruit': 'banana', 'thing': 'bicycle'}, 
    {'fruit': 'banana', 'color': 'yellow'}, 
    {'thing': 'bicycle', 'capacity': 2}
  ], 
  [
    {'c': 3, 'fruit': 'cherry', 'thing': 'chair'}, 
    {'fruit': 'cherry', 'color': 'red'}, 
    {'thing': 'chair', 'capacity': 1}
  ]
]

This works well for small samples of data, but performance is poor for large data sets due to repeated iterations over list2 and list3. This would be O(len(list1) * (len(list2) + len(list3))) or simplified O(n^2), which is not ideal.

Instead, using collections.defaultdict we can create a dictionary where the keys are either the 'fruit' or 'thing' values and the values are lists of any dictionaries that have those attributes.

This is slightly fragile, as it doesn't handle something like 'apple' as a value for a 'thing' key very well. To solve this we might have a dictionary of defaultdicts for each key name we're searching for. I leave that further exercise to the reader.

This approach is O(max(len(list1), len(list2) + len(list3))) or in simpler form O(n) since it only iterates once over any of the lists involved. This scales much better.

from collections import defaultdict

d = defaultdict(list)

for x in (list2+list3): 
  key = x.get('fruit', None) or x.get('thing', None)
  if key: d[key].append(x)

# {'apple':     [{'fruit': 'apple', 'color': 'green'}], 
#  'banana':    [{'fruit': 'banana', 'color': 'yellow'}], 
#  'cherry':    [{'fruit': 'cherry', 'color': 'red'}], 
#  'aeroplane': [{'thing': 'aeroplane', 'capacity': 100}],    
#  'bicycle':   [{'thing': 'bicycle', 'capacity': 2}], 
#  'chair':     [{'thing': 'chair', 'capacity': 1}]}

grouped_data = [
  [
    x, 
    *d[x['fruit']], 
    *d[x['thing']]
  ] 
  for x in list1
]

# [
#   [
#     {'a': 1, 'fruit': 'apple', 'thing': 'aeroplane'}, 
#     {'fruit': 'apple', 'color': 'green'}, 
#     {'thing': 'aeroplane', 'capacity': 100}
#   ], 
#   [
#     {'b': 2, 'fruit': 'banana', 'thing': 'bicycle'}, 
#     {'fruit': 'banana', 'color': 'yellow'}, 
#     {'thing': 'bicycle', 'capacity': 2}
#   ], 
#   [
#     {'c': 3, 'fruit': 'cherry', 'thing': 'chair'}, 
#     {'fruit': 'cherry', 'color': 'red'}, 
#     {'thing': 'chair', 'capacity': 1}
#   ]
# ]

For your sample data, you can simply zip the lists together, as the ordering in each list matches the others:

res = list(zip(list1, list2, list3))

If that is not going to be the case, you can build dicts from the list2 and list3 variables (using their fruit and thing properties as the keys), then use a list comprehension, indexing into those dicts using the fruit and thing properties from each list1 dict:

dict2 = { d['fruit'] : d for d in list2 }
dict3 = { d['thing'] : d for d in list3 }
res = [[ d, dict2.get(d['fruit'], None), dict3.get(d['thing'], None)] for d in list1]

Output:

[
  [
    {'a': 1, 'fruit': 'apple', 'thing': 'aeroplane'},
    {'fruit': 'apple', 'color': 'green'},
    {'thing': 'aeroplane', 'capacity': 100}
  ],
  [
    {'b': 2, 'fruit': 'banana', 'thing': 'bicycle'},
    {'fruit': 'banana', 'color': 'yellow'},
    {'thing': 'bicycle', 'capacity': 2}
  ],
  [
    {'c': 3, 'fruit': 'cherry', 'thing': 'chair'},
    {'fruit': 'cherry', 'color': 'red'},
    {'thing': 'chair', 'capacity': 1}
  ]
]