pyspark refer a different dataframe

I have two dataframes - df1 and df2. I have one common column in both (id). I want to add a new column in df1 (say A) by: using this common column id, refer a col (say 'B') in df2 and store this B-col value from df2 into df1 in col-A. df1 has more than 5 cols and and df2 has only 2 columns.

I tried left-join...but, it is resulting in a few duplicate rows (from df1, the "left" dataframe). Is there a way (write a UDF) to just get the id from df1 and using that id, refer df2 and populate a new col in df1?

I do not want to do group-by to eliminate any duplicates. I want all the rows in df1 and just use the ID col to get its respective value from df2 and populate in df1. I am new to PySpark...and do not know how to do it.

UPDATE: Here is full schema of the df1 and df2.

df1: <date, brand, hex_num, text, topic_idx>
df2: <topic_idx, cities>

Based on topic_idx, I want the respective cities to be populated in df1 in a separate col. In the final resulting df, I want the same schema as df1 + a new col "cities". The number of rows of final df must be the same as that of df1. Each row in final df (excluding "cities" col) must be identical to that of df1.

In pure python, perhaps I could have done with a apply func with a dict. I was looking for something similar....

UPDATE 2: The cities column is list of strings. For eg: ['austin', 'chicago', 'boston']. And, this list is unique for each topic_idx. In essence, in df2, the topic_idx -> city_list is unique and no two rows of df2 are identical.

From your question, it seems like df2 has duplicates in id column which is why you are getting additional rows post join.

My suggestion would be as follows

1. Remove the duplicate rows in df2 before the join if the situation permits. Or
2. Write a window function on df2, generate a row_number or rank by partition over id column and order by some other column. Then filter out the records based on the row_number/ rank of the rows within each partition, call it df3. I would prefer this as it gives you a lot more control in choosing the rows you want to keep. Then join df1 and df3.
3. Post join, you can create the new column.

If you can edit the question by providing the schema of the dataframes, I can give you the code as well. LMK if this solves the issue.

UPDATE
As per the details shared,

1. Run a group by on df2 on topic_idx and collect the cities as a list.

df2.show(truncate=False)
+---------+-------+
|topic_idx|cities |
+---------+-------+
|1        |New York|
|1        |London |
|2        |Paris  |
|2        |Berlin |
|2        |Madrid |
+---------+-------+

from pyspark.sql.functions import collect_list

df3 = df2.groupBy('topic_idx').agg(collect_list('cities').alias('city_list'))
df3.show(truncate=False)

+---------+------------------+
|topic_idx|city_list         |
+---------+------------------+
|1        |[New York, London]|
|2        |[Paris, Berlin, Madrid]|
+---------+------------------+

2. Join df3 against df1 on topic_idx.

df1.show(truncate=False)
+----------+-----+-------+---------------------+---------+
|date      |brand|hex_num|text                 |topic_idx|
+----------+-----+-------+---------------------+---------+
|2023-06-01|A    |#FF0000|Sample text 1        |1        |
|2023-06-02|B    |#00FF00|Sample text 2        |2        |
|2023-06-03|C    |#0000FF|Sample text 3        |3        |
+----------+-----+-------+---------------------+---------+

final_df = df1.join(df3, ['topic_idx'], how='left')
final_df.show(truncate=False)

+---------+----------+-----+-------+---------------------+------------------+
|topic_idx|date      |brand|hex_num|text                 |city_list         |
+---------+----------+-----+-------+---------------------+------------------+
|1        |2023-06-01|A    |#FF0000|Sample text 1        |[New York, London]|
|2        |2023-06-02|B    |#00FF00|Sample text 2        |[Paris, Berlin, Madrid]|
|3        |2023-06-03|C    |#0000FF|Sample text 3        |null              |
+---------+----------+-----+-------+---------------------+------------------+