Java Spark SQL: Merging and overwriting Datasets with identical schema

Java 11 and Spark SQL 2.13:3.3.2 here. Please note: I'm using and interested in the Java API and would appreciate Java answers, but I can probably decipher Scala-based answers and do the necessary Scala-to-Java conversions if necessary. But Java would be appreciated!

I have 2 Dataset<Row> instances, both with the same exact schema (same columns/headers, which are the same types):

data set #1 (ds1)
===
fruit,quantity
--------------
apple,50
pear,12
orange,0
kiwi,104

data set #2 (ds2)
===
id,fruit,quantity
--------------
banana,50
pineapple,25
orange,5
blueberry,15

I would like to "merge" these 2 Dataset<Row>s so that they are appended or joined to one another, but in such a way that ds2 overwrites any values in ds1 if their fruit columns match. So orange is in both data sets, but in ds2 its quantity is 5, so that should be the final listing in the resultant Dataset<Row> output. So in other words, this operation should results in a 3rd data set like so:

data set #2 (ds2)
===
id,fruit,quantity
--------------
apple,50
pear,12
orange,5
kiwi,104
banana,50
pineapple,25
blueberry,15

The order of the rows does not matter, all that matters to me is that the list of fruits in both data sets are listed in the 3rd, and that ds1 rows are updated (not inserted) if there is a matching fruit in ds2.

I took a look at Dataset#join JavaDocs, but they seem to be just useful for when you need the SQL equivalent of an inner join, but won't help me with the desired overwrite functionality.

Thanks in advance for any and all help!

You can use "full" join to get all values from both dataset, then use coalesce to get the preferred column values:

ds2 = ds2.withColumnRenamed("fruit", "fruit2").withColumnRenamed("quantity", "quantity2");
ds1.join(ds2, ds1.col("fruit").equalTo(ds2.col("fruit2")), "full")
        .withColumn("fruit", functions.coalesce(col("fruit"), col("fruit2")))
        .withColumn("quantity", functions.coalesce(col("quantity2"), col("quantity")))
        .drop("fruit2", "quantity2")
        .show();

Result:

+---------+--------+
|    fruit|quantity|
+---------+--------+
|     kiwi|     104|
|   orange|       5|
|    apple|      50|
|     pear|      12|
|   banana|      50|
|pineapple|      25|
|blueberry|      15|
+---------+--------+