Masking a pandas column based on another column with slightly different values

So what I have is two Pandas dataframes in Python with a large number of xyz-coordinates. One of them will be used to mask/remove some coordinates in the other one, but the problem is that the coordinates are very slightly different so that I cannot simply remove duplicates. As an example, let's say they look like this:

df1 = pd.DataFrame(data=None, columns=['x', 'y', 'z'])
df1.x = [104245, 252355, 547364, 135152]
df1.y = [842714, 135812, 425328, 124912]
df1.z = [125125, 547574, 364343, 346372]

df2 = pd.DataFrame(data=None, columns=['x', 'y', 'z'])
df2.x = [104230, 547298]
df2.y = [842498, 424989]
df2.z = [124976, 364001]

What I then want is for the first and second xyz-rows in df2 to remove the first and third row in df1. My idea was to create new columns with rounded values, compare those, and remove based on those. It would look something like this:

df1['id'] = np.linspace(0,len(df1)-1,len(df1))
df2['id'] = np.linspace(0,len(df2)-1,len(df2))

df3 = df1.round({'x': -3, 'y': -3, 'z': -3})
df4 = df2.round({'x': -3, 'y': -3, 'z': -3})

df5 = df3.merge(df4, on=['x', 'y', 'z'], how='inner')
df6 = df1[~df1.index.isin(df5.id_x)]

This works fine to remove some of the values, but often they round to different places. I was hoping with help if there is a better method to mask those values which are simply closest in all three coordinates. Maybe that it finds the closest xyz-pair between df1 and df2 and masks those pairs. If anyone has any ideas I would really appreciate it!

Use KDTree:

from scipy.spatial import KDTree

# compute the distance with the nearest distance from df2
distances, indices = KDTree(df2).query(df1, k=1)

# keep row if the distance is greater than 3000
out = df1[distances >= 3000]

Output:

>>> out
        x       y       z
1  252355  135812  547574
3  135152  124912  346372

You can use numpy broadcasting to consider the individual distances between the coordinates:

# convert DataFrames to numpy arrays
a1 = df1.to_numpy()
a2 = df2.to_numpy()

# define a distance below which the coordinates are considered equal
thresh = 500

# compute the distances, identify matches on all coordinates
matches = (abs(a1[:,None]-a2) <= thresh).all(axis=-1)

idx1, idx2 = np.where(matches)
# (array([0, 2]), array([0, 1]))

out = df1.drop(df1.index[idx1])

To consider the euclidean distance between the points (taking into account all coordinates simultaneously), use scipy.spatial.distance.cdist:

from scipy.spatial.distance import cdist

thresh = 1000

matches = cdist(a1, a2) <= thresh

idx1, idx2 = np.where(matches)

out = df1.drop(df1.index[idx1])

Output:

        x       y       z
1  252355  135812  547574
3  135152  124912  346372

removing the single point from df1 that is closest to each row of df2 and below a threshold

from scipy.spatial.distance import cdist

thresh = 1000

dist = cdist(a1, a2)

idx = np.argmin(dist, axis=0)

out = df1.drop(df1.index[idx[dist[idx, np.arange(len(a2))] <= thresh]])

If the distance doesn't matter and you only want to remove the closest point:

from scipy.spatial.distance import cdist

dist = cdist(a1, a2)

idx = np.argmin(dist, axis=0)

out = df1.drop(df1.index[idx])

Another possible solution, which uses numpy broadcasting:

thresh = 1000
df1[~np.any(
    np.sqrt(np.sum((df1.values - df2.values[:, None])**2, axis=2)) < thresh,
    axis=0 )]

If we want to remove the closest points, we can use:

df1.iloc[df1.index.difference(np.argmin(
    np.sqrt(np.sum((df1.values - df2.values[:, None])**2, axis=2)), axis=1)), :]

Output:

        x       y       z
1  252355  135812  547574
3  135152  124912  346372