ChunkedArray.Index on secondary column of table not working | ArrowTypeError

I am currently trying to implement the shortest path algorithm using pyarrow (first step for unweighted Graphs, second step for weighted graphs).

I am currently having an issue with the part where I need to verify if the target node is among the neighbors of the current node.

My data looks like this:

I have three columns: node, neighboring nodes and visited. The node column contains the name of each node in the graph. The neighboring nodes column contains an array of the names of the nodes that are directly connected to the node. The visited column contains a boolean value that indicates whether the node has been visited or not during a traversal algorithm.

In my example, I set the start node as 12160432, to obtain the neighboring nodes, I used the pc.filter function to retrieve the table in the red circle shown below

The next step would be to check if we reached the Target node, otherwise I will have to check the neighbors of the neighbors of my current node.

To check if the target is in the array I wanted to use the following functionIndex Function, Chunked Array as follows:

filtered_graph['neighboring_nodes'].index(10000001)

but I got the following error:
"ArrowTypeError: Could not convert 10000001 with type int: was not a sequence or recognized null for conversion to list type"

target_node = pa.scalar(10000001, type=pa.int64())
filtered_graph['neighboring_nodes'].index(target_node)

but got the same error.

Note: When using the "node" column, the index function works as intended:

(-1 means value not found)

I appreciate any guidance you can offer !

The index function is not implemented for arrays/chunked arrays of type list (eg: pa.list_(pa.int64()):

>>> pa.chunked_array([pa.array([[1,2,3], [4,5,6]])]).index([1,2,3])
ArrowNotImplementedError: Function 'index' has no kernel matching input types (list<item: int64>)

Even it it worked, it would allow you to search for a list of values that match exactly the list that is in the row. Whereas you are trying to check if an element is contained in a row.

You can achieve this but it requires you to massage the data a bit using pyarrow.compute

import pyarrow as pa
import pyarrow.compute as pc

table = pa.table({'node': [1,4], 'neighbors': [[2,3], [5,6]]})

flat_neighbors = pc.list_flatten(table['neighbors'])
flat_neighbors_index = pc.list_parent_indices(table['neighbors'])
flat_neightbors_parent = table['node'].take(flat_neighbors_index)

Which looks like this:

And then you can look up nodes and find their parents:

parents_of_2 = flat_neightbors_parent.filter(
    pc.equal(flat_neighbors, 2)
)
parents_of_2.to_pylist() # Returns [1]

Thank you 0x26res for your response !
following your logic I also found a little neat trick:

import pyarrow as pa
import pyarrow.compute as pc

table = pa.table({'node': [1,4], 'neighbors': [[2,3], [5,6]]})

flat_neighbors = pc.list_flatten(table['neighbors'])
values_to_check = pa.array([2])

mask = pc.is_in(flat_neighbors, value_set=values_to_check)
pc.any(mask) returns <pyarrow.BooleanScalar: True>