Sending an image with an ID across processes in Python, Windows

I have a project with multiple Python processes on the same computer.

Python process A is the central node, and it receives images and data from sensors.

Python process A sends images to process B for image processing/feature detection, and process B sends back some information to process A. The system is much more complex than this which is why it's separated in the first place.

One idea that I want to implement is to tag the images with an integer, let's say x. This way, process B can say "Hi process A, this is the data for image x".

The problem is, how can I send both images and this integer elegantly and rapidly across processes?

Ideas:

• Use MQTT or ZeroMQ to communicate. Turn image into string, put it in a dict with the id, send it. I think these will be slow because we are serializing lots of images and sending lots of data over these chunky protocols.
• Use Shared memory to share the image. I have prototyped this, it is very fast to send images, but encoding the integer is not elegant. I would maybe encode the integer into the RGB value of a few pixels in the corner, but this feels janky.
• Pickling might be better than string serialization since its bytes vs strings and so much faster? The multiprocessing pipes and queues seem like they are for this, but the sender and receiver seem to need to start in the same Python thread.
• Find a unique hash for each image in every process. I'm leaning towards this option right now, it's very easy to understand but probably not the fastest.

I recognize this is an architecture question and those are frowned upon, but I feel like there has to be a more elegant solution and I think this is an interesting design problem to ask.

Personally, I find Redis great for this type of thing. Imagine it as a blazing fast, "in-memory" datastructure server that can serve atomic integers, strings, images, lists, queues, sets, sorted sets, JSON, streams, PUB/SUB ... here are the commands to create/read objects.

It has bindings for C, C++, Python, Ruby, PHP and bash commandline so you can inject and check test data easily just from the shell.

And it's entirely distributed across networks.

And it handles binary data (no need for serialisation) and keys and values up to 512MB.

So you can create a Redis object containing a JPEG/PNG/TIFF or raw image of binary data (SET), then push (LPUSH) a JSON containing the object name and id onto a queue for processing.

Run as many processing nodes as you need doing pops off the queue (BRPOP), and scale by adding more.

Have another results queue where workers put another JSON with their answers and corresponding id.

You can also specify a "Time-To-Live" so that datastructures are automagically deleted after a given time.

You can also create persistance across reboots, if you want to.

You can also add redundancy, if you want to cluster it.

I have done many examples, try searching for my username and Redis and image. Hopefully, this will work.