Slow frame rate and high resource usage when extracting frames with ffmpeg in C#

I'm currently working on a project where I need to extract frames from a video using ffmpeg in C#. However, I'm facing issues with a slow frame rate and high resource usage. The code I'm using is as follows:

private bool move = false;
private int master_frame = 0;

private void pic()
{
    using (Process process = new Process())
    {
        process.StartInfo.FileName = "C:/Users/lenovo/Desktop/ffmpeg.exe";
        process.StartInfo.Arguments = $"-i \"C:/Users/lenovo/Desktop/New folder/video.mp4\" -vf \"select=gte(n\\,{master_frame})\" -vframes 1 -q:v 2 -f image2pipe -c:v bmp -";
        process.StartInfo.UseShellExecute = false;
        process.StartInfo.CreateNoWindow = true;
        process.StartInfo.RedirectStandardOutput = true;
        process.StartInfo.RedirectStandardError = true;
        process.Start();

        using (MemoryStream outputStream = new MemoryStream())
        {
            byte[] buffer = new byte[1024];
            int bytesRead;
            while ((bytesRead = process.StandardOutput.BaseStream.Read(buffer, 0, buffer.Length)) > 0)
            {
                outputStream.Write(buffer, 0, bytesRead);
            }

            pictureBox1.Invoke((MethodInvoker)(() =>
            {
                pictureBox1.Image?.Dispose();
                pictureBox1.Image = new Bitmap(outputStream);
            }));
        }
    }
}

private async void panel1_MouseUp(object sender, MouseEventArgs e)
{
    move = true;
    await Task.Run(() =>
    {
        while (move)
        {
            pic();
            master_frame++;
        }
    });
}

The issue is that the frame rate is quite slow, and the resource usage is higher than expected. I suspect that reading the output stream of ffmpeg and creating a Bitmap from a MemoryStream for each frame might be causing the performance issues.

I would appreciate any insights or suggestions on how to optimize the frame extraction process for better performance and lower resource usage. Is there a more efficient way to extract frames from a video using ffmpeg in C#? Are there any alternative approaches or optimizations that could help improve the frame extraction speed?

Thank you in advance for your help and suggestions!

TLDR: The very low framerate is due to the enormous delay there is between starting the ffmpeg process and the actual data starting to be sent by ffmpeg.

My answer to your other post ("How to Play a Video in a PictureBox Using FFmpeg in C#?"), which is essentially the same as this post, shows how to do it much faster, although still not at 60 frames per second

Measuring performances

Using Stopwatches, I have measured the time it takes for each steps of your code.

Note: I have changed the format from bmp to png for smaller individual frame sizes

Code

Here's my measuments code:

private bool move = false;
private int master_frame = 0;

Stopwatch sw = new Stopwatch();
private void pic()
{
    using (Process process = new Process())
    {
        process.StartInfo.FileName = "ffmpeg.exe";
        process.StartInfo.Arguments = $"-i \"video.mp4\" -vf \"select=gte(n\\,{master_frame})\" -vframes 1 -q:v 2 -f image2pipe -c:v png -";
        process.StartInfo.UseShellExecute = false;
        process.StartInfo.CreateNoWindow = true;
        process.StartInfo.RedirectStandardOutput = true;
        process.StartInfo.RedirectStandardError = true;
        process.Start();

        using (MemoryStream outputStream = new MemoryStream())
        {
            Console.WriteLine("-");

            sw.Restart();
            byte[] buffer = new byte[1024];
            int bytesRead;
            while ((bytesRead = process.StandardOutput.BaseStream.Read(buffer, 0, buffer.Length)) > 0)
            {
                outputStream.Write(buffer, 0, bytesRead);
            }
            sw.Stop();
            Console.WriteLine("Reading bytes: " + sw.ElapsedMilliseconds + "ms");

            framePictureBox.Invoke((MethodInvoker)(() =>
            {
                sw.Restart();
                framePictureBox.Image?.Dispose();
                sw.Stop();
                Console.WriteLine("Disposing of old picture: " + sw.ElapsedMilliseconds + "ms");

                sw.Restart();
                Bitmap bmp = new Bitmap(outputStream);
                sw.Stop();
                Console.WriteLine("Converting stream to bitmap: " + sw.ElapsedMilliseconds + "ms");

                sw.Restart();
                framePictureBox.Image = bmp;
                sw.Stop();
                Console.WriteLine("Assigning to PictureBox: " + sw.ElapsedMilliseconds + "ms");

                sw.Restart();
                framePictureBox.Refresh();
                sw.Stop();
                Console.WriteLine("Rendering: " + sw.ElapsedMilliseconds + "ms");
            }));
        }
    }
}

Measurements

And here are my results, for a 480p 30fps video that is 1 minute and 36 seconds long, on my computer, I have on average:

• Reading the bytes sent by ffmpeg: 110ms, with peaks to around 300ms
• Disposing of the previous frame: <1ms
• Converting the stream to a bitmap: 1ms
• Assigning the converted bitmap to the PictureBox: <1ms
• Rendering the PictureBox to the screen: 2ms

As a reminder, to achieve 60 FPS you'd need to render each frame in 16.67ms, or 33.33msfor 30 FPS.

What can we understand from that

It is quite blatant that the culprit here is the time it takes to read the stream that ffmpeg sends.

By the way, I have also tried @Charlieface's suggestion, and no, using process.StandardOutput.BaseStream.CopyTo(outputStream); doesn't save much time.

In fact, you could just use new Bitmap(process.StandardOutput.BaseStream); directly, but it doesn't make the whole thing that must faster.

The real issue

At first, it might seem obvious to blame ffmpeg for being slow to convert the frame we asked for in the desired format.

And as pointed out by @ChristophRackwitz, it is indeed a factor of your code's slowness.

When asking ffmpeg for a specific frame, it actually needs to decode the entire video stream up to that frame, resulting in more and more time being taken for each frame as time goes on.

This wasn't an issue on my side though since I have used a video with a very low quality and framerate. My measurements are therefore minimalistic, and in practice, it could take much longer to render each frame than what I have measured. I'll let you imagine the time it'd take to get the last frame of a 1 hour long 4K 144fps video.

There is another factor as well, and it's about process creation.

Here's the thing, when you start a new ffmpeg process instance, here's what happens:

• Windows needs to allocate space to boot a new process
• ffmpeg needs to ask Windows to open your video file
• Windows gives a file handle to ffmpeg
• ffmpeg needs to ask Windows for a stream to output its data when it'll be done converting
• Windows gives ffmpeg a stream to send its data to
• ffmpeg can finally do its job and send it to the stream
• ffmpeg needs to tell Windows to close the video file since it's done with it
• Windows closes the file
• ffmpeg needs to tell Windows to close the stream it opened since it no longer needs it
• Windows closes the stream
• ffmpeg needs to tell Windows that it has finished and can close
• Windows can free the ffmpeg process

That's quite a lot, right?

The issue here is that out of all these steps, only one of them is unique to every frame, and that is the "ffmpeg can finally do its job and send it to the stream" one.

All of the other steps, which takes a massive amount of time, are repeated for each frame.

Hence my answer to your other post, in which instead of opening a stream for each frame, we open a single stream for all of them and render them in real time. This avoids creating a new process instance for each frame, and also avoids having to decode the entire video stream from the beginning for each frame.

The end solution

You can't improve the speed of your code with the way it currently works, you need to switch from one process per frame to one process for each of them, which is what my answer to your other post does.

Thanks for reading.