How to Play a Video in a PictureBox Using FFmpeg in C#?

I'm trying to develop a C# application that plays a video within a PictureBox control using FFmpeg. I have the ffmpeg.exe file and would like to stream the video frames to update the PictureBox in real-time.

I have already tried some solutions, but I'm encountering issues with the conversion from Base64 string to image data. The error I'm receiving is:

> The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.

Could someone please provide guidance or an example of how to properly play a video in a PictureBox using FFmpeg in C# without encountering this Base64 conversion issue? I would greatly appreciate any help or code snippets demonstrating the correct approach.

Here's a summary of what I've done so far:

private void button5_Click(object sender, EventArgs e)
{
    if (_ffmpegProcess != null)
        return;

    _ffmpegProcess = new Process();
    _ffmpegProcess.StartInfo.FileName = "C:/Users/google/Desktop/ffmpeg.exe";
    _ffmpegProcess.StartInfo.Arguments = $"-i \"{"C:/Users/google/Desktop/New folder/video.mp4"}\" -vf format=rgb24 -f image2pipe -";
    _ffmpegProcess.StartInfo.RedirectStandardOutput = true;
    _ffmpegProcess.StartInfo.UseShellExecute = false;
    _ffmpegProcess.StartInfo.CreateNoWindow = true;
    _ffmpegProcess.EnableRaisingEvents = true;
    _ffmpegProcess.OutputDataReceived += FfmpegOutputDataReceived;
    _ffmpegProcess.Exited += FfmpegProcessExited;

    _ffmpegProcess.Start();
    _ffmpegProcess.BeginOutputReadLine();
}

private void StopVideo()
{
    if (_ffmpegProcess == null)
        return;

    _ffmpegProcess.OutputDataReceived -= FfmpegOutputDataReceived;
    _ffmpegProcess.Exited -= FfmpegProcessExited;
    _ffmpegProcess.Kill();
    _ffmpegProcess.Dispose();
    _ffmpegProcess = null;
}

private void FfmpegOutputDataReceived(object sender, DataReceivedEventArgs e)
{
    if (!string.IsNullOrEmpty(e.Data))
    {
        // Decode the received image data
        byte[] imageData = Convert.FromBase64String(e.Data);

        // Display the image in the PictureBox control
        pictureBox1.Invoke((MethodInvoker)(() =>
        {
            using (MemoryStream memoryStream = new MemoryStream(imageData))
            {
                pictureBox1.Image = new System.Drawing.Bitmap(memoryStream);
            }
        }));
    }
}
private void FfmpegProcessExited(object sender, EventArgs e)
{
    StopVideo();
}

private void Form1_FormClosing(object sender, FormClosingEventArgs e)
{
    StopVideo();
}

1. I have the ffmpeg.exe file available and added to my project.
2. I have created a Windows Forms application with a PictureBox control.
3. I am starting the FFmpeg process and redirecting the output to read the video frames.
4. However, I'm facing difficulties when converting the FFmpeg output data from base64 string to image data.

TLDR: ffmpeg doesn't output images as base64 strings, and also doesn't send bytes in a perfectly trimmed way so you have each picture separate from one another. My answer includes a working project and explanations on how I figured out how to do it.

Download the full VS project here. It's in the public domain, no credits needed (but still appreciated).

What your code actually does

First things first, I have reproduced your project using the code you have given.

You expected the OutputDataReceived event to contain data as a base64 string, but it is not.
By checking e.Data's value in your FfmpegOutputDataReceived function, I found the following:

As you can see, it does not contain the video data as a base64 string, but as raw data in the form of a string.

Getting data in the correct form

Why your code couldn't have worked

My first attempt was to just get the pictures as a byte array from the e.Data string, using a simple for loop that iterates through each character of the string and places it in a byte array.

However, this didn't work for the following reasons:

• In your ffmpeg command line you have passed the argument "-vf format=rgb24", which makes ffmpeg output the data in a non-standard way. Remember that C# Bitmaps can be made from BMP, GIF, EXIF, JPG, PNG, and TIFF files only.

• C# string are Unicode-encoded, which means that chars are two bytes long, and thus can't be directly converted to single byte values

• Data is sent as a string which seems to be a direct conversion from what bytes as characters would be in ANSI/CP1252 to Unicode.
  I have noticed this when I changed the output to PNG and check the header: the first PNG byte must always have value 0x89, but e.Data outputed character value 0x2030, which is the Unicode for ‰, which is 0x89 in ANSI/CP1252.
  This pretty much means that this whole data receiving method causes data loss, and I also found that 0x0D and 0x0A bytes were missing.

Fixing the ffmpeg process part

So unfortunately, your system to get ffmpeg's outputed data had to be entirely ditched.

To get the pipe'd output from ffmpeg as an intact byte array, I used the code from this answer, which, once adapted to your process, looks like this:

Stream baseStream = _ffmpegProcess.StandardOutput.BaseStream;

This line gets the data ffmpeg outputs as a Stream, which is read bytes by bytes and is guaranteed to have no data loss.

I also changed it so ffmpeg outputs PNG data instead of that rgb24 format you seemed to use, I hope it doesn't lower the quality of my answer.

In the end, the whole code snippet to initialize ffmpeg is as follows:

// Create the FFMPEG process
Process _ffmpegProcess = new Process();
_ffmpegProcess.StartInfo.FileName = "ffmpeg.exe";
_ffmpegProcess.StartInfo.Arguments = $"-hide_banner -loglevel error -i \"{"video.mp4"}\" -c:v png -f image2pipe -"; // No useless outputs, uses video.mp4 as an input, outputs it as PNG files streamed to the pipe
_ffmpegProcess.StartInfo.RedirectStandardOutput = true;
_ffmpegProcess.StartInfo.UseShellExecute = false;
_ffmpegProcess.StartInfo.CreateNoWindow = true;
_ffmpegProcess.EnableRaisingEvents = true;

// Start FFMPEG
_ffmpegProcess.Start();

// Output PNGs are streamed into the process' standard output base stream
Stream baseStream = _ffmpegProcess.StandardOutput.BaseStream;

How to handle the streamed bytes to display the video

Here is the way I suggest to handle this:

• We have a BackgroundWorker that, when activated, starts the ffmpeg process and has a while loop that reads each PNG file from ffmpeg's stream into a byte array (the buffer).

• When the buffer contains the PNG file to render, it is converted into a Bitmap

• The BackgroundWorker then reports progress as the Bitmap we have made

• The event triggered upon the BackgroundWorker's progress report receives the Bitmap and puts it inside of the picture box we want to render it in

The main reason we need to use a BackgroundWorker is that Controls can't be accessed from a thread other than the one who created them.

And multithreading is required here, because otherwise, the entire form would be frozen until the video is done playing, because it'd be busy rending frames at all times.

Cutting down ffmpeg's data into individual frames to render

Unfortunately, ffmpeg doesn't stream the data as perfectly trimmed down PNG files that you can render.
(Which, by the way, is normal since that's how process pipes work)

So what you get is a giant stream with many PNGs appended to each other.
So we need to figure out where each PNG ends so we can render them.

To split them, I refered to the PNG File Format Specifications, which indicate the following:

• PNG files are made out of a header, and an undefined amount of chunks
• The header is always 8 bytes of values 89 50 4E 47 0D 0A 1A 0A 00
• Chunks are made out of:
  • 4 bytes indicating the size (in bytes) of the chunk's content
  • 4 bytes representing the chunk's identifier
  • 4 bytes containing the CRC32 checksum of the chunk
  • The chunk's content
• The first chunk is always an IHDRchunk of size 13, and the last one is always an IEND chunk of size 0

This means that we can cut each PNGs from the stream by simply adding each chunk to a buffer until we hit the IEND chunk, in which case it means that we are done reading the current PNG, so we can send it for rendering.

Here is the code that handles this, please read the comments for detailed explanations of its inner workings:

// Buffer to hold the current PNG. 16588800 bytes should be enough to hold any 1080p PNG file (1920 * 1080 * 4 * 2 = 16588800)
// Source: https://stackoverflow.com/a/20415800/9399492
// Note: This answer claims that the max size in bytes is "width * height * 4", I multiplied the total by two to have a good enough margin.
//       16588800 bytes is 16 MB, which is absolutely nothing by today's standards.
//       So even if bigger videos need to be played, you can increase this number without worrying too much about RAM space.
byte[] buffer = new byte[16588800];

// Current writing index in the buffer
int currentIdx = 0;

// Keep going for as long as the video background worker wasn't scheduled for cancellation, or if we reached the end of the video
while (!videoBackgroundWorker.CancellationPending && !_ffmpegProcess.HasExited)
{
    // The goal with this loop is to cut down each PNG sent by FFMPEG and to display them.
    // Problem is, all PNGs are appended to each other. Cutting down each PNGs is therefore not a very easy task.
    // The PNG file format specification tells us more: http://www.libpng.org/pub/png/spec/1.2/PNG-Structure.html#Chunk-layout
    // PNG files are divided in chunks that are defined by their size in bytes, a four bytes identifier, a checksum and their content.
    // And fortunately, there is an end chunk named "IEND", of size 0.
    // Thus, all we have to do is to gather bytes from all chunks until we find the IEND chunk, in which case it means that we hit the end of the current PNG file.
    // Then, we can parse it as a C# Bitmap and go back to the beginning of this loop to read the next PNG.

    // Reset the index to 0 on every new PNG
    currentIdx = 0;

    // Read the PNG Header
    if (baseStream.Read(buffer, currentIdx, 8) <= 0)
        break;

    currentIdx += 8;

    // The PNG Header must always have these values according to http://www.libpng.org/pub/png/spec/1.2/PNG-Structure.html#PNG-file-signature
    // If these aren't present, it likely means that the stream is corrupted and therefore the video can't be further read.
    if (buffer[0] != 0x89
        || buffer[1] != 0x50
        || buffer[2] != 0x4E
        || buffer[3] != 0x47
        || buffer[4] != 0x0D
        || buffer[5] != 0x0A
        || buffer[6] != 0x1A
        || buffer[7] != 0x0A
        )
        throw new Exception("Invalid PNG header");

    // This loop will go through each chunk of the PNG file and read them into the buffer.
    // When the IEND chunk is read, the loop is broken
    while (true)
    {
        // Read the chunk header
        baseStream.Read(buffer, currentIdx, 12);

        int bIdx = currentIdx;
        currentIdx += 12;

        // Get the chunk's content size in bytes from the header we just read
        int totalSize = (buffer[bIdx] << 24) | (buffer[bIdx + 1] << 16) | (buffer[bIdx + 2] << 8) | (buffer[bIdx + 3]);

        // If the size is positive, then read the chunk's content and write it into the buffer
        if (totalSize > 0)
        {
            // We are sometimes faster than FFMPEG can output the video, so we need to keep reading for as long as we didn't read the entirety of the section
            int totalRead = 0;
            while (totalRead < totalSize)
            {
                int read = baseStream.Read(buffer, currentIdx, (totalSize - totalRead));
                currentIdx += read;
                totalRead += read;
            }
        }

        // If the size is 0 and the section identifier is IEND, then break out of the loop: we are done with this PNG file
        else if (totalSize == 0
            && buffer[bIdx + 4] == 0x49 // I
            && buffer[bIdx + 5] == 0x45 // E
            && buffer[bIdx + 6] == 0x4E // N
            && buffer[bIdx + 7] == 0x44 // D
            )
        {
            break;
        }
    }

    // Get the PNG file's bytes as a memory buffer to be converted to a C# Bitmap
    using (var ms = new MemoryStream(buffer, 0, currentIdx))
    {
        // Report progress as the Bitmap of the frame to display.
        videoBackgroundWorker.ReportProgress(0, new Bitmap(ms)); // Byte array to bitmap conversion takes about 10ms on my computer
    }
}

// Report null as progress, to indicate being done
videoBackgroundWorker.ReportProgress(0, null);

// If we broke out of the loop due to FFMPEG being done, no need to manually kill it
if (_ffmpegProcess.HasExited)
    return;

// If we broke out of the loop due to a cancellation, forcefully kill FFMPEG and dispose of it.
_ffmpegProcess.Kill();
_ffmpegProcess.Dispose();

Performance

As expected from such a hacky way to render a video, it performs very poorly.

On my computer, the part that cuts PNGs takes 1 millisecond most of the time, but every few seconds there seem to be a lag spike (maybe due to disk reading?) that makes it take 30 to 40 milliseconds for one to three frames.

The part that converts the PNG byte array into a Bitmap seems to take a constant 10 to 12 milliseconds on my computer though.

To achieve a fluid 60 frames per second, we'd need to render each frame in 16.67 milliseconds (or less, in which case we'd need to wait the remaining amount of time).

I do not doubt that there might be ways to optimize this, though remember that Windows Forms is a windows rendering framework, not a video player. It wasn't designed for kind of thing.

There might be video playing libraries that could help do all of this much more simply, but considering you wanted to use ffmpeg and render the video on a PictureBox, I sticked to those constraints, or it wouldn't have really been an answer.

End result

I have also added an FPS counter:

Do note that the framerate is uncapped, meaning that assuming you have a computer with infinite performances, the video will run at an infinite speed.

I do not know how to get the video's framerate from ffmpeg to cap it.

You can download the full VS project here. It's in the public domain, no credits needed (but still appreciated).

Note: I uploaded it on Google Drive, I'm not sure what is the recommended platform to upload zips like this, moderators please feel free to reupload the file in a more appropriate place and to edit my post to update the link

End words

That's all.

You might want to take a look at the ffmpeg documentation to see if there are better ways to stream videos, or maybe check if there is a NuGet package that could do the work for you as well.

Thanks for reading.