英文:
How to fix corrupted video (green frames) after muxing using MediaMuxer and MediaCodec
问题
I am trying to encode an mp4 video from an array of NalUnits where each unit is a frame represented by byte[] that is saved from an rtp packet from an rtsp stream. I am encoding 451 frames at a 30fps where the first frame data is the spp and pps frame combined. This is my current configuration:
Width: 720
Height: 480
Bitrate: 10,000,000
Fps: 30
Colorformat: COLOR_FormatYUV420Flexible
Key I Frame Interval: 1
Here is my method where I encode the configure the encoder:
public boolean createMp4Video(FrameQueue.Frame[] frames, File dir) {
try {
// Set up MediaMuxer
Date date = new Date();
File file = new File(dir, date + "-recording.mp4");
this.muxer = new MediaMuxer(file.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
// Set up MediaCodec
mMediaFormat = MediaFormat.createVideoFormat(codecName, width, height);
mMediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat);
mMediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, bitrate);
mMediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, framerate);
mMediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
codec = MediaCodec.createEncoderByType(codecName);
codec.configure(mMediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
codec.start();
MediaCodec.BufferInfo videoBufferInfo = new MediaCodec.BufferInfo();
// Process frames
for (int i = 0; i < frames.length; i++) {
if (i == frames.length - 1) { // if last frame, add end of stream flag
muxFrame(frames[i].getData(), i, true);
} else {
muxFrame(frames[i].getData()), i, false);
}
// Stop and release resources
muxer.stop();
muxer.release();
codec.stop();
codec.release();
listener.onMp4VideoCreated();
return true;
} catch (IOException e) {
return false;
}
}
I loop through all the frames using a simple for loop and call the following function to method:
private void muxFrame(byte[] frameData, int encodeIndex, boolean isVideoEOS) {
long presentationTimeUs = 1000000 / framerate * encodeIndex
int inputBufferIndex = codec.dequeueInputBuffer(-1);
if (inputBufferIndex >= 0) {
ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferIndex);
inputBuffer.put(frameData);
codec.queueInputBuffer(inputBufferIndex, 0, frameData.length, presentationTimeUs, 0);
}
// Create a MediaCodec.BufferInfo to hold the frame information
MediaCodec.BufferInfo muxerOutputBufferInfo = new MediaCodec.BufferInfo();
int outputBufferIndex = codec.dequeueOutputBuffer(muxerOutputBufferInfo, 1000);
switch (outputBufferIndex) {
case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
MediaFormat newFormat = codec.getOutputFormat();
mVideoTrack = muxer.addTrack(newFormat);
muxer.start();
break;
case MediaCodec.INFO_TRY_AGAIN_LATER:
break;
default:
// Set the size, presentation time, and flags of the muxerOutputBufferInfo
muxerOutputBufferInfo.size = frameData.length;
muxerOutputBufferInfo.offset = 0;
muxerOutputBufferInfo.flags = isVideoEOS ? MediaCodec.BUFFER_FLAG_END_OF_STREAM : MediaCodec.BUFFER_FLAG_KEY_FRAME;
muxerOutputBufferInfo.presentationTimeUs = presentationTimeUs;
// Write the frame data to the muxer
muxer.writeSampleData(mVideoTrack, ByteBuffer.wrap(frameData), muxerOutputBufferInfo);
codec.releaseOutputBuffer(outputBufferIndex, false);
break;
}
}
I am able to save the video and play it but all the frames are corrupted and green as such
. Here is my log when I am encoding:[Log output]
Note: I've translated the code comments and provided an overview of the code. If you have specific questions or need further assistance, please let me know.
英文:
I am trying to encode an mp4 video from an array of NalUnits where each unit is a frame represented by byte[] that is saved from an rtp packet from an rtsp stream. I am encoding 451 frames at a 30fps where the first frame data is the spp and pps frame combined. This is my current configuration:
Width: 720
Height: 480
Bitrate: 10_000_000
Fps: 30
Colorformat: COLOR_FormatYUV420Flexible
Key I Frame Interval: 1
Here is my method where I encode the configure the encoder:
public boolean createMp4Video(FrameQueue.Frame[] frames, File dir) {
try {
// Set up MediaMuxer
Date date = new Date();
File file = new File(dir, date + "-recording.mp4");
this.muxer = new MediaMuxer(file.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
// Set up MediaCodec
mMediaFormat = MediaFormat.createVideoFormat(codecName, width, height);
mMediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat);
mMediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, bitrate);
mMediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, framerate);
mMediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
codec = MediaCodec.createEncoderByType(codecName);
codec.configure(mMediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
codec.start();
MediaCodec.BufferInfo videoBufferInfo = new MediaCodec.BufferInfo();
// Process frames
for (int i = 0; i < frames.length; i++) {
if (i == frames.length - 1) { // if last frame, add end of stream flag
muxFrame(frames[i].getData(), i, true);
} else {
muxFrame(frames[i].getData()), i, false);
}
// Stop and release resources
muxer.stop();
muxer.release();
codec.stop();
codec.release();
listener.onMp4VideoCreated();
return true;
} catch (IOException e) {
return false;
}
}
I loop through all the frames using a simple for loop and call the following function to method:
private void muxFrame(byte[] frameData, int encodeIndex, boolean isVideoEOS) {
long presentationTimeUs = 1000000 / framerate * encodeIndex
int inputBufferIndex = codec.dequeueInputBuffer(-1);
if (inputBufferIndex >= 0) {
ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferIndex);
inputBuffer.put(frameData);
codec.queueInputBuffer(inputBufferIndex, 0, frameData.length, presentationTimeUs, 0);
}
// Create a MediaCodec.BufferInfo to hold the frame information
MediaCodec.BufferInfo muxerOutputBufferInfo = new MediaCodec.BufferInfo();
int outputBufferIndex = codec.dequeueOutputBuffer(muxerOutputBufferInfo, 1000);
switch (outputBufferIndex) {
case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
MediaFormat newFormat = codec.getOutputFormat();
mVideoTrack = muxer.addTrack(newFormat);
muxer.start();
break;
case MediaCodec.INFO_TRY_AGAIN_LATER:
break;
default:
// Set the size, presentation time, and flags of the muxerOutputBufferInfo
muxerOutputBufferInfo.size = frameData.length;
muxerOutputBufferInfo.offset = 0;
muxerOutputBufferInfo.flags = isVideoEOS ? MediaCodec.BUFFER_FLAG_END_OF_STREAM : MediaCodec.BUFFER_FLAG_KEY_FRAME;
muxerOutputBufferInfo.presentationTimeUs = presentationTimeUs;
// Write the frame data to the muxer
muxer.writeSampleData(mVideoTrack, ByteBuffer.wrap(frameData), muxerOutputBufferInfo);
codec.releaseOutputBuffer(outputBufferIndex, false);
break;
}
}
I am able to save the video and play it but all the frames are corrupted and green as such https://dropmefiles.com/rn6I0
Here is my log when I am encoding:
I/OMXClient: IOmx service obtained
W/OMXUtils: do not know color format 0x7f000200 = 2130706944
W/OMXUtils: do not know color format 0x7f000789 = 2130708361
I/ACodec: [OMX.MTK.VIDEO.ENCODER.AVC] using color format 0x7f420888 in place of 0x7f420888
I/ACodec_vilte: set I frame rate
I/ACodec_vilte: set framerate
I/ACodec_vilte: setMTKParameters, width: 720
I/ACodec_vilte: setMTKParameters, height: 480
I/ACodec: setupAVCEncoderParameters with [profile: Baseline] [level: Level41]
I/ACodec: [OMX.MTK.VIDEO.ENCODER.AVC] cannot encode color aspects. Ignoring.
I/ACodec: [OMX.MTK.VIDEO.ENCODER.AVC] cannot encode HDR static metadata. Ignoring.
I/ACodec: setupVideoEncoder succeeded
I/ACodec_vilte: set I frame rate
I/ACodec_vilte: set framerate
I/ACodec_vilte: setMTKParameters, width: 720
I/ACodec_vilte: setMTKParameters, height: 480
I/general: [33] Configuring video encoder - 450 frames
I/general: [33] Video encoding in progress
D/MPEG4Writer: start+ 797
I/MPEG4Writer: limits: 4294967295/0 bytes/us, bit rate: -1 bps and the estimated moov size 3191 bytes
D/MPEG4Writer: start+ 2403
D/MPEG4Writer: start- 2481
D/MPEG4Writer: start- 964
I/MPEG4Writer: setStartTimestampUs: 166665
I/MPEG4Writer: Earliest track starting time: 166665
D/MPEG4Writer: Video mStartTimestampUs=166665us
D/MPEG4Writer: reset+ 1081
D/MPEG4Writer: Video track stopping. Stop source
D/MPEG4Writer: Video track source stopping
D/MPEG4Writer: Video track source stopped
I/MPEG4Writer: Received total/0-length (439/0) buffers and encoded 439 frames. - Video
D/MPEG4Writer: Video track stopped. Stop source
D/MPEG4Writer: Stopping writer thread
D/MPEG4Writer: 0 chunks are written in the last batch
D/MPEG4Writer: Writer thread stopped
I/MPEG4Writer: Ajust the moov start time from 166665 us -> 166665 us
I/MPEG4Writer: The mp4 file will not be streamable.
D/MPEG4Writer: reset- 1187
D/MPEG4Writer: reset+ 1081
D/MPEG4Writer: Video track stopping. Stop source
I/general: [33] MP4 video created successfully
答案1
得分: 1
视频为什么有损坏的帧？ 从查看共享的字节：
1) 每帧都是关键帧 <br>
我想知道为什么你的MP4的关键帧列表部分，stss原子，将每一帧都标记为“关键帧”（为什么没有P帧和B帧？），但后来我看到你的代码有这个命令：
mMediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
由于你正在进行复用（复制/粘贴现有媒体字节），这个媒体有它自己的间隔，应该写入输出的MP4文件中... 复用过程不需要设置新的关键帧间隔。
解决方案: <br>
尝试禁用（或注释//）该行，<br>
与相同的//禁用比特率设置。
2) 你的第一帧不是关键帧<br>
你发送给复用器的第一个视频帧不是关键帧。<br>它在MP4元数据中标记为关键帧（在stss原子中），但它的实际字节是P帧类型。这意味着：在起始码0, 0, 0, 1之后，你的NAL头被找到为十六进制：42，或十进制：65）。要注意，因为十进制65不等于十六进制65，所以如果你忘记转换并认为十进制65表示这个NAL单元是关键帧，可能会令人困惑。
解决方案: <br>
确保发送一个带有十进制的NAL单元：0, 0, 0, 1, 101（或十六进制：00, 00, 00, 01, 65）。
这个101表示它是一个关键帧NAL单元（头字节值是0x65或十进制值101）。
3) 你的SPS和PPS不正确 <br>
问题再次出在错误的SPS和PPS上... <br>
在MP4中，SPS/PPS数据放入avcC部分（其中包含“AVC配置”，而AVC只是H264的另一个名称）。
如果我手动覆盖avcC部分的一部分，将原始NALU的字节复制/粘贴到其中，那么你的损坏的MP4将显示出完美的图像（颜色不会混乱）。
解决方案: <br>
选项A: 尝试设置一个包含SPS和PPS的MediaFormat。<br>这意味着根据这些键为Codec-Specfic Data添加一个条目：
格式 CSD缓冲区＃0 CSD缓冲区＃1 CSD缓冲区＃2
H.264 AVC SPS（序列参数集） PPS（图像参数集） 未使用
选项B: 在最坏的情况下，你可能需要在Android代码创建文件后手动修复字节（例如：在文件创建代码完成后，运行另一个函数，找到并覆盖新创建的MP4中的约25字节的SPS和PPS部分）。希望不需要这样的解决方案。
我无法测试Android代码，请检查是否这个示例代码创建了一个可用的MP4：
(1) 在十进制和十六进制之间进行转换（因为字节是以十六进制格式写入的）
//# 检查字节的十进制值
int temp_int = (NALU_SPS[4] & 0xFF); // 其中NALU_SPS是一个数组。
System.out.println("NALU Decimal: " + temp_int );
//# 检查字节的十六进制值
String temp_str = Integer.toString( temp_int, 16 );
temp_str = (temp_str.length() < 2) ? ("0"+temp_str) : temp_str;
System.out.println("NALU Byte: 0x" + temp_str );
在双重检查字节值时，使用上述方法将你的数组值转换为十六进制格式，因为教程和H264规范将以十六进制格式提及字节值。
例如，0xFF是十六进制FF，十进制是255。
(2) 尝试这个MediaFormat设置。
//# 设置MediaCodec
mMediaFormat = MediaFormat.createVideoFormat(codecName, width, height);
mMediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, framerate);
//# 你的SPS/PPS在一个数组中。可以是：byte[] ...或其他：int[]
byte[] NALU_SPS = { 0x00, 0x00, 0x00, 0x01, 0x67, 0x42, 0x00, 0x29, 0xE2, 0x90, 0x16, 0x87, 0xB6, 0x06, 0xAC, 0x18, 0x04, 0x1B, 0x87, 0x89, 0x11, 0x50 };
byte[] NALU_PPS = { 0x00, 0x00, 0x00, 0x01, 0x68, 0xCE, 0x3C, 0x80 };
//# 发送SPS/PPS到复用器
mMediaFormat.setByteBuffer( "csd-0", ByteBuffer.wrap( NALU_SPS ) );
mMediaFormat.setByteBuffer( "csd-1", ByteBuffer.wrap( NALU_PPS ) );
<details>
<summary>英文:</summary>
**Why does the video have corrupted frames?** From a look at the [shared bytes](https://dropmefiles.com/rn6I0):
**1) Every frame is a Key-frame** <br>
I wondered why your MP4's section for listing keyframes, the `stss` atom, had **every single frame** listed as a type "Key-frame" (why no P-frames and B-frames?) but then I see that your code has this command:
mMediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
Since you are **muxing** (copy/pasting existing media bytes), that **media has its own interval** which should be written into the output MP4... A Muxing process does not need to be given a new keyframe interval.
**solution:** <br>
Test a possible fix of disabling (or comment `//`) that line, <br>
With same `//` disabling of the `bitrate` setting.
**2) Your first frame is not a keyframe**<br>
The first video frame that you send to the muxer is not a keyframe. <br>It is marked as a key-frame in the MP4 metadata (at the **stss** atom) but it's actual bytes are of type P-frame. This means: After **start-code** `0, 0, 0, 1` your NAL header is found to be hex: `42`, or decimal: `65`). Be aware because a decimal `65` is not same as hex `65` and so it can be confusing, if you forget to convert, and think that decimal `65` means this NALU is a keyframe.
**solution:** <br>
Make sure you send a NAL unit with decimals: `0, 0, 0, 1, 101` (or in hex: `00, 00, 00, 01, 65`).
The `101` marks it as a keyframe NALU (header byte value is: `0x65` or as decimal value: `101`).
**3) You have wrong SPS and PPS** <br>
Once again the problem is a wrong SPS and PPS... <br>
Inside the MP4, the SPS/PPS data goes into the **avcC** section (which holds the "AVC configuration", and AVC is just an alternate name for H264).
If I manually overwrite a part of the `avcC` section by copy/pasting over the bytes from the raw NALUs then your corrupt MP4 shows a perfect picture (not messy colours).
**solution:** <br>
**option A:** Try to set a `MediaFormat` with the SPS and PPS included. <br>This means putting an entry for [Codec-Specfic Data](https://developer.android.com/reference/android/media/MediaCodec#CSD) according to these **keys**:
Format CSD buffer #0 CSD buffer #1 CSD buffer #2
H.264 AVC SPS (Sequence Parameter Sets) PPS (Picture Parameter Sets) Not Used
**option B:** In a worst case scenario you might have to manually fix the bytes after the MP4 file is created by your Android code (_eg:_ after the file creation code finishes, run another function which finds and overwrites some circa 25 bytes of existing SPS & PPS inside the newly created MP4). Hopefully not needed as a solution.
I cannot test Android code, please check if this example code creates a working MP4:
**(1)** Converting between Decimals into Hex (since bytes are written as hex)
//# check Byte as Decimal
int temp_int = (NALU_SPS[4] & 0xFF); //where NALU_SPS is an Array.
System.out.println("NALU Decimal: " + temp_int );
//# check Byte as Hex
String temp_str = Integer.toString( temp_int, 16 );
temp_str = (temp_str.length() < 2) ? ("0"+temp_str) : temp_str;
System.out.println("NALU Byte: 0x" + temp_str );
Use the above to convert your Array values when double-checking their byte value, since tutorials and H264 Specification will mention the byte values in a Hex format.
A byte example like `0xFF` is hex `FF` and is decimal `255`.
**(2)** Try this `MediaFormat` setup.
//# Set up MediaCodec
mMediaFormat = MediaFormat.createVideoFormat(codecName, width, height);
mMediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, framerate);
//# Your SPS/PPS in an Array. Can be: byte[] ...or Else: int[]
byte[] NALU_SPS = { 0x00, 0x00, 0x00, 0x01, 0x67, 0x42, 0x00, 0x29, 0xE2, 0x90, 0x16, 0x87, 0xB6, 0x06, 0xAC, 0x18, 0x04, 0x1B, 0x87, 0x89, 0x11, 0x50 };
byte[] NALU_PPS = { 0x00, 0x00, 0x00, 0x01, 0x68, 0xCE, 0x3C, 0x80 };
//# send SPS/PPS to the Muxer
mMediaFormat.setByteBuffer( "csd-0", ByteBuffer.wrap( NALU_SPS ) );
mMediaFormat.setByteBuffer( "csd-1", ByteBuffer.wrap( NALU_PPS ) );
//# This might not be needed (since Muxer could get from SPS/PPS)
mMediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat);
//# test with these disabled
//mMediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, bitrate);
//mMediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
</details>
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论