什么是android中的MediaCodec，MediaExtractor和MediaMuxer？

huangapple
go评论104阅读模式
英文:

What is MediaCodec, MediaExtractor and MediaMuxer in android?

问题

以下是翻译好的内容：

在安卓中，"MediaCodec"、"MediaExtractor"和"MediaMuxer"是什么意思？虽然我不是专业的视频从业者，但我基本了解编码和解码的含义。我需要知道每个类的功能以及它们在哪些用例中使用。

我还想知道：

  • 如果我想从相机预览中提取帧并创建包含某些编辑（如速度）的视频文件，我应该使用哪些类，它们是如何协同工作的？

  • 如果我想创建一个类似于ExoPlayer（不需要所有功能，只是一个简单的Dash自适应流媒体播放器），我应该使用哪些类，它们是如何协同工作的？

希望您能回答。谢谢。

英文:

What does MediaCodec, MediaExtractor and MediaMuxer mean in android? I am not a video person but I do know what encoding and decoding means, at a basic level. I need to know what are the functions of each classes and at which use cases are they used. <br/>I would also like to know:<br/>
<ul>
<li><b>If I want to extract frames from a camera preview and create a video file along with some editing (like speed), which classes should I use and how does it work together?</b></li>

<li><b>If I want to create a Video Player like Exoplayer (not all the functions but a simple Dash adaptive streaming player) , which classes should I use and how does it work together?</b></li>
</ul>
<br/>
Hope you will answer. Thank You.

答案1

得分: 1

让我首先说，如果你不了解视频编解码的工作原理，就很难理解这些 API。我建议你研究一下编码器和解码器的工作原理。

我将为每个部分提供一个过度简化的解释。

MediaCodec：

MediaCodec 类可用于访问低级媒体编解码器，即编码器/解码器组件。它是 Android 低级多媒体支持基础架构的一部分。

因此，MediaCodec 处理视频数据包/缓冲区的解码或编码，并负责与编解码器的交互。

下面是如何初始化 MediaCodec 的示例：

  1. // 通过传递 MIME 类型创建 Mediacodec 实例。它将选择适合此 MIME 类型的最佳编解码器
  2. MediaCodec mDecoder = MediaCodec.createDecoderByType(mimeType);
  3. // 传递 MediaFormat 实例和输出/渲染 Surface
  4. mDecoder.configure(format, surface, null, 0);
  5. mDecoder.start();

然后你可以开始传递数据给 MediaCodec，就像这样：

  1. ByteBuffer[] inputBuffers = mDecoder.getInputBuffers();
  2. int index = mDecoder.dequeueInputBuffer(timeout);
  3. // 检查缓冲区是否可用
  4. if (index >= 0) {
  5.     // 获取已出队的缓冲区
  6.     ByteBuffer buffer = inputBuffers[index];
  7.     // 获取样本数据大小，以确定我们是否应该继续排队更多的缓冲区或者标记流结束
  8.     int sampleSize = mExtractor.readSampleData(buffer, 0);
  9.     if (sampleSize < 0) {
  10.         // 标记 EOS，例如当你到达视频结尾时会发生这种情况。
  11.         mDecoder.queueInputBuffer(inIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
  12.     } else {
  13.         // 排队已出队的缓冲区，并传递解码器的样本时间
  14.         mDecoder.queueInputBuffer(index, 0, sampleSize, mExtractor.getSampleTime(), 0);
  15.         mExtractor.advance();
  16.     }
  17. }

然后你从输出缓冲区中出队并将其释放到你的 Surface 上：

  1. BufferInfo frameInfo = new BufferInfo();
  2. int index = mDecoder.dequeueOutputBuffer(frameInfo, timeout);
  4. switch (index) {
  5.     case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED:
  6.         break;
  7.     case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
  8.         MediaFormat newFormat = mDecoder.getOutputFormat();
  9.         break;
  10.     case MediaCodec.INFO_TRY_AGAIN_LATER:
  11.         break;
  12.     default:
  13.         break;
  14. }
  16. // 现在你可以将帧推送到 Surface 上
  17. // 这是你可以通过让线程短暂休眠来控制播放速度的地方
  19. if (index > 0) {
  20.     mDecoder.releaseOutputBuffer(bufferIndex, true);
  21. }

MediaExtractor：

MediaExtractor 用于从数据源中提取分离的、通常编码的媒体数据。

文档的描述已经很清楚了。

请看下面，我已经添加了注释以便更好理解：

  1. // 初始化提取器
  2. MediaExtractor mExtract = new MediaExtractor();
  3. mExtract.setDataSource(mSource);
  5. // 选择/设置视频轨道（如果可用）
  6. int trackIndex = selectVideoTrack(mExtract);
  7. if (trackIndex < 0)
  8.     throw new IOException("找不到轨道");
  9. mExtract.selectTrack(trackIndex);
  11. // 提取器现在已经可以使用了
  13. // 获取轨道格式
  14. mFormat = mExtractor.getTrackFormat(trackIndex);
  16. // 获取缓冲区大小以检查缓冲区是否可用
  17. // 这将由 MediaCodec 用于确定缓冲区是否可用
  18. sampleSize = mExtractor.readSampleData(buffer, 0);

MediaMuxer：

MediaMuxer 用于复用基本流。目前 MediaMuxer 支持将输出作为 MP4、Webm 和 3GP 文件。自 Android Nougat 开始，它还支持在 MP4 中复用 B 帧。

这也很直白。它用于创建视频/音频文件，例如将两个视频文件合并在一起。

英文:

Let me start of by saying that it is hard to understand this API`s if you don't understand how video encoding/decoding works. I would suggest doing research about how encoders/decoders work.

I will provide an oversimplified explanation of each.

MediaCodec:

>MediaCodec class can be used to access low-level media codecs, i.e. encoder/decoder components. It is part of the Android low-level multimedia support infrastructure

So MediaCodec handles the decoding or encoding of the video packets/buffers and is responsible for the interaction with the codec.

Here is an example of how to Initialize MediaCodec:

  1. // Create Mediacodec instance by passing a mime type. It will select the best codec for this mime type
  2. MediaCodec mDecoder = MediaCodec.createDecoderByType(mimeType); 
  3. // Pass an instance on MediaFormat and the output/rendering Surface
  4. mDecoder.configure(format, surface, null, 0); 
  5. mDecoder.start();

You would then start passing buffers to MediaCodec, like this:

  1. ByteBuffer[] inputBuffers = mDecoder.getInputBuffers();
  2. int index = mDecoder.dequeueInputBuffer(timeout);
  3. // Check if buffers are available
  4. if (index &gt;= 0) { 
  5.     // Get dequeued buffer
  6.     ByteBuffer buffer = inputBuffers[index]; 
  7.     // Get sample data size to determine if we should keep queuing more buffers or signal end of stream
  8.     int sampleSize = mExtractor.readSampleData(buffer, 0); 
  9.     if (sampleSize &lt; 0) { 
  10.         // Signal EOS, this happens when you reach the end if the video, for example.
  11.         mDecoder.queueInputBuffer(inIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM); 
  12.     } else { 
  13.         // Queue the dequeued buffer and pass the extractors sample time
  14.         mDecoder.queueInputBuffer(index, 0, sampleSize, mExtractor.getSampleTime(), 0);
  15.         mExtractor.advance(); 
  16.     } 
  17. }

You then dequeue the output buffer and release it to your surface:

  1. BufferInfo frameInfo = new BufferInfo(); 
  2. int index mDecoder.dequeueOutputBuffer(frameInfo, timeout);
  4. switch (index) { 
  5.     case
  6.         MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED: 
  7.     break; 
  8.     case
  9.         MediaCodec.INFO_OUTPUT_FORMAT_CHANGED: 
  10.         MediaFormat newFormat = mDecoder.getOutputFormat(); 
  11.     break; 
  12.     case
  13.         MediaCodec.INFO_TRY_AGAIN_LATER: break; default: 
  14.     break; 
  15. }
  17. // You can now push the frames to the surface
  18. // This is where you can control the playback speed, you can do this by letting your thread sleep momentarily
  20. if (index &gt; 0) {
  21.     mDecoder.releaseOutputBuffer(bufferIndex, true);
  22. }

MediaExtractor:

>MediaExtractor facilitates extraction of demuxed, typically encoded, media data from a data source.

The documentation description is self-explanatory.

Have a look below, I've added comments to make it more understandable:

  1. // Initialize the extractor
  2. MediaExtractor() mExtract = new MediaExtractor(); mExtract.setDataSource(mSource);
  4. // Select/set the video track (if available)
  5. int trackIndex = selectVideoTrack(mExtract);
  6. if(trackIndex &lt; 0) 
  7.     throw new IOException(&quot;Can&#39;t find track&quot;);
  8. mExtract.selectTrack(trackIndex);
  10. // The extractor is now ready to be used
  12. // Get the track format
  13. mFormat = mExtractor.getTrackFormat(trackIndex);
  15. // Get buffer size to check if a buffer is available
  16. // This will be used by MediaCodec to determine if buffers are available
  17. sampleSize = mExtractor.readSampleData(buffer, 0);

MediaMuxer:

>MediaMuxer facilitates muxing elementary streams. Currently MediaMuxer supports MP4, Webm and 3GP file as the output. It also supports muxing B-frames in MP4 since Android Nougat.

This is self-explanatory once again. It's used to create a video/audio file. For example, merging two video files together.

huangapple
  • 本文由 发表于 2020年10月23日 00:57:29
  • 转载请务必保留本文链接：https://go.coder-hub.com/64487008.html
  • android
  • android-camera
  • android-camera2
  • exoplayer
  • java
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定