Avion: AV Decoding / Encoding Library (RFC)

[arisona]

Hi everyone

As mentioned earlier in this forum, I started hacking on an AV library that aims at using the native media frameworks on each OS. Now, a first prototype with OS X / AV Foundation support is working, and I would be thankful for input re API, Features, etc.

The code is available at:

https://github.com/arisona/avion

There is a simple player using lwjgl3/OpenGL for video playback.


From the README.md

Avion is an audio/video decoding and encoding framework for Java and C++. In contrast to other frameworks, it uses of the native media frameworks on individual platforms (AVFoundation on Mac OS, Microsoft Media Foundation on Windows, ffmpeg on Linux).

The main goal is to provide a simple, intuitive API for accessing raw audio and video data, or for encoding media files from raw data. It does not include higher level functionality such as GUI elements for media players. Further, focus is on standard audio/video formats, and we do not aim at supporting legacy and rarely used format. Finally, by using the native frameworks, one does not have to deal with decoder / encoder licensing issues.

A typical application example is to decode video files, and use the video frames as OpenGL textures, e.g. for VJing tools or game engines.

Features

• Simple, immediate API for the major desktop platforms
• Multi-threaded decoding of Audio and Video streams
• Support for standard formats and containers (e.g. H.264; .mov, .mp4)
• Support for video capture devices (web cams, etc.)

TODO

• API finalization
• Windows and Linux support (Microsoft Media Foundation, ffmpeg)
• Video capture support
• Encoding support

RFC

In order to finalize the API, requests for changes, features, etc. are welcome. Also, if someone is experienced with Microsoft Media Foundation and would like to start hacking, please be in touch, or fork and I'll be happy to pull.

[Cornix]

Where is the benefit over using VLCJ (the java bindings for libVLC)?

[arisona]

The main difference is probably the use case: For most of my applications, I simply need to decode a movie and obtain the individual frames, e.g. for animated textures or similar. For this, I don't need a fully fledged AV library, with support for all possible formats etc. Compare to the philosophy of the STB image loading library vs other imaging libraries.

Other points that made me start this project: Size is very small (20KB for the current Mac OS implementation). You don't need to download extra software (you need to get VLC or libVLC in order to use VLCJ if I understand correctly). No licensing issues for commercial products (see ffmpeg). Finally, VLCJ seems to have some threading issues when decoding multiple assets at the same time (https://github.com/caprica/vlcj/blob/master/src/test/java/uk/co/caprica/vlcj/test/multi/TestMultiPlayer.java) while for example AVFoundation does pretty well (still need to see for MS media foundation though...)

[Kai]

I think it's a great idea to provide a thin wrapper over the OS'es media playback functionality.
One tiny downside of this approach, however, is that even different versions of the same operating system support different codecs and may support the same codec differently, making your video render perfectly on Windows 10, but not at all on Windows 7. So your option here is to target your application for a particular operating system version, which may not be a big issue for a particular use case.
That's why from a compatibility and realibility aspect, I think a wrapper for an existing platform-independent playback solution such as libavcodec or libvlc would be a better solution, IMO.
However, I would really like to have a look at the Windows Media Foundation. Found some nice examples in the Windows 7 SDK and here: https://msdn.microsoft.com/en-us/library/windows/desktop/ff728866(v=vs.85).aspx

[arisona]

Yes, agree - compatibility between platform frameworks will remain an issue, and needs to be tested. Also, whether the API as it's designed right now will work for all frameworks needs to be seen.

Next, I'll hack in ffmpeg support, which is quite straightforward. ffmpeg could then still be used as an alternative on all platforms if requested and the shared libraries are found.

Thanks for the Media Foundation links, will have a look.