Some devices only ever keep one buffer available in the GL queue resulting in
multiple calls to release_output_buffer only causing one frame to be rendered.
If there is a queue after amcvideodec (even playsink's small one), then
multiple buffers are pushed but only a small fraction of them are actually
rendered on time. The rest will either render some number of frames ahead of
where they are meant to be or timeout waiting for a frame that's already been
rendered.
Solved by moving the release_output_buffer into the sync_meta the is pushed
downstream. When downstream renders, the custom sync implementation attempts
to release the current buffer (if not already released) and render. Once the
frame has been rendered to the screen, the next frame is released and is
hopefully available by the time the next frame is to be rendered.
This fixes a perceived frame jitter in the output.
This provides a performance and power usage improvement by removing
the texture copy from an OES texture to 2D texture.
The flow is as follows
1. Generate the output buffer with the required sync meta with the incrementing
push counter and OES GL memory
1.1 release_output_buffer (buf, render=true) and push downstream
2. Downstream waits for on the sync meta (timed wait) or drops the frame (no wait)
2.1 Timed wait for the frame number to reach the number of frame callbacks fired
2.2 Unconditionally update the image when the wait completes (success or fail).
Sets the affine transformation matrix meta on the buffer.
3. Downstream renders as usual.
At *some* point through this the on_frame_callback may or may not fire. If it
does fire, we can finish waiting early and render. Otherwise we have to
wait for a timeout to occur which may cause more buffers to be pused into the
internal GL queue which siginificantly decreases the chances of the
on_frame_callback to fire again. This is because the frame callback only occurs
when the internal GL queue changes state from empty to non-empty.
Because there is no way to reliably correlate between the number of buffers
pushed and the number of frame callbacks received, there are a number of
workarounds in place.
1. We self-increment the ready counter when it falls behind the push counter
2. Time based waits as the frame callback may not be fired for a certain frame.
3. It is assumed that the device can render at speed or performs some QoS of
the interal GL queue (which may not match the GStreamer QoS).
It holds that we call SurfaceTexture::updateTexImage for each buffer pushed
downstream however there's no guarentee that updateTexImage will result in
the exact next frame (it could skip or duplicate) so synchronization is not
guaranteed to be accurate although it seems to be close enough to be unable
to discern visually. This has not changed from before this patch. The current
requirement for synchronization is that updateTexImage is called at the point in
time when the buffers is to be rendered.
https://bugzilla.gnome.org/show_bug.cgi?id=757285
In JNI_OnLoad() we will already get the Java VM passed and could
just directly use that. gstreamer_android-1.0.c will now provide
this to us.
Reason for this is that apparently not all Android system are
providing the JNI functions to get the currently running Java VMs, so
we would fail to get. With this we will always be able to get the Java
VM on such systems.
We only need that if no Java VM is running yet, and all usual cases,
i.e. when calling GStreamer from an actual Android app, there will already
be a Java VM we can just use.
It seems like some phones come without that symbol, let's hope they come
with the other symbol but for now don't make a missing JNI_CreateJavaVM fatal.
OMX.Exynos. codecs are existing on some devices like the
Galaxy S5 mini, and cause random crashes (of the device,
not the app!) and generally misbehave. That specific device
has other codecs that work with a different name, but let's
just give them marginal rank in case there are devices that
have no other codecs and these are actually the only working
ones
On some devices there are codecs that don't start with OMX., while
there are also some that do. And on some of these devices the ones
that don't start with OMX. just crash during initialization while
the others work. To make things even more complicated other devices
have codecs with the same name that work and no alternatives.
So just give a lower rank to these non-OMX codecs and hope that
there's an alternative with a higher rank.
Also stagefright gives codecs starting with OMX. a higher rank too and
considers other codecs that don't start with OMX. as software codecs.
We now fill GErrors for everything that could throw an exception, and method
calls now always return a gboolean and their value in an out-parameter to
distinguish failures from other values.
The first buffer does not contain more garbage than any other MP3 decoder
outputs and we don't really know how much we have to drop or not.
After this change the output has the same duration as with mad.
gst_pad_get_pad_template_caps() returns a reference which is unreferenced,
so creating a copy using gst_caps_copy() results in a reference leak.
Also remove the incorrect comment to avoid confusion in the future.
Fixes https://bugzilla.gnome.org/show_bug.cgi?id=734533
Check if libnativehelper is loaded in the process and if
it has these awful wrappers for JNI_CreateJavaVM and
JNI_GetCreatedJavaVMs that crash the app if you don't
create a JniInvocation instance first. If it isn't we
just fail here and don't initialize anything.
See this code for reference:
https://android.googlesource.com/platform/libnativehelper/+/master/JniInvocation.cpp
On Samsung Galaxy S4 it is impossible to have more than one
hardware decoder at the same time. If we do not release it
explicitly the GC only releases it whenever the whole application
is finished not whenever the activity is finished and thus a player
will not be able to work correctly
gst_amc_color_format_copy will copy in/out a frame resides at a
GstAmcBuffer. Lots of codes in gst_amc_video_*_fill_buffer are moved to
this new function.
Some hack logic needs also to be present in create_src|sink_caps, for
working around some broken codecs. These hacks are hidden
in color_format/video_format conversion -- the prototypes of these
functions are also changed to include more args for hack judgement.
Also in case of multi-color_formats mapped to one video_format, then
map that video_format back will not give the original color_format, which
causes gst_amc_codec_configure failed with something like
'does not support color format N'.
The new prototype involves with GstAmcCodecInfo and mime, which
ensures the converted color_format is supported by the codec.
A COLOR_FormatYCbYCr to GST_VIDEO_FORMAT_YUY2 mapping is also added, in
order to work around bugs in OMX.k3.video.decoder.avc(which incorrectly
reports supporting COLOR_FormatYCbYCr, which is actually
COLOR_FormatYUV420SemiPlanar). There are already hacks for this in
gst_amc_video_format_to_color_format, gst_amc_color_format_to_video_format
and gst_amc_color_format_info_set, but the codec will still not work(be
ignored because of "has unknown color formats") without adding this mapping.
If the application is using the new ART runtime it will otherwise
load dalvik and start a dalvik VM next to the ART VM.
Does not work very well obviously.
Some audio decoders (at least the MP3 decoder on MTK based devices) outputs
raw audio in batches of multiple audio frames. We need to handle that
properly, otherwise the base class will be kind of unhappy.
Add a new color format seen on my Galaxy S3
(OMX_SEC_COLOR_FormatNV12Tiled = 0x7fc00002) to the table,
but don't actually implement it - the decoder doesn't choose it.
Remove an assert that makes the plugin fail noisily and take the app down
if it sees a color format it doesn't recognise (just skip the codec instead)
Modify the debug output when plugin scanning to print color format info to
make this sort of thing easier in the future.
Flushing the decoder invalidates all buffers, so it should be done
after quiting the decoding loop. Otherwise we can jump into
"failed_release" and stop everything
https://bugzilla.gnome.org/show_bug.cgi?id=711156
