Commit graph

48 commits

Author SHA1 Message Date
Seungha Yang
d1af0b6386 codecs: h264decoder: Fix wrong annotations
What intended was GstH264Decoder actually.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1453>
2020-07-21 22:40:40 +09:00
Seungha Yang
bf659379cf codecs: h264decoder: Update document with Since marks
To make documentation CI happy with the newly added APIs.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1449>
2020-07-21 18:30:07 +09:00
Seungha Yang
973a2569d4 codecs: h264decoder: Rename API arguments
Although it's not public ones, make them consistent with vfunc
for them to be nicer.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1449>
2020-07-21 17:53:29 +09:00
Nicolas Dufresne
6523c5bcb6 h264decoder: Add ability to lookup by system_frame_number
This is to avoid having to create a reference chain in decoders from
GstVideoCodecFrame to GstH264PIcture to implementation wrapper.

So this change introduces:

 - gst_h264_dpb_remove_outputed (dpb)
 - gst_h264_dpb_get_picture(dpb, system_frame_num)
 - gst_h264_decoder_get_picture (dec, system_frame_num)

In order to ensure that frames can be looked up during the draining
process, we now first remove all (including reference) frames that
have been outputed but are still in the DPB. Then for each remaining
buffers, we remove it from the DPB to reach reference 1 and output it.

Previously we could take all not outputed outside of the DPB which would
prevent lookup by the base class.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1449>
2020-07-21 17:18:42 +09:00
Seungha Yang
82fb55bdff codecs: h264decoder: Make GstVideoCodecFrame hold the last reference of the buffer
... when gst_video_decoder_finish_frame() is called.

Some subclasses attach GstBuffer to GstH264Picture as an user data
which will increase reference count of the buffer. It would result
to buffer copy per every frame.

Background here is, GstVideoDecoder wants writable output buffer for
GstMeta handling, and if the output buffer is not writable
(i.e., reference count is not one), the buffer will be copied.
Even if underlying GstMemory wouldn't be copied, buffer copy operation
will introduce extra memory allocation overhead which is not optimal.

By this modification, subclass might be able to receive the last
reference to GstH264Picture

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1449>
2020-07-21 17:18:42 +09:00
Seungha Yang
e921a07715 codecs: h264decoder: Pass GstVideoCodecFrame to output_picture()
All subclasses are retrieving list to get target output frame, which
can be done by baseclass. And pass the ownership of the GstH264Picture
to subclass so that subclass can clear implementation dependent resources
before finishing the frame.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1449>
2020-07-21 17:18:42 +09:00
Seungha Yang
7a024a740f codecs: h264decoder: Don't attach extra ref of GstH264Picture to GstVideoCodecFrame
The lifecycle of GstH264Picture is being managed by our DPB
implementation. If it's still required, subclass can do that
by itself in the new_picture() method.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1449>
2020-07-21 17:18:42 +09:00
Seungha Yang
14f6799f93 codecs: h264decoder: Drop non-decodable frames
Otherwise GstVideoDecoder baseclass will hold the corresponding broken frames
until new reset event happens such as FLUSH or downwards state change.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1450>
2020-07-20 21:00:48 +09:00
Seungha Yang
487f9a08de codecs: h264decoder: Fix for DPB size calculation
Some bitstreams might require more DPB size than that of what we've
calculated.

This change should've been part of initial commit of h264 stateless
codec implementation but it was missed.

See also https://chromium-review.googlesource.com/c/chromium/src/+/760276/

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1385>
2020-06-30 22:20:29 +00:00
Víctor Manuel Jáquez Leal
c1b8802c71 codecs: h264picture: use g_array_remove_index_fast()
This algorithm is faster than the normal one. The problem is it
disorders the elements in the dpb, but it does not seem to mess
decoders.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1378>
2020-06-29 12:36:38 +00:00
Víctor Manuel Jáquez Leal
9ff332e15d codecs: h264decoder: update max_dpb_frames only if VUI is present
There are some streams, with HRD, where the the calculated
max_dpb_frames is zero (max_dpb_mbs is less than size mb). In order to
get the dbp size it is required to rely on the VUI parameters if they
are present.

According to the spec Annex E.2.1

**max_dec_frame_buffering** specifies the required size of the HRD
decoded picture buffer (DPB) in units of frame buffers.  It is a
requirement of bitstream conformance that the coded video sequence
shall not require a decoded picture buffer with size of more than
Max(1, max_dec_frame_buffering) frame buffers to enable the output of
decoded pictures at the output times specified by dpb_output_delay of
the picture timing SEI messages. The value of max_dec_frame_buffering
shall be greater than or equal to max_num_ref_frames. An upper bound
for the value of max_dec_frame_buffering is specified by the level
limits in clauses A.3.1, A.3.2, G.10.2.1, and H.10.2.

When the max_dec_frame_buffering syntax element is not present, the
value of max_dec_frame_buffering shall be inferred as follows:

– If profile_idc is equal to 44, 86, 100, 110, 122, or 244 and
constraint_set3_flag is equal to 1, the value of
max_dec_frame_buffering shall be inferred to be equal to 0.

– Otherwise (profile_idc is not equal to 44, 86, 100, 110, 122, or 244
or constraint_set3_flag is equal to 0), the value of
max_dec_frame_buffering shall be inferred to be equal to MaxDpbFrames.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1381>
2020-06-27 14:05:25 +02:00
Seungha Yang
a2f2ebbe01 codecs: h265picture: Don't leak pic_list GArray
Equivalent to the commit 7b8c071f9c

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1337>
2020-06-11 23:42:47 +00:00
Seungha Yang
2e0a6cc020 h264decoder: Disallow multiple slice group as we don't support FMO
Even though it might be supported by accelerator, baseclass is not
ready to support it.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1291>
2020-05-22 19:07:33 +00:00
Víctor Manuel Jáquez Leal
206d900573 codecs: h264decoder: chain finalize vmethod
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1238>
2020-05-19 16:57:09 +00:00
Nicolas Dufresne
28e30c8493 codecs: h264decoder: Use calculated values for max_pic_num/frame_num
The parser pre-calculate these already, just use them.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1238>
2020-05-19 16:57:09 +00:00
Víctor Manuel Jáquez Leal
d2aae6bb96 codecs: h264decoder: ref pic lists as decode_slice parameters
Pass reference picture lists to decode_slice() vmethods

Change gstv4l2codech264dec and gstnvh264dec accordingly.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1238>
2020-05-19 16:57:09 +00:00
Víctor Manuel Jáquez Leal
990392d595 codecs: h264decoder: handle reference picture lists
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1238>
2020-05-19 16:57:08 +00:00
Nicolas Dufresne
7e63d1f5ad codecs: h264decoder: Port from GList to GArray
Using glist requires a lot of small allocation at runtime and also
it comes with a slow sort algorithm. As we play with that for very
frame and slices, use GArray instead. Note that we cache some arrays
in the instance as there is no support for stack allocated arrays
in GArray.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1238>
2020-05-19 16:57:08 +00:00
Nicolas Dufresne
f8736626a9 codecs: h264decoder: Make get_long_ref_by_pic_num() transfer none
We don't use the extra reference, so let's just avoid the extra
ref/unref.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1238>
2020-05-19 16:57:08 +00:00
Nicolas Dufresne
a20147f255 codecs: h264decoder: Make get_short_ref_by_pic_num() transfer none
We don't use the extra reference, so let's just avoid the extra
ref/unref.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1238>
2020-05-19 16:57:08 +00:00
Seungha Yang
24367cbff0 h264decoder: Allow frame gap even if it's not allowed by SPS
It's most likely the case where some frames were dropped for some reason
(e.g., broken stream, network loss or so). However, decoder might be able to
decode following frames even if some frames are visually broken.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1232>
2020-05-01 01:55:21 +09:00
Seungha Yang
1d38728d05 h264decoder: Perform low-latency outputting only for live pipeline
Frequent outputting might drop throughput-wise performance.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1211>
2020-04-28 13:23:53 +00:00
Seungha Yang
00ad871628 h264decoder: Output decoded pictures as soon as possible
In case of IDR, any previously decoded pictures must be drained
before the IDR and POC of IDR should be zero. So we can output
IDR immediately. Also, when POC of current picture is expected to be
the next output POC, decoder can output the picture as well
without waiting.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1211>
2020-04-28 13:23:53 +00:00
Víctor Manuel Jáquez Leal
851557af20 codecs: fix gir generation
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1218>
2020-04-26 19:26:32 +02:00
Víctor Manuel Jáquez Leal
23e99a7a77 codecs: set sys frame num before new_picture()
Derive classes might need to operate on the current frame at their
new_picture() vmethod, so it would be needed to set system_frame_number
on picture before calling the vmethod.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1209>
2020-04-23 19:54:17 +00:00
Seungha Yang
8325e42ade codecs: Fix some documentation
Add missing "transfer full" annotation, etc

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1079>
2020-04-22 23:20:44 +00:00
Seungha Yang
a89b819191 codecs: Add new baseclass for VP8 decoder
This implemenation is similar to VP9 but much simpler than it.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1079>
2020-04-22 23:20:44 +00:00
Seungha Yang
9524e6adec vp9decoder: Fix small typo
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1079>
2020-04-22 23:20:44 +00:00
Seungha Yang
d72fcbe18a codecs: Reorganize the order of vfunc
... to the order in which it is actually called

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1079>
2020-04-22 23:20:44 +00:00
Seungha Yang
ba07768ca4 h264decoder: Don't handle gap frame num for the first picture
If the first picture is not IDR, it would have non-zero frame_num
but it's not gap. We should skip gap frame handling in that case
2020-04-20 13:55:49 +00:00
Seungha Yang
6a7ea29084 h264decoder: Add some debug messages 2020-04-20 13:55:49 +00:00
Víctor Manuel Jáquez Leal
a3395167a1 codecs: h264picture: typo in GstH264PictureField enum 2020-04-19 18:40:12 +02:00
Seungha Yang
5a29917a63 codecs: Add minimal state validation
... to prevent requesting decoding before the preparation.
For instance, baseclass should not request decoding a picture if there
is no parsed valid headers, since subclass is most likely
not ready to decoding it.
2020-04-09 19:33:56 +00:00
Thibault Saunier
b9e0ccd8b7 codecs: Add basic documentation stubs 2020-04-08 08:47:46 -04:00
Nicolas Dufresne
b79c949789 codecs: h264: Fix DPB size calculation
As per specification in A.3.1 h) and A.3.2 f), the maximum size of the DPB is
16. Fix the maximum in the fine and fix the formula to use MIN instead of MAX
so that we no longer always use the maximum for the profile/level.
2020-04-07 07:53:27 +00:00
Nicolas Dufresne
7b8c071f9c codecs: h264dpb: Don't leak pic_list GArray
The contents was cleared, but the array was never released.
2020-03-31 09:34:05 -04:00
Nicolas Dufresne
bd7db13412 codecs: h264: Do not ignore end_picture() return value
If decoding failed because end_picture() failed, set the picture to
nonexisting, this way output_picture() will be skipped. This avoids confusing
special cases in output_picture() implementation.
2020-03-31 09:34:05 -04:00
Seungha Yang
5609d8751c h265decoder: Pass max_dpb_size to new_sequence vfunc
same as we are doing in h264decoder
2020-03-08 16:11:35 +09:00
Seungha Yang
ba3430a059 h265decoder: Fix for output and removal picture from DPB
See C.5.2.2 Output and removal of pictures from the DPB.

If the number of pictures in the DPB is greater than or equal to
sps_max_dec_pic_buffering_minus1[HighestTid] + 1, then the picture
should be outputted.
2020-03-07 12:20:03 +09:00
Seungha Yang
8b0d5e1272 meson: codecs: Don't install header
This library is not intended to be public. APIs are likely to
change over time and should not be disclosed to people yet.
2020-03-06 16:44:01 +09:00
Seungha Yang
fdf06fa363 h264decoder: Rename some internals
No more a part of d3d11 implementation
2020-03-05 15:27:42 +00:00
Seungha Yang
f487648b03 vp9decoder: Update document
s/GstH264Picture/GstVp9Picture/g and minor update since this baseclass
is no more d3d11 specific one.
2020-03-05 15:27:42 +00:00
Seungha Yang
82f1d5e8be codecs: Change output_picture() to mandatory implementation
GstVideoCodecFrame is expected to be consumed by subclass
per output_picture(). So the implementation cannot be optional.
2020-03-05 15:27:42 +00:00
Seungha Yang
83b056cb77 h265decoder: Port to nal type classification macro
... and remove namespaceless macro methods from baseclass
2020-03-05 23:22:39 +09:00
Nicolas Dufresne
56c3b40c30 h264picture: Add system_frame_num
This allow cross-referencing GstH264Picture and GstVideoCodecFrame.
2020-03-05 03:06:16 +00:00
Nicolas Dufresne
c6b8157dd6 codecs: Pass the max_dpb_size to new_segment virtual
On new_segment, the decoder is expected to negotiate. The decoder may want to
pre-allocate the needed buffers. Pass the max_dpb_size as this is needed to
determin how many buffers should be allocated.
2020-03-05 03:06:16 +00:00
Nicolas Dufresne
f3f60f5a4c codecs: Add missing auto cleanup funcs 2020-03-05 03:06:16 +00:00
Nicolas Dufresne
00d04784d3 Move CODEC base classes into it's own library
This introduce a library which contains a set of base classes which
handles the parsing and the state tracking for the purpose of decoding
different CODECs. Currently H264, H265 and VP9 are supported. These
bases classes are used to decode with low level decoding API like DXVA,
NVDEC, VDPAU, VAAPI and V4L2 State Less decoders. The new library is
named gstreamer-codecs-1.0 / libgstcodecs.
2020-03-05 03:06:16 +00:00