Improve closure of gaps in frame_num by strictly following and trying
to fill them with previous reference frames. So, they are now tracked
thus avoiding insertion of dummy ("greenish") frames.
If the new picture to be added to the DPB is not a first field, then
it shall be the second field of the previous picture that was added
before.
This removes the need for dpb_find_picture() now that we track the
immediately preceding decoded picture, in decode order.
When a dummy "other-field" is inserted, it is assumed to inherit the
reference flags from the first field, and the sliding window decoded
reference picture marking process is also executed so that corrupted
frames are moved out as early as possible.
While doing so, we also try to output frames that now contain a single
valid field picture, prior to inserting any other picture into the DPB.
Note: this may be superfluous currently based on the fact that dpb_add()
combines the two most recent pairable fields, but this process would be
further simplified later on.
Mark the picture as "corrupted" if it is reconstructed from corrupted
references or if those references are fake, e.g. resulting from lost
frames.
This is useful for notifying the upper layer, or downstream elements,
that the decoded frame may contain artefacts.
https://bugzilla.gnome.org/show_bug.cgi?id=703921
Fix uninitialized variables when decoding SPS and PPS NAL units from
"codec-data" buffers. This is particularly important when seeking ops
are involved, and the new persistent states are used more often.
https://bugzilla.gnome.org/show_bug.cgi?id=750094
Implement decoding process for gaps in frame_num (8.5.2). This
also somewhat supports unintentional loss of pictures.
https://bugzilla.gnome.org/show_bug.cgi?id=745048https://bugzilla.gnome.org/show_bug.cgi?id=703921
Original-patch-by: Wind Yuan <feng.yuan@intel.com>
[fixed derivation of POC, ensured clone is valid for reference,
actually fixed detection of gaps in FrameNum by PrevRefFrameNum]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Try to identify missing first fields too, thus disregarding any
intermediate gaps in frames. We also assume that we keep the same
field sequence, i.e. if previous frames were in top-field-first
(TFF) order, then so are subsequent frames.
Note that insertion of dummy first fields need to operate in two
steps: (i) create the original first field that the current field
will inherit from, and (ii) submit that field into the DPB prior
to initializing the current (other) field POC values but after any
reference flag was set. i.e. copy reference flags from the child
(other field) to the parent (first field).
https://bugzilla.gnome.org/show_bug.cgi?id=745048
Interlaced H.264 video frames always have two fields to decode and
display. However, in some cases, e.g. packet loss, one of the field
can be missing. This perturbs the reference picture marking process,
whereby the number of references available in DPB no longer matches
the expected value.
This patch adds initial support for missing field within a decoded
frame. The current strategy taken is to find out the nearest field,
by POC value, and with the same parity.
https://bugzilla.gnome.org/show_bug.cgi?id=745048
Try to maintain a "top-field-first" (TFF) flag, even if the H.264 standard
does not mandate it. This will be useful for tracking missing fields, and
also for more correct _split_fields() implementation for frames in the DPB.
Don't try to decode pictures until the first I-frame is received within
the currently active sequence. There is no point is decoding and then
displaying frames with artifacts.
Fix decoding of end_of_seq() NAL unit so that to not submit the current
picture for decoding again. This is pretty vintage code that dates back
before the existing of the whole decoder units machinery.
One issue that could be arising if that code was kept is that we could
have submitted a picture, and subsequently a GstVideoCodec frame, twice.
Once without the decode_only flag set, and once with that flag set. The
end result is that the GstVideoDecoder would release the codec frame
twice, thus releasing stale data.
In short, the piece of code that is removed by this patch is for once
completely obsolete for a while, and secondly error-prone in corner
cases.
Use the SEI pic_timing() message to track and propagate down the repeat
first field (RFF) flag. This is only initial support as there is one
other condition that could induce the RFF flag, which is not handled
yet.
Fix the decoding process for picture order count type 0 when the previous
picture had a memory_management_control_operation = 5. In particular, fix
the actual variable type for prev_pic_structure to hold the full bits of
the picture structure.
In practice, this used to work though, due to the underlying type used to
express a gboolean.
Use the SEI pic_timing() message to track the pic_struct variable when
present, or infer it from the regular slice header flags field_pic_flag
and bottom_field_flag. This fixes temporal sequence ordering when the
output pictures are to be displayed.
https://bugzilla.gnome.org/show_bug.cgi?id=739291
When a DPB flush is required, e.g. at a natural and of stream or issued
explicitly through an IDR, try to detect any frame left in the DPB that
is interlaced but does not contain two decoded fields. In that case, mark
the picture as having a single field only.
This avoids a hang while decoding tv_cut.mkv.
Simplify the dpb_output() function to exclusively rely on the frame store
buffer to output, since this is now always provided. Besides, also fix
cases where split fields would not be displayed.
This is a regression from f48b1e0.
Cope with latest changes from codecparsers/h264. It is now required
to explicitly clear the GstH264PPS structure as it could contain
additional allocations (slice_group_ids).
Slightly optimize decoding process by submitting the current VA surface
for decoding earlier to the hardware, and perform the reference picture
marking process and DPB update process afterwards.
This is a minor optimization to let the video decode engine kick in work
earlier, thus improving parallel resources utilization.
Fix decoding of interlaced streams where a first field (e.g. B-slice)
was immediately output and the current decoded field is to be paired
with that former frame, which is no longer in DPB.
https://bugzilla.gnome.org/show_bug.cgi?id=701340
Optimize the process to detect new pictures or start of new access
units by checking if the previous NAL unit was the end of a picture,
or the end of the previous access unit.
Add support for MVC streams with multiple SPS and subset SPS headers
emitted regularly, e.g. at around every I-frame. Track the maximum
number of views in ensure_context() and really reset the DPB size to
the expected value, always. i.e. even if it decreased. dpb_reset()
only cares of ensuring the DPB allocation.
Fix the compaction process when the DPB is cleared for a specific
view, i.e. fix the process of filling in the holes resulting from
removing frame buffers matching the current picture.
Fix search for a picture in the DPB that has a lower POC value than
the current picture. The dpb_find_lowest_poc() function will return
a picture with the lowest POC in DPB and that is marked as "needed
for output", but an additional check against the actual POC value
of the current picture is needed.
This is a regression from 1c46990.
https://bugzilla.gnome.org/show_bug.cgi?id=732130
Fix dpb_clear() to clear previous frame buffers only if they actually
exist to begin with. If the decoder bailed out early, e.g. when it
does not support a specific profile, that array of previous frames
might not be allocated beforehand.
We can avoid scanning for start codes again if the bitstream is fed
in NALU chunks. Currently, we always scan for start codes, and keep
track of remaining bits in a GstAdapter, even if, in practice, we
are likely receiving one GstBuffer per NAL unit. i.e. h264parse with
"nal" alignment.
https://bugzilla.gnome.org/show_bug.cgi?id=723284
[use gst_adapter_available_fast() to determine the top buffer size]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
The `vaapipostproc' element could never determine if the H.264 stream
was interlaced, and thus always assumed it to be progressive. Fix the
H.264 decoder to report interlace-mode accordingly, thus allowing the
vaapipostproc element to automatically enable deinterlacing.
Factor out the removal process of unused inter-view only reference
pictures from the DPB, prior to the possible insertion of the current
picture.
Ideally, the compiler could still opt for generating two loops. But
at least, the code is now clearer for maintenance.
Improve process for the removal of pictures from DPB before possible
insertion of the current picture (C.4.4) for H.264 MVC inter-view only
reference components. In particular, handle cases where picture to be
inserted is not the last one of the access unit and if it was already
output and is no longer marked as used for reference, including for
decoding next view components within the same access unit.
While invoking the DPB bumping process in presence of many views,
it could be necessary to output previous pictures that are ready,
in a whole. i.e. emitting all view components from the very first
view order index zero to the very last one in its original access
unit; and not starting from the view order index of the picture
that caused the DPB bumping process to be invoked.
As a reminder, the maximum number of frames in DPB for MultiView
High profile with more than 2 views is not necessarily a multiple
of the number of views.
This fixes decoding of MVCNV-4.264.
Let the utility layer handle dynamic growth of the inter-view pictures
array. By definition, setting a new size to the array will effectively
grow the array, but would also fill in the newly created elements with
empty entries (NULL), thus also increasing the reported length, which
is not correct.
When decoding Multiview High profile streams with a large number of
views, it is not possible to make the VAPictureParameterBufferH264.
ReferenceFrames[] array hold the complete DPB, with all possibly
active pictures to be used for inter-view prediction in the current
access unit.
So reduce the scope of the ReferenceFrames[] array to only include
the set of reference pictures that are going to be used for decoding
the current picture. Basically, this is a union of all RefPicListX[]
array, for all slices constituting the decoded picture.
The inter-view reference components and inter-view only reference
components that are included in the reference picture lists shall
be considered as not being marked as "used for short-term reference"
or "used for long-term reference". This means that reference flags
should all be removed from VAPictureH264.flags.
This fixes decoding of MVCNV-2.264.
If the VA driver exposes ad-hoc H.264 MVC profiles, then we have to
be careful to detect profiles changes and not reset the underlying
VA context erroneously. In MVC situations, we could indeed get a
profile_idc change for every SPS that gets activated, alternatively
(base-view -> non-base view -> base-view, etc.).
An improved fix would be to characterize the exact profile to use
once and for all when SPS NAL units are parsed. This would also
allow for fallbacks to a base-view decoding only mode.
Optimize lookups of view ids / view order indices by caching the result
of the calculatiosn right into the GstVaapiParserInfoH264 struct. This
terribly simplifies is_new_access_unit() and find_first_field() functions.
Add safe fallbacks for MVC profiles:
- all MultiView High profile streams with 2 views at most can be decoded
with a Stereo High profile compliant decoder ;
- all Stereo High profile streams with only progressive views can be
decoded with a MultiView High profile compliant decoder ;
- all drivers that support slice-level decoding could normally support
MVC profiles when the DPB holds at most 16 frames.
In order to have a stricter conforming implementation, we need to carefully
detect access unit boundaries. Additional operations could be necessary to
perform at those boundaries.
Detect the first VCL NAL unit of a picture for MVC, based on the
view_id as per H.7.4.1.2.4. Note that we only need to detect new
view components.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Always cache the previous NAL unit so that we could check whether
there is a Prefix NAL unit immediately preceding the current slice
or IDR NAL unit. In that case, the NAL unit metadata is copied into
the current NAL unit. Otherwise, some default values are inferred,
tentatively. e.g. view_id shall be set to 0 and inter_view_flag to 1.
[infer default values for slice if previous NAL was not a Prefix]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Allow decoding for base views of MVC encoded streams. For now, just skip
the slice extension and prefix NAL units, and skip non-base view frames.
Signed-off-by: Xiaowei Li <xiaowei.a.li@intel.com>
[fixed memory leak, improved check for MVC NAL units]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Factor out process by which the decoded picture with the lowest POC
is found, and possibly output. Likewise, the storage and marking of
a reference decoded, or non-reference decoded picture, into the DPB
could also be simplified as they mostly share the same operations.
