gstreamer/gst-libs/gst/codecparsers
Marek Vasut 8239ff343f gsth264parser: Fix handling of NALs with emulation byte set
In case a set of NALs with emulation_prevention_three_byte is decoded using
hardware decoder like Hantro G1, wrong struct v4l2_ctrl_h264_decode_params
.dec_ref_pic_marking_bit_size is passed into the kernel, which results in
decoding artifacts. Subtract the number of emulation three bytes from the
.dec_ref_pic_m->bit_size to get the correct bit size and avoid having these
artifacts. Apply the exact same fix to slice->pic_order_cnt_bit_size as well.

The following NALs (7, 8, 6, 5) decode with artifacts,
.dec_ref_pic_marking_bit_size is set to 10 without this patch.
00000000  00 00 00 01 27 4d 00 20  89 8b 60 3c 04 bf 2e 02  |....'M. ..`<....|
00000010  d4 18 04 18 c0 c0 01 77  00 00 5d c1 7b df 05 00  |.......w..].{...|
00000020  00 00 01 28 ee 1f 20 00  00 01 06 05 10 b9 ed b9  |...(.. .........|
00000030  30 5d 21 4b 71 83 71 2c  10 a3 14 bb 29 80 00 00  |0]!Kq.q,....)...|
00000040  01 25 b8 00 05 00 00 03  03 7f fa 78 1e e7 fd fe  |.%.........x....|
                         ^^^^^^^^^^^^--- emulation 3 byte
00000050  b4 62 7a 31 ff 7d 81 fd  26 d8 62 b6 d6 25 46 ae  |.bz1.}..&.b..%F.|

The following NALs (7, 8, 6, 5) decode fine,
.dec_ref_pic_marking_bit_size is set to 2 without this patch.
00000000  00 00 00 01 27 4d 00 20  89 8b 60 3c 04 bf 2e 02  |....'M. ..`<....|
00000010  d4 18 04 18 c0 c0 01 77  00 00 5d c1 7b df 05 00  |.......w..].{...|
00000020  00 00 01 28 ee 1f 20 00  00 01 06 05 10 b9 ed b9  |...(.. .........|
00000030  30 5d 21 4b 71 83 71 2c  10 a3 14 bb 29 80 00 00  |0]!Kq.q,....)...|
00000040  01 25 b8 00 04 c0 00 03  7f fa 78 1e e7 fd fe b4  |.%........x.....|
00000050  62 7a 31 ff 7d 81 fd 26  d8 62 b6 d6 25 46 ae ce  |bz1.}..&.b..%F..|

Fixes: d0d65fa875 ("codecparsers: h264: record dec_ref_pic_marking() size")
Fixes: 0cc7d6f093 ("codecparsers: h264: record pic_order_cnt elements size")
Signed-off-by: Marek Vasut <marex@denx.de>
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2517>
2021-09-09 16:49:41 +00:00
..
codecparsers-prelude.h libs: fix API export/import and 'inconsistent linkage' on MSVC 2018-09-24 14:45:56 +01:00
dboolhuff.AUTHORS codecparsers: vp8: add GStreamer native utilities. 2014-04-18 16:32:40 +02:00
dboolhuff.c libs: fix API export/import and 'inconsistent linkage' on MSVC 2018-09-24 14:45:56 +01:00
dboolhuff.h codecparsers: Add another G_GNUC_INTERNAL 2015-12-30 11:54:14 +02:00
dboolhuff.LICENSE codecparsers: vp8: add GStreamer native utilities. 2014-04-18 16:32:40 +02:00
dboolhuff.PATENTS codecparsers: vp8: add GStreamer native utilities. 2014-04-18 16:32:40 +02:00
gstav1parser.c av1parser: Fix tile size calculation 2021-06-29 07:12:27 +00:00
gstav1parser.h codecparsers: av1: fix underflow in last_frame_idx 2021-07-12 17:49:17 +00:00
gsth264parser.c gsth264parser: Fix handling of NALs with emulation byte set 2021-09-09 16:49:41 +00:00
gsth264parser.h h264parser: Add APIs to allow update SPS/PPS 2020-04-09 19:33:56 +00:00
gsth265parser.c h2645parser: Catch overflows in AVC/HEVC NAL unit length calculations 2021-03-24 09:22:48 +00:00
gsth265parser.h h265parse: Add missing const qualifier 2020-11-02 18:20:32 +00:00
gstjpeg2000sampling.c jpeg2000parse, openjpeg: add support for YCrCb 4:1:1 sampling 2021-06-14 11:05:45 +02:00
gstjpeg2000sampling.h jpeg2000parse, openjpeg: add support for YCrCb 4:1:1 sampling 2021-06-14 11:05:45 +02:00
gstjpegparser.c libs: fix API export/import and 'inconsistent linkage' on MSVC 2018-09-24 14:45:56 +01:00
gstjpegparser.h codecparsers: GST_EXPORT -> GST_CODEC_PARSERS_API 2018-03-13 12:34:24 +00:00
gstmpeg4parser.c documentation: fixed a heap o' typos 2019-11-05 09:11:25 -05:00
gstmpeg4parser.h libs: codecparser: add missing unstable api warning 2020-03-16 18:07:54 +01:00
gstmpegvideometa.c gst: don't use volatile to mean atomic 2021-03-22 14:34:36 +11:00
gstmpegvideometa.h codecparsers: GST_EXPORT -> GST_CODEC_PARSERS_API 2018-03-13 12:34:24 +00:00
gstmpegvideoparser.c codecparsers: fix debug category initialisation 2019-05-25 15:29:25 +02:00
gstmpegvideoparser.h gstmpegvideoparser: Documentation fixes 2019-07-05 00:17:58 +10:00
gstvc1parser.c vc1parser : fix a miswrite 2019-11-11 02:12:34 +00:00
gstvc1parser.h documentation: fixed a heap o' typos 2019-11-05 09:11:25 -05:00
gstvp8parser.c codecparsers: vp8parser: clear the frame_hdr before parsing. 2020-09-25 17:07:43 +08:00
gstvp8parser.h libs: codecparser: add missing unstable api warning 2020-03-16 18:07:54 +01:00
gstvp8rangedecoder.c libs: fix API export/import and 'inconsistent linkage' on MSVC 2018-09-24 14:45:56 +01:00
gstvp8rangedecoder.h codecparsers: GST_EXPORT -> GST_CODEC_PARSERS_API 2018-03-13 12:34:24 +00:00
gstvp9parser.c vp9parser: Add new API for parsing superframe info 2020-03-02 01:31:38 +09:00
gstvp9parser.h codecparsers: vp9parser: Use macro to define the size of filter_level in Segmentation. 2021-07-02 08:44:15 +00:00
meson.build Meson: Use pkg-config generator 2020-10-23 11:14:18 -04:00
nalutils.c nalutils: Introduce NAL writer helper methods 2020-03-30 07:59:10 +00:00
nalutils.h nalutils: Improve slightly the error trace 2020-06-20 19:47:49 +00:00
parserutils.c libs: fix API export/import and 'inconsistent linkage' on MSVC 2018-09-24 14:45:56 +01:00
parserutils.h codecparsers: Spread some G_GNUC_INTERNAL in various places 2015-12-30 11:40:59 +02:00
vp8utils.c libs: fix API export/import and 'inconsistent linkage' on MSVC 2018-09-24 14:45:56 +01:00
vp8utils.h codecparsers: Spread some G_GNUC_INTERNAL in various places 2015-12-30 11:40:59 +02:00
vp9utils.c codecparsers: vp9: move license blurb into header and remove aux files 2015-12-30 11:04:32 +00:00
vp9utils.h codecparsers: vp9: move license blurb into header and remove aux files 2015-12-30 11:04:32 +00:00