Commit graph

80 commits

Author SHA1 Message Date
Marek Vasut
5d2d602e4a v4l2codecs: Avoid QBUF/DQBUF struct timeval .tv_usec wrap-around at frame 1000000
When decoding stream using hardware V4L2 decoder element, in any of the
currently supported formats, the decoding will fail once frame number
1000000 is reached. The reported error clearly indicates a wrap-around
occured, instead of receiving decoded frame 1000000, frame 0 is received
from the hardware V4L2 decoder driver.

The problem is actually not in the driver itself, but rather in gstreamer,
which uses `struct v4l2_buffer` member `.timestamp` in a special way. The
timestamp of buffers with encoded data added to the SINK (input) queue of
the driver is copied by the driver into matching buffers with decoded data
added to the SOURCE (output) queue of the driver. In fact, the timestamp
is not a timestamp at all, but rather in this special case, only part of
it is used as an incrementing frame counter.

The `.timestamp` is of type `struct timeval`, which is defined in
`sys/time.h` [1]. Only the `tv_usec` member of this structure is used
for the incrementing frame counter. However, suseconds_t tv_usec [2]
may be limited to range [-1, 1000000]:
"
[XSI] The type suseconds_t shall be a signed integer type capable of
      storing values at least in the range [-1, 1000000].
"
Therefore, once frame 1000000 is reached, a rollover occurs and decoding
fails.

Fix this by using both `struct timeval` members, `.tv_sec` and `.tv_usec`
with matching modular arithmetic, this way the failure would occur again
just short of 2^84 frames, which should be plenty.

[1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_time.h.html
[2] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_types.h.html

A test case using stateless hardware h264 decoder, the WARN/ERROR output
in gstreamer log indicates a failure occurred. With this change, that
error no longer occurs and the WARN/ERROR are not present:
```
pc$ gst-launch-1.0 videotestsrc num-buffers=1001001 pattern=6 ! \
                   video/x-raw,width=16,height=16,format=I420 ! \
                   x264enc ! filesink location=/tmp/test.h264

dut$ GST_DEBUG="*:3" gst-launch-1.0 filesrc location=/tmp/test.h264 ! \
                                    h264parse ! v4l2slh264dec ! fakesink
...
0:03:51.393677606 12111     0x370df400 WARN      \
  v4l2codecs-decoder gstv4l2decoder.c:1157:gst_v4l2_request_set_done:<v4l2decoder2> \
  Requested frame 1000000, but driver returned frame 0.
0:03:51.394140597 12111     0x370df400 WARN      \
  v4l2codecs-decoder gstv4l2decoder.c:1157:gst_v4l2_request_set_done:<v4l2decoder2> \
  Requested frame 1000001, but driver returned frame 1.
0:03:51.394425216 12111     0x370df400 WARN      \
  v4l2codecs-decoder gstv4l2decoder.c:1157:gst_v4l2_request_set_done:<v4l2decoder2> \
  Requested frame 1000002, but driver returned frame 2.
0:03:51.394665211 12111     0x370df400 WARN      \
  v4l2codecs-decoder gstv4l2decoder.c:1157:gst_v4l2_request_set_done:<v4l2decoder2> \
  Requested frame 1000003, but driver returned frame 3.
0:03:51.394785833 12111     0x370df400 WARN      \
  v4l2codecs-h264dec gstv4l2codech264dec.c:1059:gst_v4l2_codec_h264_dec_output_picture:<v4l2slh264dec0> \
  error: Failed to decode frame 1000000
ERROR: from element /GstPipeline:pipeline0/v4l2slh264dec:v4l2slh264dec0: Failed to decode frame 1000000
```

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5598>
2023-11-04 15:51:45 +01:00
Nicolas Dufresne
41f478e97d v4l2codecs: h265: Fix entry_point_offsets array leak
This array was being leaked.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5499>
2023-10-17 15:06:26 -04:00
Nicolas Dufresne
88fd9908fe v4l2codecs: Fix tiled formats stride conversion
While adding arbitrary tile support, a round up operation was badly
converter. This caused the Y component of the stride to be 0. This
eventually lead to a crash in glupoad preceded by the following
assertion.

  gst_gl_buffer_allocation_params_new: assertion 'alloc_size > 0' failed

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5458>
2023-10-11 14:13:53 +00:00
Seungha Yang
97fc02cfe3 av1decoder: Port to GstCodecPicture struct
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5285>
2023-09-08 11:51:23 +00:00
Seungha Yang
a73c6d7fb6 vp9decoder: Port to GstCodecPicture struct
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5285>
2023-09-08 11:51:23 +00:00
Seungha Yang
4571fac946 vp8decoder: Port to GstCodecPicture struct
... and remove unused "pts" variable from GstVp8Picture struct

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5285>
2023-09-08 11:51:23 +00:00
Seungha Yang
a5ddf7af3b mpeg2decoder: Port to GstCodecPicture struct
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5285>
2023-09-08 11:51:23 +00:00
Seungha Yang
6e7cab43be h265decoder: Port to GstCodecPicture struct
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5285>
2023-09-08 11:51:23 +00:00
Seungha Yang
ea3dfadbed h264decoder: Port to GstCodecPicture struct
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5285>
2023-09-08 11:51:23 +00:00
Nicolas Dufresne
8710911e6b v4l2codecs: Enable Mediatek 10bit 4:2:0 support
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3444>
2023-08-08 16:08:16 +00:00
Daniel Almeida
ec188eb82e v4l2codecs: Add AV1 decoder
This patch adds AV1 stateless v4l2 decode support using the
new v4l2 request API to communicate with hardware accelerators.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1011>
2023-06-16 01:30:40 +00:00
James Cowgill
70ff80a873 v4l2codecs: Always chain up to parent decide_allocation function
The `gst_video_decoder_negotiate_pool` function expects the
`decide_allocation` function to always provide a pool and will fail to
negotiate if the pool is missing. If we return immediately (even if we
don't need to do anything special) negotiation will fail if the
downstream element does not propose a pool.

Fix by chaining up to the default `decide_allocation` function which
adds a fallback pool if one was not already proposed.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4630>
2023-06-15 17:02:28 +00:00
Nicolas Dufresne
0f5d10f617 v4l2codecs: Enable Verisilicon 10bit 4:2:0 support
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3447>
2023-06-15 14:32:32 +00:00
Nicolas Dufresne
aea74db1a2 v4l2: Sync headers to current media_stage
commit d78b9d6671decdaedb539635b1d0a34f8f5934f8

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3447>
2023-06-15 14:32:32 +00:00
Nicolas Dufresne
5f2a664415 v4l2codecs: h265: Set num_delta_pocs_of_ref_rps_idx
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4413>
2023-06-13 14:34:45 +00:00
Nicolas Dufresne
6a7835e106 v4l2codecs: Extend the API with num_delta_pocs_of_ref_rps_idx
This value is an alternative to short_term_ref_pic_set_size and can be used
to parse the RPS portion of a slice header, instead of skipping over it.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4413>
2023-06-13 14:34:45 +00:00
Michael Grzeschik
5d82deb2c5 uvcsink: add new bin element around v4l2sink
This patch adds an element to stream video data to an uvc video gadget.
The element handles the uvc events STREAMON, STREAMOFF, SETUP and DATA.
to start, stop and configure the video buffer flow by the use of pad
probes. It works with linux kernels of versions higher than v6.1.

The element makes use of the v4l2sink proxy property v4l2sink::device
to locate the corresponding device to parse the configfs for additional
data.

The code in uvc.c is basically derived from /lib/uvc.c in
https://git.ideasonboard.org/uvc-gadget.git.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1304>
2023-06-01 19:19:05 +00:00
Nicolas Dufresne
e709e2d97c meson: Add a wrap file for libgudev
And allow fallback to it.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4447>
2023-04-19 22:47:19 +00:00
Sebastian Dröge
ccad9a7338 plugins: Fix various trivial clang compiler warnings
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4210>
2023-03-18 16:16:55 +02:00
Tim-Philipp Müller
81716a1097 gst-plugins-bad: re-indent with GNU indent 2.2.12
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4182>
2023-03-17 03:18:54 +00:00
Nicolas Dufresne
5980fb76e7 video: Add arbitrary tile dimensions support
In current tile representation, only tiles with power of two
width and height in bytes are supported. This limitation
prevents adding more complex tiles formats.

In this patch, we deprecate tile_ws and tile_hs from GstVideoFormatInfo and
replace if with an array of GstVideoTileInfo. Each plane tiles are then
described with their pixels width/height, line stride and total size.
The helper gst_video_format_info_get_tile_sizes() that depends on the
deprecated API is also being removed. This can simply be removed as it wasn't
in any stable release yet.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3424>
2022-11-18 22:59:29 +00:00
Nicolas Dufresne
d3c5fc815e v4l2codecs: Allow output caps to be updated
This change allow output caps to be updated even though we stay in
streaming state. This is needed so that any upstream updated to fields
like framerate, hdr data, etc. can result in a downstream caps event
being pushed.

Previously, any of these changes was being ignored and the downstream
caps would not reflect it.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3328>
2022-11-16 13:12:38 +00:00
Christopher Obbard
0c83ddcdf7 v4l2codecs: Warn user when no media devices are found
Currently if the user is not able to access the devices under /dev/media*,
either due to no media devices present on the system or simply no permission
to access the device, v4l2codecs initialises with no features or debug messages.

Since calling `GST_DEBUG="v4l2*:7" gst-inspect-1.0 v4l2codecs` is a typical way
to diagnose why element(s) failed to enumerate, we should be more verbose here
when the user is not able to access any /dev/media* device. So print a simple
debug message in this case to aid debugging.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3088>
2022-09-29 19:12:14 +00:00
Thibault Saunier
6a4425e46a meson: Call pkgconfig.generate in the loop where we declare plugins dependencies
Removing some copy pasted code

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2970>
2022-09-01 21:17:35 +00:00
Nicolas Dufresne
316b182a86 v4l2codecs: h264: Reset the number of slices on picture start
This way, if we aborted a frame in the middle, we don't endup thinking
there is multiple slices to send.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1079>
2022-07-22 10:21:03 -04:00
Nicolas Dufresne
c2144ec586 v4l2codecs: h264: Fix initialization of the slice array
The function g_array_sized_new() leaves the len to 0, but the slice
implementation assumes it would be set to 4. Sending multiple slices is
not yet support for H.264 as no driver needed it yet, but if that code
was to be used it would have overflowed as the array would never grow as
multiple 0 by 2 always results in 0.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1079>
2022-07-22 10:21:03 -04:00
Nicolas Dufresne
a707d984ae v4l2codecs: h264: Rename count variable to num_controls
This is the counter for the used control slot. The term num_controls was
introduced in the HEVC decoder and felt like a better choice.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1079>
2022-07-22 10:21:02 -04:00
Nicolas Dufresne
a2b219c7e1 v4l2codecs: hevc: Add an API check
This adds an API check and bump recommended base kernel version to 5.20.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1079>
2022-07-22 10:21:02 -04:00
Benjamin Gaignard
0eb7028570 v4l2codecs: Add v4l2 stateless H265 decoder
Add v4l2 stateless H265 decoder.

Co-authored-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
Co-authored-by: Sebastian Fricke <sebastian.fricke@collabora.com>
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1079>
2022-07-22 10:21:02 -04:00
Benjamin Gaignard
70941696ed v4l2codecs: Increase minimum payload size
Fluster MAXBINS_{A-C}_TI_5 HEVC tests files require to store
a payload larger than usual given their resolutions.
Increase the minimum payload size to handle them correctly.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1079>
2022-07-22 10:21:02 -04:00
Nicolas Dufresne
f1c98e2c9e v4l2codecs: Align v4l2 headers with HEVC final uAPI
HEVC uAPI have landed for kernel 5.10.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1079>
2022-07-22 10:21:02 -04:00
Benjamin Gaignard
d3a01eccae v4l2codecs: enumerate all possible formats
Some decoder may also allow to (down) scale the decoded video.
Use VIDIOC_ENUM_FRAMESIZES iotcl to enumerate the possible output
formats resolutions.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2589>
2022-06-29 19:18:16 +00:00
Benjamin Gaignard
a6f8cc4b95 v4l2codecs: Add P010 pixel format
Copy V4L2_PIX_FMT_P010 define from linux header.
V4L2_PIX_FMT_P010 is the little endian definition of P010 so map
it GST_VIDEO_FORMAT_P010_10LE.
Add it v4l2 default video formats to allows v4l2 decoders to
enumerate and use it.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2590>
2022-06-13 16:44:29 +02:00
Dmitry Osipenko
fd694d840a v4l2codecs: h264: Set frame type flags
V4L spec now requires decode_params flags to be set in accordance to the
frame's type. In particular this is required by H.264 decoder of NVIDIA
Tegra SoC to operate properly. Set the flags based on type of parsed
slices.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1757>
2022-05-06 13:35:59 +00:00
Seungha Yang
6a4c42c03f codecs: Signal required DPB size for AV1,MPEG2,VP8, and VP9 via new_sequence()
Make all codecs consistent so that subclass can know additional DPB
size requirement depending on render-delay configuration regardless
of codec. Note that render-delay feature is not implemented for AV1
yet but it's planned.

Also, consider new_sequence() is mandatory requirement, not optional

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2343>
2022-05-03 14:17:49 +00:00
Nicolas Dufresne
ab839f3642 v4l2codecs: Port to the new tile size helper
This will complete the scaling of width/height shift scaling
as only a partial implementation was made.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2190>
2022-05-02 19:10:43 +00:00
Nicolas Dufresne
c9d8b8a972 v4l2codecs: Detect missing M2M_HOLD_CAPTURE_BUF
Produce an error if we try to use the feature of holding capture buffer
but it is not supported by the driver. Ingoring this can lead to stalls
as the driver will run-out of capture buffer to decode into. This
affects slice decoders but also split-field interlaced decoding.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2009>
2022-03-25 21:40:28 +00:00
Nicolas Dufresne
efa360ce82 v4l2codecs: h264: Set other field poc if available
Whenever the other field POC is available, pass it back to the
driver. This should not be strictly required, but it makes things
easier to debug and more forgiven.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2009>
2022-03-25 21:40:28 +00:00
Nicolas Dufresne
dcd62c35f0 v4l2codecs: Trace request by it FD
This helps when crossing the debug logs with the kernel logs.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2009>
2022-03-25 21:40:28 +00:00
Nicolas Dufresne
d24dc74170 v4l2codecs: h264: Fix dpb entry flag FLAG_FIELD
The logic to guess back this value was broken. Use the value
now saved into the picture instead. This was tested using
LibreELEC patched 5.15 kernel, and fixed the interlaced decode
issues.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2009>
2022-03-25 21:40:28 +00:00
Benjamin Gaignard
98cb44ffc6 v4l2codecs: Fix memory leak
Free pending_requests array when releasing decoder to avoid memory leak

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2000>
2022-03-21 16:06:05 +00:00
Dmitry Osipenko
248f143fa3 v4l2codecs: Support YUV420M format
YUV420M format of V4L essentially is I420 with separate memory planes.
It's used by NVIDIA Tegra V4L decoder. Support this format by v4l2codecs.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1727>
2022-02-21 21:58:19 +03:00
Nicolas Dufresne
8883809ef9 v4l2codecs: Enable MediaTek MM21 (NV12_16L32S)
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1567>
2022-02-20 22:32:55 +00:00
Nicolas Dufresne
3005dbc55a v4l2codecs: Add support for multi planar buffers
V4L2 have some pixel formats that produces 2 dmabuf, this add
support for these format.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1567>
2022-02-20 22:32:55 +00:00
Nicolas Dufresne
bab9041c4b Port plugins to gst_video_format_info_extrapolate_stride()
This reduces code duplication and simplify addition of new
pixel formats into related plugins.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1567>
2022-02-20 22:32:55 +00:00
Nicolas Dufresne
cf3d64129b v4l2codecs: vp9: Fix reset_frame_context parameter
It was assumed that the kernel parameters would match with the bitstream value
but instead the author when with another set of value. Surprisingly, this
makes no difference with the resulting fluster score.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1748>
2022-02-18 23:55:59 +00:00
Nicolas Dufresne
5a9df9e850 v4l2codecs: vp9: Only fill compressed headers if needed
Fixes: 13944cf3ee ("v4l2codecs: vp9: Make compressed hdr control optional")
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1748>
2022-02-18 23:55:59 +00:00
Tim Mooney
54a1f60077 v4l2: include <sys/ioccom.h> on Illumos
Needed for _IOR/_IORW

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1524>
2022-02-18 21:34:49 +00:00
Dmitry Osipenko
b757d12cff v4l2codecs: h264: Correct scaling matrix ABI check
Scaling matrix V4L UAPI control not presents on NVIDIA Tegra, the default
matrix should be used in this case. Mark scaling matrix presence optional.

Fixes: 47bfa71530 ("v4l2codecs: h264: Improve ABI check ")
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1726>
2022-02-16 13:16:17 +00:00
Nicolas Dufresne
7a9e11ae1b v4l2codecs: vp9: Fix duplicating a duplicated picture
Duplicating a picture what was already a dup was leading to a crash. Rename
the custom picture flags as HOLDS_BUFFER to make its meaning clear. Then save
then ref and store the picture as userdata, so it can be obtained when
duplicating. Finally, mark the doplicated as HOLDS_BUFFER to avoid thinking it
holds a request.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1681>
2022-02-09 22:37:55 +00:00