mirrors/gstreamer
mirrors/gstreamer
1
1
Fork 0
mirror of https://gitlab.freedesktop.org/gstreamer/gstreamer.git synced 2024-12-03 06:56:46 +00:00
Code Issues 1 Projects Releases Wiki Activity
gstreamer/subprojects/gst-devtools/validate/launcher/testsuites/check.py

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

283 lines
20 KiB
Python
Raw Normal View History

Revert "Revert "validate:launcher: Add an app handler for unit tests described in meson"" This reverts commit 5656e2a1b208155be36cf16b08a9ffcc95694328.
2017-01-03 16:01:31 +00:00
# -*- Mode: Python -*- vi:si:et:sw=4:sts=4:ts=4:syntax=python
#
# Copyright (c) 2016,Thibault Saunier <thibault.saunier@osg.samsung.com>
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with this program; if not, write to the
# Free Software Foundation, Inc., 51 Franklin St, Fifth Floor,
# Boston, MA 02110-1301, USA.
"""
GStreamer unit tests
"""
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
from launcher import utils
Revert "Revert "validate:launcher: Add an app handler for unit tests described in meson"" This reverts commit 5656e2a1b208155be36cf16b08a9ffcc95694328.
2017-01-03 16:01:31 +00:00
check: Move CI specific ignored tests into their own list Instead of ignoring the tests always, check if a CI env var is present and only ignore the tests then. Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2023-09-28 17:16:37 +00:00
import os
Revert "Revert "validate:launcher: Add an app handler for unit tests described in meson"" This reverts commit 5656e2a1b208155be36cf16b08a9ffcc95694328.
2017-01-03 16:01:31 +00:00
TEST_MANAGER = "check"
validate: check: Fix known-not-leaky to work Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3940>
2020-02-26 01:19:19 +00:00
KNOWN_NOT_LEAKY = r'^gst-devtools.*|^gstreamer.*|^gst-plugins-base|^gst-plugins-ugly|^gst-plugins-good'
validate:launcher: Add a way to specify a set of tests to run under the leak tracer https://bugzilla.gnome.org/show_bug.cgi?id=767856
2017-02-08 20:46:23 +00:00
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
# These tests take very long compared to what they add, so let's skip them.
LONG_VALGRIND_TESTS = [
(r'check.[a-z-]*.generic_states.test_state_changes_down_seq', 'enough to run one of the sequences'),
(r'check.[a-z-]*.generic_states.test_state_changes_up_seq', 'enough to run one of the sequences',),
ci: switch the Fedora base image to f34 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2020-07-10 09:16:35 +00:00
(r'check.[a-z-]*.generic_states.test_state_changes_up_and_down_seq', 'enough to run the sequences'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gstreamer.gst_gstelement.test_foreach_pad$', '48s'),
(r'check.gstreamer.gst_gstinfo.info_post_gst_init_category_registration$', '21s'),
(r'check.gstreamer.gst_gstsystemclock.test_resolution$', '60s'),
(r'check.gstreamer.libs_aggregator.test_infinite_seek$', '20s'),
(r'check.gstreamer.libs_aggregator.test_infinite_seek_50_src$', '20s'),
(r'check.gstreamer.libs_gstharness.test_harness_element_ref$', '20s'),
(r'check.gstreamer.pipelines_simple_launch_lines.test_2_elements$', '58s'),
(r'check.gstreamer.pipelines_stress.test_stress$', '54s'),
(r'check.gstreamer.pipelines_stress.test_stress_preroll$', '27s'),
ci: switch the Fedora base image to f34 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2020-07-10 09:16:35 +00:00
(r'check.gst-plugins-base.elements_appsrc.test_appsrc_limits', '53.37s'),
(r'check.gst-plugins-base.elements_appsrc.test_appsrc_send_event_before_buffer', '49.25s'),
(r'check.gst-plugins-base.elements_appsrc.test_appsrc_send_event_before_sample', '51.39s'),
(r'check.gst-plugins-base.elements_appsrc.test_appsrc_send_event_between_caps_buffer', '56.13s'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-plugins-base.elements_appsrc.test_appsrc_block_deadlock$', '265.595s'),
(r'check.gst-plugins-base.elements_audioresample.test_fft$', '91.247s'),
(r'check.gst-plugins-base.elements_audioresample.test_timestamp_drift$', '141.784s'),
validate:check: Fix some mistakes translating regex from gitlab-ci.yml
2019-04-17 20:46:31 +00:00
(r'check.gst-plugins-base.elements-videoscale.*', 'superlong'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-plugins-base.libs_video.test_overlay_blend$', '74.096s'),
(r'check.gst-plugins-base.libs_video.test_video_color_convert$', '345.271s'),
(r'check.gst-plugins-base.libs_video.test_video_formats$', '70.987s'),
(r'check.gst-plugins-base.libs_video.test_video_size_convert$', '56.387s'),
(r'check.gst-plugins-base.elements_audiointerleave.test_audiointerleave_2ch_pipeline_$', '5 *51.069s'),
(r'check.gst-plugins-base.elements_multifdsink.test_client_kick$', '46.909s'),
(r'check.gst-plugins-base.elements_videotestsrc.test_all_patterns$', '?'),
(r'check.gst-plugins-base.elements_videotestsrc.test_patterns_are_deterministic$', '?'),
(r'check.gst-plugins-good.elements_shapewipe.test_general$', '325s'),
(r'check.gst-plugins-good.elements_videocrop.test_cropping$', '245s'),
validate:check: Fix some mistakes translating regex from gitlab-ci.yml
2019-04-17 20:46:31 +00:00
(r'check.gst-plugins-good.elements_videomixer.*', '30s (alsodeprecated)'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-plugins-good.elements_rtp_payloading.rtp_jpeg_packet_loss$', '109s'),
(r'check.gst-plugins-good.elements_videomixer.test_play_twice_then_add_and_play_again$', '55s'),
(r'check.gst-plugins-good.pipelines_effectv.test_quarktv$', '53s'),
(r'check.gst-plugins-good.elements_deinterlace.test_mode_disabled_passthrough$', '52s'),
(r'check.gst-plugins-good.elements_deinterlace.test_mode_auto_deinterlaced_passthrough$', '28s'),
(r'check.gst-plugins-good.elements_deinterleave.test_2_channels_caps_change$', '30s'),
(r'check.gst-plugins-good.elements_deinterleave.test_2_channels$', '22s'),
(r'check.gst-plugins-good.elements_rtpjitterbuffer.test_fill_queue$', '22s'),
(r'check.gst-plugins-good.elements_splitmux.test_splitmuxsink_async$', '20s'),
(r'check.gst-plugins-good.elements_videomixer.test_play_twice$', '22s'),
(r'check.gst-editing-services.nle_simple.test_one_after_other$', '40s'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-editing-services.nle_nleoperation.test_pyramid_operations_expandable', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.nle_nleoperation.test_complex_operations', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.nle_nleoperation.test_pyramid_operations', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.nle_nleoperation.test_pyramid_operations2', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.nle_complex.test_one_above_another', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.nle_nleoperation.test_simple_operation', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.seek_with_stop.check_clock_sync', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.edit_while_seeked_with_stop', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.seek_with_stop', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.nle_tempochange.test_tempochange_seek', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.nle_tempochange.test_tempochange_play', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
]
VALGRIND_BLACKLIST = [
(r'check.gstreamer.gst_gstsystemclock.test_stress_cleanup_unschedule', '?'),
(r'check.gstreamer.gst_gstsystemclock.test_stress_reschedule', '?'),
(r'check.gstreamer.tools_gstinspect', '?'),
(r'check.gst-plugins-base.elements_videoscale', '?'),
(r'check.gst-plugins-base.pipelines_gl_launch_lines', '?'),
(r'check.gst-plugins-base.libs_gstgl', 'driver leaks / memory access'),
(r'check.gst-plugins-base.elements_gl', 'driver leaks / memory access'),
(r'check.gst-plugins-base.elements_libvisual', 'uninitialized memory access'),
(r'check.gst-plugins-base.generic_states', 'need to add gl elements to ignore list but only if using valgrind'),
(r'check.gst-plugins-good.elements_rtpjitterbuffer.test_push_backward_seq', 'flaky in valgrind'),
(r'check.gst-plugins-good.elements_rtpjitterbuffer.test_push_unordered', 'flaky in valgrind'),
(r'check.gst-plugins-bad.elements_assrender', '?'),
check: Add valgrind leak exception for glcolorconvert The following commit will add a test triggering a leak in GLX / the driver. Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5948>
2024-01-23 16:34:24 +00:00
(r'check.gst-plugins-bad.elements_autovideoconvert.test_autovideoconvert_videoconvert', 'Leak with GLX, https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/3216'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-plugins-bad.elements_camerabin', '?'),
(r'check.gst-plugins-bad.elements_line21', '?'),
(r'check.gst-plugins-bad.elements_mpeg2enc', '?'),
(r'check.gst-plugins-bad.elements_mplex', '?'),
(r'check.gst-plugins-bad.elements_mxfmux', '?'),
check: Add valgrind leak exception for srt test The srt unittest test_src_listener_sink_call will sometimes fail under valgrind with the following splat: Memcheck, a memory error detector Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info Parent PID: 14579 HEAP SUMMARY: in use at exit: 799,848 bytes in 2,182 blocks total heap usage: 64,090 allocs, 61,908 frees, 37,891,032 bytes allocated 120 bytes in 1 blocks are definitely lost in loss record 1,563 of 1,681 at 0x4842FF5: operator new(unsigned long) (vg_replace_malloc.c:422) by 0x6031E29: srt::sync::SetThreadLocalError(CUDTException const&) (sync_posix.cpp:461) by 0x5FCD77E: CUDT::epoll_wait(int, std::set<int, std::less<int>, std::allocator<int> >*, std::set<int, std::less<int>, std::allocator<int> >*, long, std::set<int, std::less<int>, std::allocator<int> >*, std::set<int, std::less<int>, std::allocator<int> >*) [clone .cold] (api.cpp:3796) by 0x5FE2F79: UDT::epoll_wait2(int, int*, int*, int*, int*, long, int*, int*, int*, int*) (api.cpp:4277) by 0x5F0C626: gst_srt_object_read (gstsrtobject.c:1569) by 0x5F0F978: gst_srt_src_fill (gstsrtsrc.c:180) by 0x5F5A2A1: gst_base_src_default_create (gstbasesrc.c:1620) by 0x5F5C9AE: gst_base_src_get_range (gstbasesrc.c:2630) by 0x5F5EF5A: gst_base_src_loop (gstbasesrc.c:2959) by 0x4918B13: gst_task_func (gsttask.c:399) by 0x4A60B33: g_thread_pool_thread_proxy.lto_priv.0 (gthreadpool.c:354) by 0x4A5DC41: g_thread_proxy (gthread.c:826) by 0x4F532A4: start_thread (pthread_create.c:481) by 0x4C71322: clone (clone.S:95) An issue has been started against libsrt here: https://github.com/Haivision/srt/issues/2867 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6098>
2024-02-09 11:16:55 +00:00
(r'check.gst-plugins-bad.elements_srt.test_src_listener_sink_caller', 'Need to investigate libsrt leaks (flaky)'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-plugins-bad.elements_x265enc', '?'),
(r'check.gst-plugins-bad.elements_zbar', '?'),
Revert "Revert "launcher: add webrtcbin datachannel tests to valgrind skip list"" This reverts commit ff79af843bc40a554795ee2a0682cfd12272dd24. Looks like most of these are still problematic and flaky in valgrind, so re-add to skiplist for now. https://gitlab.freedesktop.org/thaytan/gst-plugins-base/-/jobs/4275045
2020-08-26 14:47:23 +00:00
(r'check.gst-plugins-bad.elements_webrtcbin.test_data_channel_remote_notify', 'Need to fix leaks'),
(r'check.gst-plugins-bad.elements_webrtcbin.test_data_channel_pre_negotiated', 'Need to fix leaks'),
(r'check.gst-plugins-bad.elements_webrtcbin.test_data_channel_low_threshold', 'Need to fix leaks'),
(r'check.gst-plugins-bad.elements_webrtcbin.test_data_channel_transfer_data', 'Need to fix leaks'),
(r'check.gst-plugins-bad.elements_webrtcbin.test_data_channel_transfer_string', 'Need to fix leaks'),
(r'check.gst-plugins-bad.elements_webrtcbin.test_data_channel_create_after_negotiate', 'Need to fix leaks'),
(r'check.gst-plugins-bad.elements_webrtcbin.test_data_channel_max_message_size', 'Need to fix leaks'),
(r'check.gst-plugins-bad.elements_webrtcbin.test_renego_data_channel_add_stream', 'Need to fix leaks (possibly flaky)'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-libav.generic_plugin_test', '?'),
(r'check.gst-libav.generic_libavcodec_locking', '?'),
(r'check.gst-libav.elements_avdemux_ape', '?'),
(r'check.gst-editing-services.pythontests', 'Need to figure out how to introduce python suppressions'),
tests: Blacklist tests that are currently producing many CI failures Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1113>
2021-10-08 13:44:50 +00:00
(r'check.gst-editing-services.check_keyframes_in_compositor_two_sources', 'Valgrind exit with an exitcode 20 but shows no issue: https://gitlab.freedesktop.org/thiblahute/gst-editing-services/-/jobs/4079972'),
(r'check.gst-plugins-good.elements_splitmuxsrc.test_splitmuxsrc_sparse_streams', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/739'),
ci: switch the Fedora base image to f34 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2020-07-10 09:16:35 +00:00
(r'check.gst-plugins-good.elements_udpsrc.test_udpsrc_empty_packet', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/740'),
(r'check.gst-plugins-bad.elements_svthevc*', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/3011'),
]
ges:test: Document some GES failures Namely races in: * check.gst-editing-services.edit_while_seeked_with_stop_it * check.gst-editing-services.check_layer_activness_gaps_it See https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/803 and https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/804 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1130>
2021-10-13 03:17:27 +00:00
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
BLACKLIST = [
launcher: testsuites: skip systemclock stress tests These are very flaky when the build bots are under load.
2019-04-19 12:02:43 +00:00
(r'check.gstreamer.gst_gstsystemclock.test_stress_cleanup_unschedule', 'flaky under high server load'),
(r'check.gstreamer.gst_gstsystemclock.test_stress_reschedule', 'flaky under high server load'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gstreamer.pipelines_seek.test_loopback_2$', '?'),
(r'check.gstreamer.gst_gstelement.test_foreach_pad$', '?'),
(r'check.gstreamer.libs_baseparse.parser_pull_short_read$', '?'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-plugins-base.elements_multisocketsink.test_sending_buffers_with_9_gstmemories$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/779'),
(r'check.gst-plugins-base.elements_multisocketsink.test_client_next_keyframe$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/779'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-plugins-base.elements_multisocketsink.test_add_client$', ''),
ci: switch the Fedora base image to f34 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2020-07-10 09:16:35 +00:00
(r'check.gst-plugins-base.elements_multisocketsink.test_burst_client_bytes$', ''),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-plugins-base.libs_gstglcolorconvert.test_reorder_buffer$', '?'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-plugins-base.elements_audiotestsrc.test_layout$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/781'),
(r'check.gst-plugins-good.elements_souphttpsrc.test_icy_stream$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/782'),
(r'check.gst-plugins-good.elements_rtpbin.test_sender_eos$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/783'),
(r'check.gst-plugins-good.elements_rtpbin.test_cleanup_recv$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/784'),
(r'check.gst-plugins-good.elements_flvmux.test_incrementing_timestamps$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/785'),
(r'check.gst-plugins-good.elements_flvmux.test_video_caps_late$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/786'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-plugins-base.elements_appsrc.test_appsrc_blocked_on_caps$', '?'),
ci: Check for closed bugs for unit tests too And add FIXMEs for test that should have been reenabled but haven't, we will reenable them when CI is stable enough! Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:19:46 +00:00
(r'check.gst-plugins-good.elements_splitmux.test_splitmuxsink$', '[FIXME -- SHOULD BE FIXED] https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/626'),
(r'check.gst-plugins-good.elements_splitmux.test_splitmuxsrc_sparse_streams$', '[FIXME -- SHOULD BE FIXED] https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/739'),
(r'check.gst-plugins-good.elements_splitmux.test_splitmuxsrc_caps_change$', '[FIXME -- SHOULD BE FIXED] https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/547'),
(r'check.gst-plugins-bad.elements_dtls.test_data_transfer$', '[FIXME -- SHOULD BE FIXED] https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/811'),
(r'check.gst-plugins-bad.elements_dtls.test_create_and_unref$', '[FIXME -- SHOULD BE FIXED] https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/811'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-plugins-bad.elements_camerabin.test_image_video_cycle$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/787'),
(r'check.gst-plugins-bad.elements_camerabin.test_single_video_recording$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/787#note_1101544'),
(r'check.gst-plugins-bad.elements_camerabin.test_multiple_video_recordings$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/787#note_1101546'),
(r'check.gst-plugins-bad.elements_curlhttpsrc.test_multiple_http_requests$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/788'),
(r'check.gst-plugins-good.elements_rtpsession.test_multiple_senders_roundrobin_rbs$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/789'),
(r'check.gst-plugins-bad.elements_shm.test_shm_live$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/790'),
ci: Fail tests if we forget to checkout expectation files And add missing expectation files Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7400>
2024-08-21 13:24:58 +00:00
(r'check.gst-plugins-bad.validate.autovideoconvert.renegotiate$', ''),
ci: Check for closed bugs for unit tests too And add FIXMEs for test that should have been reenabled but haven't, we will reenable them when CI is stable enough! Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:19:46 +00:00
(r'check.gst-plugins-good.elements_splitmux.test_splitmuxsink_async$', '[FIXME -- SHOULD BE FIXED] https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/791'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-plugins-bad.elements_netsim.netsim_stress$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/792'),
(r'check.gst-editing-services.nle_complex.test_one_expandable_another$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/793'),
(r'check.gst-editing-services.nle_simple.test_simplest$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/794'),
(r'check.gst-editing-services.ges_basic.test_ges_pipeline_change_state$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/795'),
ci: Check for closed bugs for unit tests too And add FIXMEs for test that should have been reenabled but haven't, we will reenable them when CI is stable enough! Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:19:46 +00:00
(r'check.gst-editing-services.pythontests.pyunittest.python.test_timeline.TestTransitions.test_transition_type$', '[FIXME -- SHOULD BE FIXED] https://gitlab.freedesktop.org/gstreamer/gst-editing-services/issues/62'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-editing-services.pythontests.pyunittest.python.test_timeline.TestTransitions.test_auto_transition$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/796'),
(r'check.gst-plugins-base.pipelines_tcp.test_that_tcpserversink_and_tcpclientsrc_are_symmetrical$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/797'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gstreamer.elements_capsfilter.test_unfixed_downstream_caps$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/issues/335'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-rtsp-server.gst_rtspclientsink.test_record$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/798'),
(r'check.gst-rtsp-server.gst_rtspserver.test_shared_udp$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/799'),
(r'check.gst-plugins-base.elements_audiomixer.test_flush_start_flush_stop$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/800'),
(r'check.gstreamer-sharp.SdpTests$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/801'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-devtools.validate.launcher_tests.test_validate.launch_pipeline.not_negotiated.caps_query_failure.play_15s$', '?'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-editing-services.nle_simple.test_one_bin_after_other$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/802'),
validate: blacklist gstreamer-vaapi checks They still can be checked by running the tests explicitly.
2019-10-25 10:26:58 +00:00
(r'check.gstreamer-vaapi.*$', 'only run the tests explicitly'),
tests: Blacklist tests that are currently producing many CI failures Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1113>
2021-10-08 13:44:50 +00:00
(r'check.gst-rtsp-server.gst_rtspserver.test_multiple_transports', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/767'),
check: Move CI specific ignored tests into their own list Instead of ignoring the tests always, check if a CI env var is present and only ignore the tests then. Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2023-09-28 17:16:37 +00:00
]
CI_BLACKLIST = [
(r'check.gst-plugins-bad.elements_vk*', 'Mesa in the CI image is older, will start passing once we update to llvm16 and mesa 23.1'),
(r'check.gst-plugins-bad.libs_vk*', 'Mesa in the CI image is older, will start passing once we update to llvm16 and mesa 23.1'),
ci: switch the Fedora base image to f34 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2020-07-10 09:16:35 +00:00
(r'check.gst-plugins-good.elements_souphttpsrc2.test_icy_stream', 'flaky in valgrind, leaks in CI but not locally'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
]
ci: switch the Fedora base image to f34 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2020-07-10 09:16:35 +00:00
tests:webrtcbin: Make known flaky webrtcbin as such See https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/773 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1113>
2021-10-11 14:05:36 +00:00
KNOWN_ISSUES = {
validate: filter check.gst-editing-services.complex_effect_bin_desc Add this check to the known issues list, fails way too often. https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/2973 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6003>
2024-01-29 00:08:55 +00:00
"https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/2973": {
"tests": [
"check.gst-editing-services.complex_effect_bin_desc"
],
"issues": [
{
'returncode': 238,
'sometimes': True,
},
{
"issue-id": "validateflow::mismatch",
"summary": "The recorded log does not match the expectation file.",
"level": "critical",
# "details": "Mismatch error in pad videosink:sink, line 4. Expected:\nbuffer: checksum=369888c2612267760fcfaa74e52fc53bd73e4d15, pts=0:00:00.000000000, dur=0:00:00.033333333, meta=GstVideoMeta\nActual:\nbuffer: checksum=b7764dce84f311119c4c36b511ba5adb66de76af, pts=0:00:00.000000000, dur=0:00:00.033333333, meta=GstVideoMeta\n",
},
],
},
tests:webrtcbin: Make known flaky webrtcbin as such See https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/773 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1113>
2021-10-11 14:05:36 +00:00
"https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/773": {
"tests": [
r"check.gst-plugins-bad.elements_webrtcbin.*",
],
"max_retries": 1,
},
ges:tests: Mark ges 'check_edit_in_frames_with_framerate_mistmatch as flaky See https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/775 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1113>
2021-10-11 14:45:50 +00:00
"https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/775": {
"tests": [
r"check.gst-editing-services.check_edit_in_frames_with_framerate_mismatch",
],
"max_retries": 1,
ges:tests: Mark check.gst-plugins-base.validate.giosrc.read-growing-file as flaky See https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/776 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1113>
2021-10-11 15:03:02 +00:00
},
"https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/776": {
"tests": [
r"check.gst-plugins-base.validate.giosrc.read-growing-file",
],
"max_retries": 1,
},
ges:test: Document some GES failures Namely races in: * check.gst-editing-services.edit_while_seeked_with_stop_it * check.gst-editing-services.check_layer_activness_gaps_it See https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/803 and https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/804 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1130>
2021-10-13 03:17:27 +00:00
"https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/803": {
"tests": [
"check.gst-editing-services.edit_while_seeked_with_stop"
],
"issues": [
{
'returncode': None,
'sometimes': True,
},
{
'timeout': True,
'sometimes': True,
},
],
},
"https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/804": {
"tests": [
"check.gst-editing-services.check_layer_activness_gaps"
],
"issues": [
{
'returncode': 18,
'sometimes': True,
},
{
"issue-id": "scenario::execution-error",
"summary": "The execution of an action did not properly happen",
"level": "critical",
"detected-on": "check_layer_activness_gaps.scenario",
# "details": "\n> check_layer_activness_gaps.scenario:22\n 22 | check-property, target-element-factory-name=videotestsrc, property-name=pattern, property-value="Blue"\n >\n > <src>::pattern expected value: '(gchararray)Blue' different than observed: '(gchararray)"100\%\ Black"'",
},
],
},
tests: Move srtp test known issue def to the 'check' testuite Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3979>
2023-02-16 01:06:39 +00:00
"https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1777": {
"tests": [
"check.gst-plugins-bad.elements_srtp.test_play.*"
],
"issues": [
{
'returncode': 1,
'sometimes': True,
},
],
},
tests:webrtcbin: Make known flaky webrtcbin as such See https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/773 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1113>
2021-10-11 14:05:36 +00:00
}
Revert "Revert "validate:launcher: Add an app handler for unit tests described in meson"" This reverts commit 5656e2a1b208155be36cf16b08a9ffcc95694328.
2017-01-03 16:01:31 +00:00
def setup_tests(test_manager, options):
validate:launcher: Add a way to specify a set of tests to run under the leak tracer https://bugzilla.gnome.org/show_bug.cgi?id=767856
2017-02-08 20:46:23 +00:00
if options.gst_check_leak_trace_testnames == 'known-not-leaky':
options.gst_check_leak_trace_testnames = KNOWN_NOT_LEAKY
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
test_manager.set_default_blacklist(BLACKLIST)
if options.valgrind:
test_manager.set_default_blacklist(VALGRIND_BLACKLIST)
if options.long_limit <= utils.LONG_TEST:
test_manager.set_default_blacklist(LONG_VALGRIND_TESTS)
check: Move CI specific ignored tests into their own list Instead of ignoring the tests always, check if a CI env var is present and only ignore the tests then. Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2023-09-28 17:16:37 +00:00
if 'CI_COMMIT_SHA' in os.environ:
test_manager.set_default_blacklist(CI_BLACKLIST)
tests:webrtcbin: Make known flaky webrtcbin as such See https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/773 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1113>
2021-10-11 14:05:36 +00:00
test_manager.add_expected_issues(KNOWN_ISSUES)
validate:launcher:check: Make sure to register tests from the testsuite Instead of having them listed from the app manager. This is needed to avoid backtrace as tests now have to be register when setting up the testsuite.
2017-07-19 15:49:09 +00:00
test_manager.register_tests()
Revert "Revert "validate:launcher: Add an app handler for unit tests described in meson"" This reverts commit 5656e2a1b208155be36cf16b08a9ffcc95694328.
2017-01-03 16:01:31 +00:00
return True
Reference in a new issue Copy permalink