mirrors/gstreamer
mirrors/gstreamer
1
1
Fork 0
mirror of https://gitlab.freedesktop.org/gstreamer/gstreamer.git synced 2024-12-23 00:36:51 +00:00
Code Issues 1 Projects Releases Wiki Activity
gstreamer/subprojects/gst-devtools/validate/launcher/testsuites/check.py

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

263 lines
18 KiB
Python
Raw Normal View History

Revert "Revert "validate:launcher: Add an app handler for unit tests described in meson"" This reverts commit 5656e2a1b208155be36cf16b08a9ffcc95694328.
2017-01-03 16:01:31 +00:00
# -*- Mode: Python -*- vi:si:et:sw=4:sts=4:ts=4:syntax=python
#
# Copyright (c) 2016,Thibault Saunier <thibault.saunier@osg.samsung.com>
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with this program; if not, write to the
# Free Software Foundation, Inc., 51 Franklin St, Fifth Floor,
# Boston, MA 02110-1301, USA.
"""
GStreamer unit tests
"""
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
from launcher import utils
Revert "Revert "validate:launcher: Add an app handler for unit tests described in meson"" This reverts commit 5656e2a1b208155be36cf16b08a9ffcc95694328.
2017-01-03 16:01:31 +00:00
check: Move CI specific ignored tests into their own list Instead of ignoring the tests always, check if a CI env var is present and only ignore the tests then. Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2023-09-28 17:16:37 +00:00
import os
Revert "Revert "validate:launcher: Add an app handler for unit tests described in meson"" This reverts commit 5656e2a1b208155be36cf16b08a9ffcc95694328.
2017-01-03 16:01:31 +00:00
TEST_MANAGER = "check"
validate:launcher: Add a way to specify a set of tests to run under the leak tracer https://bugzilla.gnome.org/show_bug.cgi?id=767856
2017-02-08 20:46:23 +00:00
KNOWN_NOT_LEAKY = r'^check.gst-devtools.*|^check.gstreamer.*|^check-gst-plugins-base|^check.gst-plugins-ugly|^check.gst-plugins-good'
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
# These tests take very long compared to what they add, so let's skip them.
LONG_VALGRIND_TESTS = [
(r'check.[a-z-]*.generic_states.test_state_changes_down_seq', 'enough to run one of the sequences'),
(r'check.[a-z-]*.generic_states.test_state_changes_up_seq', 'enough to run one of the sequences',),
ci: switch the Fedora base image to f34 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2020-07-10 09:16:35 +00:00
(r'check.[a-z-]*.generic_states.test_state_changes_up_and_down_seq', 'enough to run the sequences'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gstreamer.gst_gstelement.test_foreach_pad$', '48s'),
(r'check.gstreamer.gst_gstinfo.info_post_gst_init_category_registration$', '21s'),
(r'check.gstreamer.gst_gstsystemclock.test_resolution$', '60s'),
(r'check.gstreamer.libs_aggregator.test_infinite_seek$', '20s'),
(r'check.gstreamer.libs_aggregator.test_infinite_seek_50_src$', '20s'),
(r'check.gstreamer.libs_gstharness.test_harness_element_ref$', '20s'),
(r'check.gstreamer.pipelines_simple_launch_lines.test_2_elements$', '58s'),
(r'check.gstreamer.pipelines_stress.test_stress$', '54s'),
(r'check.gstreamer.pipelines_stress.test_stress_preroll$', '27s'),
ci: switch the Fedora base image to f34 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2020-07-10 09:16:35 +00:00
(r'check.gst-plugins-base.elements_appsrc.test_appsrc_limits', '53.37s'),
(r'check.gst-plugins-base.elements_appsrc.test_appsrc_send_event_before_buffer', '49.25s'),
(r'check.gst-plugins-base.elements_appsrc.test_appsrc_send_event_before_sample', '51.39s'),
(r'check.gst-plugins-base.elements_appsrc.test_appsrc_send_event_between_caps_buffer', '56.13s'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-plugins-base.elements_appsrc.test_appsrc_block_deadlock$', '265.595s'),
(r'check.gst-plugins-base.elements_audioresample.test_fft$', '91.247s'),
(r'check.gst-plugins-base.elements_audioresample.test_timestamp_drift$', '141.784s'),
validate:check: Fix some mistakes translating regex from gitlab-ci.yml
2019-04-17 20:46:31 +00:00
(r'check.gst-plugins-base.elements-videoscale.*', 'superlong'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-plugins-base.libs_video.test_overlay_blend$', '74.096s'),
(r'check.gst-plugins-base.libs_video.test_video_color_convert$', '345.271s'),
(r'check.gst-plugins-base.libs_video.test_video_formats$', '70.987s'),
(r'check.gst-plugins-base.libs_video.test_video_size_convert$', '56.387s'),
(r'check.gst-plugins-base.elements_audiointerleave.test_audiointerleave_2ch_pipeline_$', '5 *51.069s'),
(r'check.gst-plugins-base.elements_multifdsink.test_client_kick$', '46.909s'),
(r'check.gst-plugins-base.elements_videotestsrc.test_all_patterns$', '?'),
(r'check.gst-plugins-base.elements_videotestsrc.test_patterns_are_deterministic$', '?'),
(r'check.gst-plugins-good.elements_shapewipe.test_general$', '325s'),
(r'check.gst-plugins-good.elements_videocrop.test_cropping$', '245s'),
validate:check: Fix some mistakes translating regex from gitlab-ci.yml
2019-04-17 20:46:31 +00:00
(r'check.gst-plugins-good.elements_videomixer.*', '30s (alsodeprecated)'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-plugins-good.elements_rtp_payloading.rtp_jpeg_packet_loss$', '109s'),
(r'check.gst-plugins-good.elements_videomixer.test_play_twice_then_add_and_play_again$', '55s'),
(r'check.gst-plugins-good.pipelines_effectv.test_quarktv$', '53s'),
(r'check.gst-plugins-good.elements_deinterlace.test_mode_disabled_passthrough$', '52s'),
(r'check.gst-plugins-good.elements_deinterlace.test_mode_auto_deinterlaced_passthrough$', '28s'),
(r'check.gst-plugins-good.elements_deinterleave.test_2_channels_caps_change$', '30s'),
(r'check.gst-plugins-good.elements_deinterleave.test_2_channels$', '22s'),
(r'check.gst-plugins-good.elements_rtpjitterbuffer.test_fill_queue$', '22s'),
(r'check.gst-plugins-good.elements_splitmux.test_splitmuxsink_async$', '20s'),
(r'check.gst-plugins-good.elements_videomixer.test_play_twice$', '22s'),
(r'check.gst-editing-services.nle_simple.test_one_after_other$', '40s'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-editing-services.nle_nleoperation.test_pyramid_operations_expandable', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.nle_nleoperation.test_complex_operations', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.nle_nleoperation.test_pyramid_operations', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.nle_nleoperation.test_pyramid_operations2', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.nle_complex.test_one_above_another', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.nle_nleoperation.test_simple_operation', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.seek_with_stop.check_clock_sync', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.edit_while_seeked_with_stop', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.seek_with_stop', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.nle_tempochange.test_tempochange_seek', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
(r'check.gst-editing-services.nle_tempochange.test_tempochange_play', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/780'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
]
VALGRIND_BLACKLIST = [
(r'check.gstreamer.gst_gstsystemclock.test_stress_cleanup_unschedule', '?'),
(r'check.gstreamer.gst_gstsystemclock.test_stress_reschedule', '?'),
(r'check.gstreamer.tools_gstinspect', '?'),
(r'check.gst-plugins-base.elements_videoscale', '?'),
(r'check.gst-plugins-base.pipelines_gl_launch_lines', '?'),
(r'check.gst-plugins-base.libs_gstgl', 'driver leaks / memory access'),
(r'check.gst-plugins-base.elements_gl', 'driver leaks / memory access'),
(r'check.gst-plugins-base.elements_libvisual', 'uninitialized memory access'),
(r'check.gst-plugins-base.generic_states', 'need to add gl elements to ignore list but only if using valgrind'),
(r'check.gst-plugins-good.elements_rtpjitterbuffer.test_push_backward_seq', 'flaky in valgrind'),
(r'check.gst-plugins-good.elements_rtpjitterbuffer.test_push_unordered', 'flaky in valgrind'),
(r'check.gst-plugins-bad.elements_assrender', '?'),
(r'check.gst-plugins-bad.elements_camerabin', '?'),
(r'check.gst-plugins-bad.elements_line21', '?'),
(r'check.gst-plugins-bad.elements_mpeg2enc', '?'),
(r'check.gst-plugins-bad.elements_mplex', '?'),
(r'check.gst-plugins-bad.elements_mxfmux', '?'),
(r'check.gst-plugins-bad.elements_x265enc', '?'),
(r'check.gst-plugins-bad.elements_zbar', '?'),
Revert "Revert "launcher: add webrtcbin datachannel tests to valgrind skip list"" This reverts commit ff79af843bc40a554795ee2a0682cfd12272dd24. Looks like most of these are still problematic and flaky in valgrind, so re-add to skiplist for now. https://gitlab.freedesktop.org/thaytan/gst-plugins-base/-/jobs/4275045
2020-08-26 14:47:23 +00:00
(r'check.gst-plugins-bad.elements_webrtcbin.test_data_channel_remote_notify', 'Need to fix leaks'),
(r'check.gst-plugins-bad.elements_webrtcbin.test_data_channel_pre_negotiated', 'Need to fix leaks'),
(r'check.gst-plugins-bad.elements_webrtcbin.test_data_channel_low_threshold', 'Need to fix leaks'),
(r'check.gst-plugins-bad.elements_webrtcbin.test_data_channel_transfer_data', 'Need to fix leaks'),
(r'check.gst-plugins-bad.elements_webrtcbin.test_data_channel_transfer_string', 'Need to fix leaks'),
(r'check.gst-plugins-bad.elements_webrtcbin.test_data_channel_create_after_negotiate', 'Need to fix leaks'),
(r'check.gst-plugins-bad.elements_webrtcbin.test_data_channel_max_message_size', 'Need to fix leaks'),
(r'check.gst-plugins-bad.elements_webrtcbin.test_renego_data_channel_add_stream', 'Need to fix leaks (possibly flaky)'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-libav.generic_plugin_test', '?'),
(r'check.gst-libav.generic_libavcodec_locking', '?'),
(r'check.gst-libav.elements_avdemux_ape', '?'),
(r'check.gst-editing-services.pythontests', 'Need to figure out how to introduce python suppressions'),
tests: Blacklist tests that are currently producing many CI failures Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1113>
2021-10-08 13:44:50 +00:00
(r'check.gst-editing-services.check_keyframes_in_compositor_two_sources', 'Valgrind exit with an exitcode 20 but shows no issue: https://gitlab.freedesktop.org/thiblahute/gst-editing-services/-/jobs/4079972'),
(r'check.gst-plugins-good.elements_splitmuxsrc.test_splitmuxsrc_sparse_streams', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/739'),
ci: switch the Fedora base image to f34 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2020-07-10 09:16:35 +00:00
(r'check.gst-plugins-good.elements_udpsrc.test_udpsrc_empty_packet', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/740'),
(r'check.gst-plugins-bad.elements_svthevc*', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/3011'),
]
ges:test: Document some GES failures Namely races in: * check.gst-editing-services.edit_while_seeked_with_stop_it * check.gst-editing-services.check_layer_activness_gaps_it See https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/803 and https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/804 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1130>
2021-10-13 03:17:27 +00:00
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
BLACKLIST = [
launcher: testsuites: skip systemclock stress tests These are very flaky when the build bots are under load.
2019-04-19 12:02:43 +00:00
(r'check.gstreamer.gst_gstsystemclock.test_stress_cleanup_unschedule', 'flaky under high server load'),
(r'check.gstreamer.gst_gstsystemclock.test_stress_reschedule', 'flaky under high server load'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gstreamer.pipelines_seek.test_loopback_2$', '?'),
(r'check.gstreamer.gst_gstelement.test_foreach_pad$', '?'),
(r'check.gstreamer.libs_baseparse.parser_pull_short_read$', '?'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-plugins-base.elements_multisocketsink.test_sending_buffers_with_9_gstmemories$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/779'),
(r'check.gst-plugins-base.elements_multisocketsink.test_client_next_keyframe$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/779'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-plugins-base.elements_multisocketsink.test_add_client$', ''),
ci: switch the Fedora base image to f34 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2020-07-10 09:16:35 +00:00
(r'check.gst-plugins-base.elements_multisocketsink.test_burst_client_bytes$', ''),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-plugins-base.libs_gstglcolorconvert.test_reorder_buffer$', '?'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-plugins-base.elements_audiotestsrc.test_layout$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/781'),
(r'check.gst-plugins-good.elements_souphttpsrc.test_icy_stream$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/782'),
(r'check.gst-plugins-good.elements_rtpbin.test_sender_eos$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/783'),
(r'check.gst-plugins-good.elements_rtpbin.test_cleanup_recv$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/784'),
(r'check.gst-plugins-good.elements_flvmux.test_incrementing_timestamps$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/785'),
(r'check.gst-plugins-good.elements_flvmux.test_video_caps_late$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/786'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-plugins-base.elements_appsrc.test_appsrc_blocked_on_caps$', '?'),
ci: Check for closed bugs for unit tests too And add FIXMEs for test that should have been reenabled but haven't, we will reenable them when CI is stable enough! Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:19:46 +00:00
(r'check.gst-plugins-good.elements_splitmux.test_splitmuxsink$', '[FIXME -- SHOULD BE FIXED] https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/626'),
(r'check.gst-plugins-good.elements_splitmux.test_splitmuxsrc_sparse_streams$', '[FIXME -- SHOULD BE FIXED] https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/739'),
(r'check.gst-plugins-good.elements_splitmux.test_splitmuxsrc_caps_change$', '[FIXME -- SHOULD BE FIXED] https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/547'),
(r'check.gst-plugins-bad.elements_dtls.test_data_transfer$', '[FIXME -- SHOULD BE FIXED] https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/811'),
(r'check.gst-plugins-bad.elements_dtls.test_create_and_unref$', '[FIXME -- SHOULD BE FIXED] https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/811'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-plugins-bad.elements_camerabin.test_image_video_cycle$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/787'),
(r'check.gst-plugins-bad.elements_camerabin.test_single_video_recording$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/787#note_1101544'),
(r'check.gst-plugins-bad.elements_camerabin.test_multiple_video_recordings$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/787#note_1101546'),
(r'check.gst-plugins-bad.elements_curlhttpsrc.test_multiple_http_requests$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/788'),
(r'check.gst-plugins-good.elements_rtpsession.test_multiple_senders_roundrobin_rbs$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/789'),
(r'check.gst-plugins-bad.elements_shm.test_shm_live$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/790'),
ci: Check for closed bugs for unit tests too And add FIXMEs for test that should have been reenabled but haven't, we will reenable them when CI is stable enough! Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:19:46 +00:00
(r'check.gst-plugins-good.elements_splitmux.test_splitmuxsink_async$', '[FIXME -- SHOULD BE FIXED] https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/791'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-plugins-bad.elements_netsim.netsim_stress$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/792'),
(r'check.gst-editing-services.nle_complex.test_one_expandable_another$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/793'),
(r'check.gst-editing-services.nle_simple.test_simplest$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/794'),
(r'check.gst-editing-services.ges_basic.test_ges_pipeline_change_state$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/795'),
ci: Check for closed bugs for unit tests too And add FIXMEs for test that should have been reenabled but haven't, we will reenable them when CI is stable enough! Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:19:46 +00:00
(r'check.gst-editing-services.pythontests.pyunittest.python.test_timeline.TestTransitions.test_transition_type$', '[FIXME -- SHOULD BE FIXED] https://gitlab.freedesktop.org/gstreamer/gst-editing-services/issues/62'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-editing-services.pythontests.pyunittest.python.test_timeline.TestTransitions.test_auto_transition$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/796'),
(r'check.gst-plugins-base.pipelines_tcp.test_that_tcpserversink_and_tcpclientsrc_are_symmetrical$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/797'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gstreamer.elements_capsfilter.test_unfixed_downstream_caps$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/issues/335'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-rtsp-server.gst_rtspclientsink.test_record$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/798'),
(r'check.gst-rtsp-server.gst_rtspserver.test_shared_udp$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/799'),
(r'check.gst-plugins-base.elements_audiomixer.test_flush_start_flush_stop$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/800'),
(r'check.gstreamer-sharp.SdpTests$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/801'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
(r'check.gst-devtools.validate.launcher_tests.test_validate.launch_pipeline.not_negotiated.caps_query_failure.play_15s$', '?'),
tests: Move all unit tests issues to the GStreamer monorepo Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1121>
2021-10-12 20:14:49 +00:00
(r'check.gst-editing-services.nle_simple.test_one_bin_after_other$', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/802'),
validate: blacklist gstreamer-vaapi checks They still can be checked by running the tests explicitly.
2019-10-25 10:26:58 +00:00
(r'check.gstreamer-vaapi.*$', 'only run the tests explicitly'),
tests: Blacklist tests that are currently producing many CI failures Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1113>
2021-10-08 13:44:50 +00:00
(r'check.gst-rtsp-server.gst_rtspserver.test_multiple_transports', 'https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/767'),
check: Move CI specific ignored tests into their own list Instead of ignoring the tests always, check if a CI env var is present and only ignore the tests then. Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2023-09-28 17:16:37 +00:00
]
CI_BLACKLIST = [
(r'check.gst-plugins-bad.elements_vk*', 'Mesa in the CI image is older, will start passing once we update to llvm16 and mesa 23.1'),
(r'check.gst-plugins-bad.libs_vk*', 'Mesa in the CI image is older, will start passing once we update to llvm16 and mesa 23.1'),
ci: switch the Fedora base image to f34 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2020-07-10 09:16:35 +00:00
(r'check.gst-plugins-good.elements_souphttpsrc2.test_icy_stream', 'flaky in valgrind, leaks in CI but not locally'),
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
]
ci: switch the Fedora base image to f34 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2020-07-10 09:16:35 +00:00
tests:webrtcbin: Make known flaky webrtcbin as such See https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/773 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1113>
2021-10-11 14:05:36 +00:00
KNOWN_ISSUES = {
"https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/773": {
"tests": [
r"check.gst-plugins-bad.elements_webrtcbin.*",
],
"max_retries": 1,
},
ges:tests: Mark ges 'check_edit_in_frames_with_framerate_mistmatch as flaky See https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/775 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1113>
2021-10-11 14:45:50 +00:00
"https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/775": {
"tests": [
r"check.gst-editing-services.check_edit_in_frames_with_framerate_mismatch",
],
"max_retries": 1,
ges:tests: Mark check.gst-plugins-base.validate.giosrc.read-growing-file as flaky See https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/776 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1113>
2021-10-11 15:03:02 +00:00
},
"https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/776": {
"tests": [
r"check.gst-plugins-base.validate.giosrc.read-growing-file",
],
"max_retries": 1,
},
ges:test: Document some GES failures Namely races in: * check.gst-editing-services.edit_while_seeked_with_stop_it * check.gst-editing-services.check_layer_activness_gaps_it See https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/803 and https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/804 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1130>
2021-10-13 03:17:27 +00:00
"https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/803": {
"tests": [
"check.gst-editing-services.edit_while_seeked_with_stop"
],
"issues": [
{
'returncode': None,
'sometimes': True,
},
{
'timeout': True,
'sometimes': True,
},
],
},
"https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/804": {
"tests": [
"check.gst-editing-services.check_layer_activness_gaps"
],
"issues": [
{
'returncode': 18,
'sometimes': True,
},
{
"issue-id": "scenario::execution-error",
"summary": "The execution of an action did not properly happen",
"level": "critical",
"detected-on": "check_layer_activness_gaps.scenario",
# "details": "\n> check_layer_activness_gaps.scenario:22\n 22 | check-property, target-element-factory-name=videotestsrc, property-name=pattern, property-value="Blue"\n >\n > <src>::pattern expected value: '(gchararray)Blue' different than observed: '(gchararray)"100\%\ Black"'",
},
],
},
tests: Move srtp test known issue def to the 'check' testuite Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3979>
2023-02-16 01:06:39 +00:00
"https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1777": {
"tests": [
"check.gst-plugins-bad.elements_srtp.test_play.*"
],
"issues": [
{
'returncode': 1,
'sometimes': True,
},
],
},
tests:webrtcbin: Make known flaky webrtcbin as such See https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/773 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1113>
2021-10-11 14:05:36 +00:00
}
Revert "Revert "validate:launcher: Add an app handler for unit tests described in meson"" This reverts commit 5656e2a1b208155be36cf16b08a9ffcc95694328.
2017-01-03 16:01:31 +00:00
def setup_tests(test_manager, options):
validate:launcher: Add a way to specify a set of tests to run under the leak tracer https://bugzilla.gnome.org/show_bug.cgi?id=767856
2017-02-08 20:46:23 +00:00
if options.gst_check_leak_trace_testnames == 'known-not-leaky':
options.gst_check_leak_trace_testnames = KNOWN_NOT_LEAKY
validate:testsuites: Add unit tests blacklists
2019-04-17 14:30:02 +00:00
test_manager.set_default_blacklist(BLACKLIST)
if options.valgrind:
test_manager.set_default_blacklist(VALGRIND_BLACKLIST)
if options.long_limit <= utils.LONG_TEST:
test_manager.set_default_blacklist(LONG_VALGRIND_TESTS)
check: Move CI specific ignored tests into their own list Instead of ignoring the tests always, check if a CI env var is present and only ignore the tests then. Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1060>
2023-09-28 17:16:37 +00:00
if 'CI_COMMIT_SHA' in os.environ:
test_manager.set_default_blacklist(CI_BLACKLIST)
tests:webrtcbin: Make known flaky webrtcbin as such See https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/773 Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1113>
2021-10-11 14:05:36 +00:00
test_manager.add_expected_issues(KNOWN_ISSUES)
validate:launcher:check: Make sure to register tests from the testsuite Instead of having them listed from the app manager. This is needed to avoid backtrace as tests now have to be register when setting up the testsuite.
2017-07-19 15:49:09 +00:00
test_manager.register_tests()
Revert "Revert "validate:launcher: Add an app handler for unit tests described in meson"" This reverts commit 5656e2a1b208155be36cf16b08a9ffcc95694328.
2017-01-03 16:01:31 +00:00
return True
Reference in a new issue Copy permalink