Commit graph

46 commits

Author SHA1 Message Date
Thibault Saunier
766f339fb8 launcher: Add support for running tests inside rr
Allowing us to easily run the tests forever and then replay the
failures!

https://rr-project.org/

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>
2020-05-26 17:19:36 -04:00
Thibault Saunier
0b41b8a8c0 validate:launcher: Add a generator to generate test for frame accurate seeking 2020-03-12 12:35:45 -03:00
Thibault Saunier
2e45db5ba4 validate:launcher: Enhance support for running ssim tests
Using a special 'ssim' variable in pipeline dicts to activate it
Similare to what we do for validateflow
2020-03-11 16:36:20 -03:00
Thibault Saunier
080fdf8e2f validate:launcher: Add support for the imagesequence protocol 2020-03-06 16:05:38 -03:00
Thibault Saunier
3343f166da validate:launcher: Move get_fakesink_for_media_type to utils
So it can be reused in other apps like GES
2019-06-26 12:09:03 -04:00
Thibault Saunier
0a56447bcc validate:launcher: Add python suppression files 2019-04-11 13:17:14 +00:00
Thibault Saunier
f7517e503c validate:launcher: Do not user python 3.5 features 2019-04-11 13:17:14 +00:00
Thibault Saunier
787939f750 validate:launcher: Add a list of well known subpression files from gst-build subprojects
Fixes https://gitlab.freedesktop.org/gstreamer/gst-devtools/issues/38
2019-04-03 13:38:42 +00:00
Thibault Saunier
01aa026e5a validate:launcher: Try to send SIGINT before killing processes
Giving a chance for validate to print reports
2019-04-03 13:38:42 +00:00
Thibault Saunier
f2e180e51f validate:launcher: Add a way to force coloration 2019-04-03 13:38:42 +00:00
Thibault Saunier
ddb81f29e8 validate:launcher: factor out TTY check and enhance iteration output 2019-04-03 13:38:42 +00:00
Thibault Saunier
c8a3b34a3a validate:launcher: Add a way to force stdout coloring 2019-04-03 13:38:42 +00:00
Thibault Saunier
3a826e1e3d validate:launcher: Implement bug checks for gitlab
And use new gitlab urls for all the bugs
2019-04-03 13:38:42 +00:00
Thibault Saunier
3d5099ee41 validate:launcher: Use gst-integration-testsuites subproject as default testsuite repo if avalaible 2019-04-03 13:38:42 +00:00
Thibault Saunier
773f2e456c validate:launcher: Move '_format_config_template' to the utilities
So it can be reused by GES tests
2019-03-11 18:14:23 -03:00
Thibault Saunier
1c51d3499e validate:launcher: Add a GST_VALIDATE_LAUNCHER_MAIN_DIR env variable
This is generally usefull so we do not have to pass -M every time we launch the launcher

And it adds support for nesting launcher calls always respecting the provided main directory

+ Fix some new pep8 errors
2018-11-30 11:00:10 -03:00
Thibault Saunier
69ec48feef validate:launcher: Allow retrieving coredumps from within flatpak 2018-07-01 12:55:06 -04:00
Thibault Saunier
58c90448ca validate: launcher: Add support for running tests with a pushfile source
Introducing the `.media_info.push` media info extension, which is meant
to let the launcher know that those file should run with the "pushfile://"
protocol.

And allow symlinking "normal" `.media_info` to their `.pushfile` variant
so that both can share the exact same content.
2018-06-15 12:05:14 -04:00
Thibault Saunier
c1f89b4acb validate: launcher: Add a way to simply run SSIM checks on rendered files
We will run a simple pipeline with the IQA element to run ssim (dssim)
tests on the rendered files, comparing it with a reference file.

For now we use the very empiric 1.0 value as a ssim error threshold and
the goal is basically to detect completely broken renderings.
2018-05-13 09:44:17 -04:00
Thibault Saunier
25e3b90225 validate:launcher: printc is accepting object as arguments
Make sure in all code paths those are converted to strings
2018-04-15 20:07:19 -03:00
Thibault Saunier
a70658a464 launcher: Print test number in the order they finish
Instead of the test index in the list of tests as it is
meaningless to the user and feels weird.

Also minor fix in the test name display when running with --forever.
2018-03-23 21:04:46 -03:00
Thibault Saunier
2b40c6c306 launcher: Clean up outpout
Make our stdout output simpler to follow by:
  - Not printing the tests we launch (it is not really useful in the end)
  - Using `\r` when printing the passed tests
  - Not reprinting all the test in a now useless summary
2018-03-23 18:12:16 -03:00
Edward Hervey
1a95559045 validate: Reduce time waiting for subprocess to stop
stopping the subprocess is done from the main thread, this would
throttle starting/stopping any tests by one second.
Start with 50ms, and gradually increase the wait between iterations
2017-12-03 11:38:08 +01:00
Thibault Saunier
4dc95f0fc2 launcher: Automatically disable output coloration if not supported 2017-08-26 10:52:25 -03:00
Thibault Saunier
3c62c315a9 validate: Implement RTSP support 2017-06-07 11:17:53 -04:00
Thibault Saunier
addd76fc85 validate: Mark tests as SKIPPED when installation is missing a GStreamer plugin 2017-02-07 13:12:09 -03:00
Thibault Saunier
f24e22446b validate: Properly kill subprocesses on windows 2017-01-03 16:24:19 -03:00
Thibault Saunier
acfdd04e01 validate:launcher: Allow specifying timeout as a known issue
And minor cleanups
2016-11-28 19:37:08 -03:00
Thibault Saunier
e132c11a95 validate:launcher: Handle checking bug status for expected failures 2016-11-28 18:40:00 -03:00
Thibault Saunier
2857eaf2ad validate: launcher: Allow checking if bugs linked to blacklist is fixed 2016-11-16 17:39:45 -03:00
Thibault Saunier
8bd0cb13e8 validate:launcher: Timeout if running gdb takes too much time 2016-11-14 18:44:27 -03:00
Thibault Saunier
20879e9c69 validate:launcher: Fix usage in an uninstalled environment 2016-11-09 17:38:04 -03:00
Thibault Saunier
5452672e89 validate:launcher: Try to generate a backtrace on segfaults 2016-11-09 10:19:45 -03:00
Thibault Saunier
1e51aeb942 validate:launcher: Port to Python3
And sync logging.py with Pitivi version
2016-11-09 10:13:42 -03:00
Thibault Saunier
9c9fe14293 meson:validate:test: Properly set paths to run launcher based tests
Adding a --validate-tools-path option to the launcher, allowing
to pass it from meson.
2016-09-09 12:24:54 -03:00
Vineeth T M
65560d8b03 Validate: Fix consider, launch spelling mistakes
https://bugzilla.gnome.org/show_bug.cgi?id=763289
2016-03-08 09:28:11 +02:00
Sebastian Dröge
233c99fbb7 validate/launcher: Treat DASH like HLS in another place 2015-08-15 16:19:46 +02:00
Vineeth T M
074406bc2c validate:launcher: unquote the path to remove special characters
When getting path from url using, url2path, it is returning
special characters (%20 for space etc..), instead of returning
plain path. path.unquote replaces the same..

https://bugzilla.gnome.org/show_bug.cgi?id=752611
2015-07-22 16:39:43 +01:00
Guillaume Desmottes
f6681ff003 validate: rename get_valgrind_suppression_file()
Summary:
This function is actually not specific to valgrind so we can make it more
generic.

Reviewers: thiblahute

Differential Revision: http://phabricator.freedesktop.org/D162
2015-05-11 21:55:24 +02:00
Guillaume Desmottes
c5c39d88b1 validate: move look_for_file_in_source_dir and get_valgrind_suppression_file to utils
Reviewers: thiblahute

Differential Revision: http://phabricator.freedesktop.org/D130
2015-05-01 15:15:26 +02:00
Thibault Saunier
22fba81138 validate: Rename gst-qa-assets to gst-integration-testsuites 2014-12-05 20:51:43 +01:00
Thibault Saunier
71566688bf validate: Remove the default testsuite implementation
The default testsuite implementation should belong to the default
asset repo where we have the corresponding knowledge.

We should style manage a sensible list of known blacklisted tests,
encoding profiles, and generators in GstValidate itself and allow testsuite
actual implementations to easily use them though the register_default_*
methods.

This allow us to be able to remove the ugly execfile() call.
2014-12-05 12:17:40 +01:00
Thibault Saunier
36a2d61068 validate:launcher: Force clock sync for some protocols
In HLS for example, not having clock sync might lead to races and failures
do not test that for now
2014-11-20 12:09:29 +01:00
Mathieu Duponchelle
9bdc6e9b6d validate-launcher: utils: fix various pyflakes / uncaught pep8 issues.
https://bugzilla.gnome.org/show_bug.cgi?id=739208
2014-10-30 15:40:17 +01:00
Mathieu Duponchelle
8c1e84b5f4 validate-launcher: pep8ify sources.
https://bugzilla.gnome.org/show_bug.cgi?id=739208
2014-10-30 15:40:17 +01:00
Mathieu Duponchelle
b0beefacfa validate-launcher: restructure filesystem
https://bugzilla.gnome.org/show_bug.cgi?id=739091
2014-10-30 15:31:29 +01:00
Renamed from validate/tools/launcher/utils.py (Browse further)