Commit graph

413 commits

Author SHA1 Message Date
Thibault Saunier
556bc0bb4f launcher: Raise an exception when provided scenario can't be found 2019-08-19 10:33:20 +00:00
Mathieu Duponchelle
5a068bff72 validate: Update blacklisting reason for fast forward rtsp
While https://gitlab.freedesktop.org/gstreamer/gst-rtsp-server/issues/14
was merged, the client side (in particular rtpbasedepayload) still
isn't expected to work appropriately
2019-08-01 21:04:12 +02:00
Mathieu Duponchelle
93114b55ab json tests: assume all user pipelines can seek reverse
When doing a targeted test, it is up to the user to make sure
their pipeline + scenario behaves correctly.
2019-07-26 02:26:20 +02:00
Thibault Saunier
ba4275ad00 validate:launcher: Pass the right timeout_factor is passed to subprojects 2019-07-08 23:37:31 -04:00
Thibault Saunier
3343f166da validate:launcher: Move get_fakesink_for_media_type to utils
So it can be reused in other apps like GES
2019-06-26 12:09:03 -04:00
Thibault Saunier
7d471ee25e validate: Set 'LOGSDIR' variable in scenarios and config files
Implementing support for variables in config files.
2019-06-23 03:10:34 -04:00
Thibault Saunier
a994dd0ca8 validate:launcher: Strip env vars in command line outputing verbose
But activate if activating verbosity more than once
2019-06-18 18:13:03 -04:00
Thibault Saunier
010e18862f validate:launcher: Avoid repeating failure info in summaries 2019-06-11 21:03:56 +00:00
Thibault Saunier
7161b21334 validate:launcher: Do not dump to big log files
Avoiding ' The script exceeded the maximum execution time set for the job' in GitLab
2019-05-13 11:37:38 -04:00
Tim-Philipp Müller
22e179cbc1 launcher: testsuites: skip systemclock stress tests
These are very flaky when the build bots are under load.
2019-05-04 21:00:33 +00:00
Thibault Saunier
0544232d73 validate:check: Mare nle test_simple_operation as long 2019-04-17 18:10:12 -04:00
Thibault Saunier
fe6443090c validate:check: Fix some mistakes translating regex from gitlab-ci.yml 2019-04-17 18:10:12 -04:00
Thibault Saunier
d60ea5da63 validate:launcher: Set ORC_CODE=backup when running gst unit tests under valgrind 2019-04-17 11:04:45 -04:00
Thibault Saunier
1fb0d73947 validate:check: Blacklist and mark some GES tests as long under valgrind 2019-04-17 10:43:42 -04:00
Thibault Saunier
96f996b66a validate:testsuites: Add unit tests blacklists 2019-04-17 10:30:02 -04:00
Thibault Saunier
b9cb5db869 validate:launcher: Ignore possibly lost for now 2019-04-17 09:06:55 -04:00
Mathieu Duponchelle
96361bd426 TestsManager: _add_blacklist in set_default_blacklist
Otherwise test suites that want to set_default_blacklist and
add tests in setup_tests were seeing their blacklist ignored.

Split up and rename set_blacklists() to complete the refactoring
2019-04-17 02:31:30 +02:00
Mathieu Duponchelle
61d0605852 Check indirect leaks 2019-04-17 02:29:36 +02:00
Thibault Saunier
3fa393c498 validate:launcher: Do not copy logs for non flaky tests
That looks weird for users and is incorrect
2019-04-15 16:13:52 -04:00
Thibault Saunier
81b1368237 validate:launcher: Fix printing debug logs URIs 2019-04-15 11:50:16 -04:00
Thibault Saunier
e182272262 validate:launcher: Do not stop tcp server when reiterating tests runs 2019-04-12 12:33:25 -04:00
Thibault Saunier
d709cb54a9 validate:launcher: Associate issues with the bug they come from
Making it simpler to follow when print the known issues
2019-04-12 10:13:15 -04:00
Thibault Saunier
81f198f812 validate:launcher: Indent known issues printing 2019-04-12 10:12:45 -04:00
Thibault Saunier
e1625da9ed validate:launcher: Print the number of times the test was run when --forever 2019-04-11 10:38:46 -04:00
Thibault Saunier
0a56447bcc validate:launcher: Add python suppression files 2019-04-11 13:17:14 +00:00
Thibault Saunier
d053a07706 validate:launcher: Use md viewer to dump md logs if avalaible
And enhance the markdown
See https://github.com/axiros/terminal_markdown_viewer
2019-04-11 13:17:14 +00:00
Thibault Saunier
ebf0844d2c validate:launcher: Avoid forking when running gstcheck tests in gdb 2019-04-11 13:17:14 +00:00
Thibault Saunier
8639dc528c validate:launcher: Add an option to output HTML if commonmark is installed 2019-04-11 13:17:14 +00:00
Thibault Saunier
f7517e503c validate:launcher: Do not user python 3.5 features 2019-04-11 13:17:14 +00:00
Thibault Saunier
1294cb7b9b validate:launcher: Move all logs to one single log files to be displayed to end user 2019-04-11 13:17:14 +00:00
Thibault Saunier
446f1d8197 validate:launcher: Make extra_logfiles a set and fix their names 2019-04-11 13:17:14 +00:00
Thibault Saunier
2fcdc27c3a validate:launcher: Remove spurious print 2019-04-11 13:17:14 +00:00
Thibault Saunier
adbbfa1ec1 validate:launcher: Remove noise about empty known issues in logs 2019-04-11 13:17:14 +00:00
Thibault Saunier
148bb081b8 validate:launcher: Fix printing application name in the logs 2019-04-11 13:17:14 +00:00
Thibault Saunier
787939f750 validate:launcher: Add a list of well known subpression files from gst-build subprojects
Fixes https://gitlab.freedesktop.org/gstreamer/gst-devtools/issues/38
2019-04-03 13:38:42 +00:00
Thibault Saunier
92f1979ec9 validate: Add a mecanism to rerun failling tests
And add a way to mark some 'flakes' as tolerated
2019-04-03 13:38:42 +00:00
Thibault Saunier
520c2102cc validate:launcher: Reset process on clean 2019-04-03 13:38:42 +00:00
Thibault Saunier
e279b1ff56 validate:launcher: Dump applied known issues in the logs 2019-04-03 13:38:42 +00:00
Thibault Saunier
de007b6819 validate:launcher: Put all logs inside the failure node
Pleasing gitlab CI reporting system
2019-04-03 13:38:42 +00:00
Thibault Saunier
8da6ecef13 validate:launcher: By default use cpu_count / 2 2019-04-03 13:38:42 +00:00
Thibault Saunier
b0cbae0fa7 validate:launcher: Print name of the test to be debugged
When --debug was passed
2019-04-03 13:38:42 +00:00
Thibault Saunier
abaa382847 validate: Generate dot files and link to them on CI 2019-04-03 13:38:42 +00:00
Thibault Saunier
01aa026e5a validate:launcher: Try to send SIGINT before killing processes
Giving a chance for validate to print reports
2019-04-03 13:38:42 +00:00
Thibault Saunier
28d413f059 validate:launcher: Add a way to say that a known issues can happen several times 2019-04-03 13:38:42 +00:00
Thibault Saunier
ba69336bf8 validate:launcher: Remove now useless code to check sending EOS brings down the pipeline
If we want to make an exception we can use known issues these days
2019-04-03 13:38:42 +00:00
Thibault Saunier
d73e283691 validate:launcher: Be a bit more resilient in GstValidateListener
And handle exception decoding received json info
2019-04-03 13:38:42 +00:00
Thibault Saunier
374917b9c4 validate:launcher: Add a way to require a non fatal error when matching a fatal one 2019-04-03 13:38:42 +00:00
Thibault Saunier
f2e180e51f validate:launcher: Add a way to force coloration 2019-04-03 13:38:42 +00:00
Thibault Saunier
ddb81f29e8 validate:launcher: factor out TTY check and enhance iteration output 2019-04-03 13:38:42 +00:00
Thibault Saunier
0a2af1b027 validate:launcher: Add generated known issue list to the logs instead of printing then 2019-04-03 13:38:42 +00:00