Commit graph

404 commits

Author SHA1 Message Date
Tim-Philipp Müller
22e179cbc1 launcher: testsuites: skip systemclock stress tests
These are very flaky when the build bots are under load.
2019-05-04 21:00:33 +00:00
Thibault Saunier
0544232d73 validate:check: Mare nle test_simple_operation as long 2019-04-17 18:10:12 -04:00
Thibault Saunier
fe6443090c validate:check: Fix some mistakes translating regex from gitlab-ci.yml 2019-04-17 18:10:12 -04:00
Thibault Saunier
d60ea5da63 validate:launcher: Set ORC_CODE=backup when running gst unit tests under valgrind 2019-04-17 11:04:45 -04:00
Thibault Saunier
1fb0d73947 validate:check: Blacklist and mark some GES tests as long under valgrind 2019-04-17 10:43:42 -04:00
Thibault Saunier
96f996b66a validate:testsuites: Add unit tests blacklists 2019-04-17 10:30:02 -04:00
Thibault Saunier
b9cb5db869 validate:launcher: Ignore possibly lost for now 2019-04-17 09:06:55 -04:00
Mathieu Duponchelle
96361bd426 TestsManager: _add_blacklist in set_default_blacklist
Otherwise test suites that want to set_default_blacklist and
add tests in setup_tests were seeing their blacklist ignored.

Split up and rename set_blacklists() to complete the refactoring
2019-04-17 02:31:30 +02:00
Mathieu Duponchelle
61d0605852 Check indirect leaks 2019-04-17 02:29:36 +02:00
Thibault Saunier
3fa393c498 validate:launcher: Do not copy logs for non flaky tests
That looks weird for users and is incorrect
2019-04-15 16:13:52 -04:00
Thibault Saunier
81b1368237 validate:launcher: Fix printing debug logs URIs 2019-04-15 11:50:16 -04:00
Thibault Saunier
e182272262 validate:launcher: Do not stop tcp server when reiterating tests runs 2019-04-12 12:33:25 -04:00
Thibault Saunier
d709cb54a9 validate:launcher: Associate issues with the bug they come from
Making it simpler to follow when print the known issues
2019-04-12 10:13:15 -04:00
Thibault Saunier
81f198f812 validate:launcher: Indent known issues printing 2019-04-12 10:12:45 -04:00
Thibault Saunier
e1625da9ed validate:launcher: Print the number of times the test was run when --forever 2019-04-11 10:38:46 -04:00
Thibault Saunier
0a56447bcc validate:launcher: Add python suppression files 2019-04-11 13:17:14 +00:00
Thibault Saunier
d053a07706 validate:launcher: Use md viewer to dump md logs if avalaible
And enhance the markdown
See https://github.com/axiros/terminal_markdown_viewer
2019-04-11 13:17:14 +00:00
Thibault Saunier
ebf0844d2c validate:launcher: Avoid forking when running gstcheck tests in gdb 2019-04-11 13:17:14 +00:00
Thibault Saunier
8639dc528c validate:launcher: Add an option to output HTML if commonmark is installed 2019-04-11 13:17:14 +00:00
Thibault Saunier
f7517e503c validate:launcher: Do not user python 3.5 features 2019-04-11 13:17:14 +00:00
Thibault Saunier
1294cb7b9b validate:launcher: Move all logs to one single log files to be displayed to end user 2019-04-11 13:17:14 +00:00
Thibault Saunier
446f1d8197 validate:launcher: Make extra_logfiles a set and fix their names 2019-04-11 13:17:14 +00:00
Thibault Saunier
2fcdc27c3a validate:launcher: Remove spurious print 2019-04-11 13:17:14 +00:00
Thibault Saunier
adbbfa1ec1 validate:launcher: Remove noise about empty known issues in logs 2019-04-11 13:17:14 +00:00
Thibault Saunier
148bb081b8 validate:launcher: Fix printing application name in the logs 2019-04-11 13:17:14 +00:00
Thibault Saunier
787939f750 validate:launcher: Add a list of well known subpression files from gst-build subprojects
Fixes https://gitlab.freedesktop.org/gstreamer/gst-devtools/issues/38
2019-04-03 13:38:42 +00:00
Thibault Saunier
92f1979ec9 validate: Add a mecanism to rerun failling tests
And add a way to mark some 'flakes' as tolerated
2019-04-03 13:38:42 +00:00
Thibault Saunier
520c2102cc validate:launcher: Reset process on clean 2019-04-03 13:38:42 +00:00
Thibault Saunier
e279b1ff56 validate:launcher: Dump applied known issues in the logs 2019-04-03 13:38:42 +00:00
Thibault Saunier
de007b6819 validate:launcher: Put all logs inside the failure node
Pleasing gitlab CI reporting system
2019-04-03 13:38:42 +00:00
Thibault Saunier
8da6ecef13 validate:launcher: By default use cpu_count / 2 2019-04-03 13:38:42 +00:00
Thibault Saunier
b0cbae0fa7 validate:launcher: Print name of the test to be debugged
When --debug was passed
2019-04-03 13:38:42 +00:00
Thibault Saunier
abaa382847 validate: Generate dot files and link to them on CI 2019-04-03 13:38:42 +00:00
Thibault Saunier
01aa026e5a validate:launcher: Try to send SIGINT before killing processes
Giving a chance for validate to print reports
2019-04-03 13:38:42 +00:00
Thibault Saunier
28d413f059 validate:launcher: Add a way to say that a known issues can happen several times 2019-04-03 13:38:42 +00:00
Thibault Saunier
ba69336bf8 validate:launcher: Remove now useless code to check sending EOS brings down the pipeline
If we want to make an exception we can use known issues these days
2019-04-03 13:38:42 +00:00
Thibault Saunier
d73e283691 validate:launcher: Be a bit more resilient in GstValidateListener
And handle exception decoding received json info
2019-04-03 13:38:42 +00:00
Thibault Saunier
374917b9c4 validate:launcher: Add a way to require a non fatal error when matching a fatal one 2019-04-03 13:38:42 +00:00
Thibault Saunier
f2e180e51f validate:launcher: Add a way to force coloration 2019-04-03 13:38:42 +00:00
Thibault Saunier
ddb81f29e8 validate:launcher: factor out TTY check and enhance iteration output 2019-04-03 13:38:42 +00:00
Thibault Saunier
0a2af1b027 validate:launcher: Add generated known issue list to the logs instead of printing then 2019-04-03 13:38:42 +00:00
Thibault Saunier
c8a3b34a3a validate:launcher: Add a way to force stdout coloring 2019-04-03 13:38:42 +00:00
Thibault Saunier
ac7efe9500 validate:launcher: Rework expected-issues data format
Instead of having the issues centered on the test classes, they
are now focusing on the "bug".

And harmise names on `expected_issue` not `expected_failures`
2019-04-03 13:38:42 +00:00
Thibault Saunier
6a4639352b validate:launcher: Allow printing log URL on a CI server 2019-04-03 13:38:42 +00:00
Thibault Saunier
06b8f23a34 validate:launcher: Fix getting skip-parser when generating media descriptor
On old version it didn't exist
2019-04-03 13:38:42 +00:00
Thibault Saunier
19dfd63408 validate:launcher: Review default for --mute
`--mute` is now True by default and added `--unmute` if the user wants
to have visual/audio feedback. It was really annoying to have thousand
of window pop up by default
2019-04-03 13:38:42 +00:00
Thibault Saunier
a77825ecbc validate:launcher: Allow referencing known issue by exiting signal names 2019-04-03 13:38:42 +00:00
Thibault Saunier
7dbdb84ca8 validate:launcher: Output markdown as much as possible 2019-04-03 13:38:42 +00:00
Thibault Saunier
3a826e1e3d validate:launcher: Implement bug checks for gitlab
And use new gitlab urls for all the bugs
2019-04-03 13:38:42 +00:00
Thibault Saunier
8d00a74f1a validate:launcher: Enhance and standardize output 2019-04-03 13:38:42 +00:00