Commit graph

258 commits

Author SHA1 Message Date
Thibault Saunier 2036d8292f validate: Enhance error reporting when scenario or configs are invalid 2020-02-25 11:28:35 -03:00
Thibault Saunier 2f135d430b validate:launcher: Handle validate report bigger than allowed by the socket
This almost never happens but I had a case where we had a report
with a GstSample in the caps that were reported leading to an
error printed.
2020-02-13 00:11:45 +00:00
Thibault Saunier d17c2ff65c validate:launcher: Reference exception in a var as we use it in the handling 2020-02-13 00:11:45 +00:00
Thibault Saunier 600c5a27ed validate:launcher: Print the duration of the test run in the logs 2020-02-04 16:09:36 -03:00
Stéphane Cerveau a2e926ff0a gst-validate-launcher: separate known error from passed tests
Introduce known_error in statistics to keep in mind the expected
error result.
2020-01-24 14:52:49 +00:00
Nicolas Dufresne 1bc8e92efc launcher: Allow partionning the tests
This introduce new command line options, --parts and --part-index. When
--parts is set to a value larger then 1, the tests will be split in the
same number of group. The group number identified by --part-index will
be executed.

This is being added in orther to support gliblab CI parallel feature.
2020-01-15 10:04:11 -05:00
Thibault Saunier e4ca67938e validate:launcher: Try to dump logs with bat if avalaible
We got to many issues with mdv, it seems not to be maintained
bat is a very good replacement.
2020-01-06 13:14:59 +00:00
Thibault Saunier 195d3a3edc validate:launcher: Enhance progress reporting using a progress bar
This also allows us to properly report progress on the CI
2019-12-30 16:52:22 +00:00
Thibault Saunier 562750213f validate: launcher: Make encoding extra check use common code path
Reusing the reporting infrastructure instead of shurtcuting it
2019-11-20 18:07:48 -03:00
Thibault Saunier 8a6ee4841b validate: Add vp9 transcoding tests
Making sure to encode small frames as vp9enc is slow.
2019-11-20 18:07:47 -03:00
Thibault Saunier 80de180cc7 validate:launcher: Avoid clashes when importing testsuite
This introduce an hard dependency on python >= 3.5, same as meson
2019-11-11 18:57:27 -03:00
Thibault Saunier 2153d82f34 validate:launcher: Fix the 'can-happen-several-times' known issue field 2019-10-02 19:22:31 +00:00
Thibault Saunier 461e479802 validate:launcher: Do not hardcode pathsep 2019-08-28 15:34:31 -04:00
Thibault Saunier 3343f166da validate:launcher: Move get_fakesink_for_media_type to utils
So it can be reused in other apps like GES
2019-06-26 12:09:03 -04:00
Thibault Saunier 7d471ee25e validate: Set 'LOGSDIR' variable in scenarios and config files
Implementing support for variables in config files.
2019-06-23 03:10:34 -04:00
Thibault Saunier a994dd0ca8 validate:launcher: Strip env vars in command line outputing verbose
But activate if activating verbosity more than once
2019-06-18 18:13:03 -04:00
Thibault Saunier 010e18862f validate:launcher: Avoid repeating failure info in summaries 2019-06-11 21:03:56 +00:00
Thibault Saunier 7161b21334 validate:launcher: Do not dump to big log files
Avoiding ' The script exceeded the maximum execution time set for the job' in GitLab
2019-05-13 11:37:38 -04:00
Thibault Saunier b9cb5db869 validate:launcher: Ignore possibly lost for now 2019-04-17 09:06:55 -04:00
Mathieu Duponchelle 96361bd426 TestsManager: _add_blacklist in set_default_blacklist
Otherwise test suites that want to set_default_blacklist and
add tests in setup_tests were seeing their blacklist ignored.

Split up and rename set_blacklists() to complete the refactoring
2019-04-17 02:31:30 +02:00
Mathieu Duponchelle 61d0605852 Check indirect leaks 2019-04-17 02:29:36 +02:00
Thibault Saunier 3fa393c498 validate:launcher: Do not copy logs for non flaky tests
That looks weird for users and is incorrect
2019-04-15 16:13:52 -04:00
Thibault Saunier 81b1368237 validate:launcher: Fix printing debug logs URIs 2019-04-15 11:50:16 -04:00
Thibault Saunier e182272262 validate:launcher: Do not stop tcp server when reiterating tests runs 2019-04-12 12:33:25 -04:00
Thibault Saunier d709cb54a9 validate:launcher: Associate issues with the bug they come from
Making it simpler to follow when print the known issues
2019-04-12 10:13:15 -04:00
Thibault Saunier 81f198f812 validate:launcher: Indent known issues printing 2019-04-12 10:12:45 -04:00
Thibault Saunier e1625da9ed validate:launcher: Print the number of times the test was run when --forever 2019-04-11 10:38:46 -04:00
Thibault Saunier d053a07706 validate:launcher: Use md viewer to dump md logs if avalaible
And enhance the markdown
See https://github.com/axiros/terminal_markdown_viewer
2019-04-11 13:17:14 +00:00
Thibault Saunier 8639dc528c validate:launcher: Add an option to output HTML if commonmark is installed 2019-04-11 13:17:14 +00:00
Thibault Saunier 1294cb7b9b validate:launcher: Move all logs to one single log files to be displayed to end user 2019-04-11 13:17:14 +00:00
Thibault Saunier 446f1d8197 validate:launcher: Make extra_logfiles a set and fix their names 2019-04-11 13:17:14 +00:00
Thibault Saunier adbbfa1ec1 validate:launcher: Remove noise about empty known issues in logs 2019-04-11 13:17:14 +00:00
Thibault Saunier 148bb081b8 validate:launcher: Fix printing application name in the logs 2019-04-11 13:17:14 +00:00
Thibault Saunier 787939f750 validate:launcher: Add a list of well known subpression files from gst-build subprojects
Fixes https://gitlab.freedesktop.org/gstreamer/gst-devtools/issues/38
2019-04-03 13:38:42 +00:00
Thibault Saunier 92f1979ec9 validate: Add a mecanism to rerun failling tests
And add a way to mark some 'flakes' as tolerated
2019-04-03 13:38:42 +00:00
Thibault Saunier 520c2102cc validate:launcher: Reset process on clean 2019-04-03 13:38:42 +00:00
Thibault Saunier e279b1ff56 validate:launcher: Dump applied known issues in the logs 2019-04-03 13:38:42 +00:00
Thibault Saunier b0cbae0fa7 validate:launcher: Print name of the test to be debugged
When --debug was passed
2019-04-03 13:38:42 +00:00
Thibault Saunier abaa382847 validate: Generate dot files and link to them on CI 2019-04-03 13:38:42 +00:00
Thibault Saunier 28d413f059 validate:launcher: Add a way to say that a known issues can happen several times 2019-04-03 13:38:42 +00:00
Thibault Saunier ba69336bf8 validate:launcher: Remove now useless code to check sending EOS brings down the pipeline
If we want to make an exception we can use known issues these days
2019-04-03 13:38:42 +00:00
Thibault Saunier d73e283691 validate:launcher: Be a bit more resilient in GstValidateListener
And handle exception decoding received json info
2019-04-03 13:38:42 +00:00
Thibault Saunier 374917b9c4 validate:launcher: Add a way to require a non fatal error when matching a fatal one 2019-04-03 13:38:42 +00:00
Thibault Saunier ddb81f29e8 validate:launcher: factor out TTY check and enhance iteration output 2019-04-03 13:38:42 +00:00
Thibault Saunier 0a2af1b027 validate:launcher: Add generated known issue list to the logs instead of printing then 2019-04-03 13:38:42 +00:00
Thibault Saunier ac7efe9500 validate:launcher: Rework expected-issues data format
Instead of having the issues centered on the test classes, they
are now focusing on the "bug".

And harmise names on `expected_issue` not `expected_failures`
2019-04-03 13:38:42 +00:00
Thibault Saunier 6a4639352b validate:launcher: Allow printing log URL on a CI server 2019-04-03 13:38:42 +00:00
Thibault Saunier 06b8f23a34 validate:launcher: Fix getting skip-parser when generating media descriptor
On old version it didn't exist
2019-04-03 13:38:42 +00:00
Thibault Saunier a77825ecbc validate:launcher: Allow referencing known issue by exiting signal names 2019-04-03 13:38:42 +00:00
Thibault Saunier 7dbdb84ca8 validate:launcher: Output markdown as much as possible 2019-04-03 13:38:42 +00:00