Thibault Saunier
2036d8292f
validate: Enhance error reporting when scenario or configs are invalid
2020-02-25 11:28:35 -03:00
Thibault Saunier
2f135d430b
validate:launcher: Handle validate report bigger than allowed by the socket
...
This almost never happens but I had a case where we had a report
with a GstSample in the caps that were reported leading to an
error printed.
2020-02-13 00:11:45 +00:00
Thibault Saunier
d17c2ff65c
validate:launcher: Reference exception in a var as we use it in the handling
2020-02-13 00:11:45 +00:00
Thibault Saunier
600c5a27ed
validate:launcher: Print the duration of the test run in the logs
2020-02-04 16:09:36 -03:00
Stéphane Cerveau
a2e926ff0a
gst-validate-launcher: separate known error from passed tests
...
Introduce known_error in statistics to keep in mind the expected
error result.
2020-01-24 14:52:49 +00:00
Nicolas Dufresne
1bc8e92efc
launcher: Allow partionning the tests
...
This introduce new command line options, --parts and --part-index. When
--parts is set to a value larger then 1, the tests will be split in the
same number of group. The group number identified by --part-index will
be executed.
This is being added in orther to support gliblab CI parallel feature.
2020-01-15 10:04:11 -05:00
Thibault Saunier
e4ca67938e
validate:launcher: Try to dump logs with bat if avalaible
...
We got to many issues with mdv, it seems not to be maintained
bat is a very good replacement.
2020-01-06 13:14:59 +00:00
Thibault Saunier
195d3a3edc
validate:launcher: Enhance progress reporting using a progress bar
...
This also allows us to properly report progress on the CI
2019-12-30 16:52:22 +00:00
Thibault Saunier
562750213f
validate: launcher: Make encoding extra check use common code path
...
Reusing the reporting infrastructure instead of shurtcuting it
2019-11-20 18:07:48 -03:00
Thibault Saunier
8a6ee4841b
validate: Add vp9 transcoding tests
...
Making sure to encode small frames as vp9enc is slow.
2019-11-20 18:07:47 -03:00
Thibault Saunier
80de180cc7
validate:launcher: Avoid clashes when importing testsuite
...
This introduce an hard dependency on python >= 3.5, same as meson
2019-11-11 18:57:27 -03:00
Thibault Saunier
2153d82f34
validate:launcher: Fix the 'can-happen-several-times' known issue field
2019-10-02 19:22:31 +00:00
Thibault Saunier
461e479802
validate:launcher: Do not hardcode pathsep
2019-08-28 15:34:31 -04:00
Thibault Saunier
3343f166da
validate:launcher: Move get_fakesink_for_media_type to utils
...
So it can be reused in other apps like GES
2019-06-26 12:09:03 -04:00
Thibault Saunier
7d471ee25e
validate: Set 'LOGSDIR' variable in scenarios and config files
...
Implementing support for variables in config files.
2019-06-23 03:10:34 -04:00
Thibault Saunier
a994dd0ca8
validate:launcher: Strip env vars in command line outputing verbose
...
But activate if activating verbosity more than once
2019-06-18 18:13:03 -04:00
Thibault Saunier
010e18862f
validate:launcher: Avoid repeating failure info in summaries
2019-06-11 21:03:56 +00:00
Thibault Saunier
7161b21334
validate:launcher: Do not dump to big log files
...
Avoiding ' The script exceeded the maximum execution time set for the job' in GitLab
2019-05-13 11:37:38 -04:00
Thibault Saunier
b9cb5db869
validate:launcher: Ignore possibly lost for now
2019-04-17 09:06:55 -04:00
Mathieu Duponchelle
96361bd426
TestsManager: _add_blacklist in set_default_blacklist
...
Otherwise test suites that want to set_default_blacklist and
add tests in setup_tests were seeing their blacklist ignored.
Split up and rename set_blacklists() to complete the refactoring
2019-04-17 02:31:30 +02:00
Mathieu Duponchelle
61d0605852
Check indirect leaks
2019-04-17 02:29:36 +02:00
Thibault Saunier
3fa393c498
validate:launcher: Do not copy logs for non flaky tests
...
That looks weird for users and is incorrect
2019-04-15 16:13:52 -04:00
Thibault Saunier
81b1368237
validate:launcher: Fix printing debug logs URIs
2019-04-15 11:50:16 -04:00
Thibault Saunier
e182272262
validate:launcher: Do not stop tcp server when reiterating tests runs
2019-04-12 12:33:25 -04:00
Thibault Saunier
d709cb54a9
validate:launcher: Associate issues with the bug they come from
...
Making it simpler to follow when print the known issues
2019-04-12 10:13:15 -04:00
Thibault Saunier
81f198f812
validate:launcher: Indent known issues printing
2019-04-12 10:12:45 -04:00
Thibault Saunier
e1625da9ed
validate:launcher: Print the number of times the test was run when --forever
2019-04-11 10:38:46 -04:00
Thibault Saunier
d053a07706
validate:launcher: Use md viewer to dump md logs if avalaible
...
And enhance the markdown
See https://github.com/axiros/terminal_markdown_viewer
2019-04-11 13:17:14 +00:00
Thibault Saunier
8639dc528c
validate:launcher: Add an option to output HTML if commonmark is installed
2019-04-11 13:17:14 +00:00
Thibault Saunier
1294cb7b9b
validate:launcher: Move all logs to one single log files to be displayed to end user
2019-04-11 13:17:14 +00:00
Thibault Saunier
446f1d8197
validate:launcher: Make extra_logfiles a set and fix their names
2019-04-11 13:17:14 +00:00
Thibault Saunier
adbbfa1ec1
validate:launcher: Remove noise about empty known issues in logs
2019-04-11 13:17:14 +00:00
Thibault Saunier
148bb081b8
validate:launcher: Fix printing application name in the logs
2019-04-11 13:17:14 +00:00
Thibault Saunier
787939f750
validate:launcher: Add a list of well known subpression files from gst-build subprojects
...
Fixes https://gitlab.freedesktop.org/gstreamer/gst-devtools/issues/38
2019-04-03 13:38:42 +00:00
Thibault Saunier
92f1979ec9
validate: Add a mecanism to rerun failling tests
...
And add a way to mark some 'flakes' as tolerated
2019-04-03 13:38:42 +00:00
Thibault Saunier
520c2102cc
validate:launcher: Reset process on clean
2019-04-03 13:38:42 +00:00
Thibault Saunier
e279b1ff56
validate:launcher: Dump applied known issues in the logs
2019-04-03 13:38:42 +00:00
Thibault Saunier
b0cbae0fa7
validate:launcher: Print name of the test to be debugged
...
When --debug was passed
2019-04-03 13:38:42 +00:00
Thibault Saunier
abaa382847
validate: Generate dot files and link to them on CI
2019-04-03 13:38:42 +00:00
Thibault Saunier
28d413f059
validate:launcher: Add a way to say that a known issues can happen several times
2019-04-03 13:38:42 +00:00
Thibault Saunier
ba69336bf8
validate:launcher: Remove now useless code to check sending EOS brings down the pipeline
...
If we want to make an exception we can use known issues these days
2019-04-03 13:38:42 +00:00
Thibault Saunier
d73e283691
validate:launcher: Be a bit more resilient in GstValidateListener
...
And handle exception decoding received json info
2019-04-03 13:38:42 +00:00
Thibault Saunier
374917b9c4
validate:launcher: Add a way to require a non fatal error when matching a fatal one
2019-04-03 13:38:42 +00:00
Thibault Saunier
ddb81f29e8
validate:launcher: factor out TTY check and enhance iteration output
2019-04-03 13:38:42 +00:00
Thibault Saunier
0a2af1b027
validate:launcher: Add generated known issue list to the logs instead of printing then
2019-04-03 13:38:42 +00:00
Thibault Saunier
ac7efe9500
validate:launcher: Rework expected-issues data format
...
Instead of having the issues centered on the test classes, they
are now focusing on the "bug".
And harmise names on `expected_issue` not `expected_failures`
2019-04-03 13:38:42 +00:00
Thibault Saunier
6a4639352b
validate:launcher: Allow printing log URL on a CI server
2019-04-03 13:38:42 +00:00
Thibault Saunier
06b8f23a34
validate:launcher: Fix getting skip-parser when generating media descriptor
...
On old version it didn't exist
2019-04-03 13:38:42 +00:00
Thibault Saunier
a77825ecbc
validate:launcher: Allow referencing known issue by exiting signal names
2019-04-03 13:38:42 +00:00
Thibault Saunier
7dbdb84ca8
validate:launcher: Output markdown as much as possible
2019-04-03 13:38:42 +00:00