Commit graph

186 commits

Author SHA1 Message Date
Thibault Saunier
64afff0c3e launcher:scenario: Fix the way we compute scenario path/name when paths supplied
We were just iterating over the list without any check and could end
up with a patch that was not corresponding to the actual scenario.
2018-09-08 11:13:19 -03:00
Thibault Saunier
f0cfdf9d14 validate:launcher: Use fakevideosink everywhere it makes sense. 2018-09-07 16:19:40 -03:00
Alicia Boya García
5ff299f4e7 gst-validate-launcher: Print copypaste-friendlier commands
This patch removes the quotes surrounding the command shown by
gst-validate to reproduce the issues -- which were troublesome when
copying and pasting.

It also introduces escaping for the arguments, so that the command line
can be copied and pasted in the terminal without further changes.

https://bugzilla.gnome.org/show_bug.cgi?id=796897
2018-07-30 16:13:25 -04:00
Thibault Saunier
615a372274 validate: launcher: Print some ERROR log when inspecting scenario fails 2018-07-21 11:43:40 -04:00
Thibault Saunier
4f009f4098 validate:launcher: Minor output string message 2018-07-12 19:13:21 -04:00
Thibault Saunier
69ec48feef validate:launcher: Allow retrieving coredumps from within flatpak 2018-07-01 12:55:06 -04:00
Thibault Saunier
acb3a276c8 validate:launcher: Fix the --forever switch
It was not stopping on error.

https://bugzilla.gnome.org/show_bug.cgi?id=796608
2018-06-17 08:45:23 -04:00
Thibault Saunier
58c90448ca validate: launcher: Add support for running tests with a pushfile source
Introducing the `.media_info.push` media info extension, which is meant
to let the launcher know that those file should run with the "pushfile://"
protocol.

And allow symlinking "normal" `.media_info` to their `.pushfile` variant
so that both can share the exact same content.
2018-06-15 12:05:14 -04:00
Thibault Saunier
b2e71e1404 validate: media-check: Add a way to skip pluggin parsers
This is useful when you want to check only the demuxer output.

- Keep the information in the media file so that we can launch media-check
  with the proper arguments in the launcher. Update it accordingly.
- Refactor compare_streams to simplify it, which in the end leads to
  reporting all the issues instead of exiting on the first one.
2018-06-15 12:05:14 -04:00
Alicia Boya García
40dfb7174e gst-validate-launcher: let gdb handle SIGINT itself
Otherwise both gdb and gst-validate-launcher will react to ^C at the
same time, gdb will be killed by SIGHUP (because gst-validate-launcher
quitted in consequence of the ^C) and the terminal state will be left
garbled because readline inside gdb had disabled echo.

https://bugzilla.gnome.org/show_bug.cgi?id=796396
2018-05-25 12:59:41 +02:00
Alicia Boya García
bfa143caa9 gst-validate-launcher: disable timeouts when debugging in gdb interactively
An interactive debugging session can be going for a long time, we don't
want any timeouts in that case.

https://bugzilla.gnome.org/show_bug.cgi?id=796397
2018-05-25 12:59:40 +02:00
Alicia Boya García
2c32ccd7a7 gst-validate-launcher: Stop in --gdb by default, add --gdb-non-stop
This patch modifies the default behavior of --gdb to not run and quit
automatically the test, but rather wait for user input. This is
usually much more convenient to debug all kinds of bugs.

The automatic run behavior has been moved to a new command switch:
--gdb-non-stop

https://bugzilla.gnome.org/show_bug.cgi?id=796389
2018-05-25 12:59:38 +02:00
Thibault Saunier
bb71fd9944 validate:launcher: Error out loudly if the testlist changes
When --fail-on-testlist-change is set.
2018-05-15 14:45:54 -04:00
Thibault Saunier
2dc7dd1257 validate: launcher: Make sure testsuites are used/configured once only 2018-05-13 19:10:59 -04:00
Thibault Saunier
ef9ff93405 validate:launcher: Add a way to check if a gst feature is present
And make sure iqa is present to run IQA tests.
2018-05-13 16:35:41 -04:00
Thibault Saunier
b00ab02254 validate:launcher: Cleanup the way we find where -validate tools are 2018-05-13 09:44:17 -04:00
Thibault Saunier
c1f89b4acb validate: launcher: Add a way to simply run SSIM checks on rendered files
We will run a simple pipeline with the IQA element to run ssim (dssim)
tests on the rendered files, comparing it with a reference file.

For now we use the very empiric 1.0 value as a ssim error threshold and
the goal is basically to detect completely broken renderings.
2018-05-13 09:44:17 -04:00
Thibault Saunier
77ee198b1b validate:launcher: Add support for specifying a workdir in tests 2018-04-15 20:50:22 -03:00
Thibault Saunier
8382fddbe9 validate:launcher: Add a method to find tests in a TestManager 2018-04-12 23:30:19 -03:00
Thibault Saunier
55dee0b8a6 validate:launcher: Extract method to set a validate configuration on tests 2018-04-12 23:07:16 -03:00
Thibault Saunier
a70658a464 launcher: Print test number in the order they finish
Instead of the test index in the list of tests as it is
meaningless to the user and feels weird.

Also minor fix in the test name display when running with --forever.
2018-03-23 21:04:46 -03:00
Thibault Saunier
cd1c4eb44d launcher: Minor indentation issue fixes 2018-03-23 18:12:16 -03:00
Thibault Saunier
2b40c6c306 launcher: Clean up outpout
Make our stdout output simpler to follow by:
  - Not printing the tests we launch (it is not really useful in the end)
  - Using `\r` when printing the passed tests
  - Not reprinting all the test in a now useless summary
2018-03-23 18:12:16 -03:00
Edward Hervey
d71d28523d validate/baseclasses: Release un-needed data when test ends
This was keeping around 500-700kB of data for each test, which was
gradually raising memory usage of a full run by 100MB+

The reports are definitely not needed, and we only need to keep
information from the subprocess env variable that we might need
later on for final reporting
2017-12-18 10:51:05 +01:00
Edward Hervey
cb04515cbd validate/baseclasses: Don't leak several hundred MB of XML
The xml-based MediaDescriptor were keeping open the XML file and the
associated ElementTree structures, resulting in memory usage of several
hundred megabytes.

Instead cache the information we need immediately and release the
XML structure
2017-12-18 09:48:21 +01:00
Edward Hervey
92285ef261 validate: Use a single TCPServer for subprocess communication
Instead of creating a separate TCPServer for each test, just create
one which handles all connections in a threaded fashion.

Shaves off ~500ms per test

https://bugzilla.gnome.org/show_bug.cgi?id=791159
2017-12-03 12:23:29 +01:00
Edward Hervey
58e62f651c validate-launcher: Allow running tests out-of-order
When the --shuffle option is used, the tests will be run out of order.
This optimizes CPU utilization since it allows running synchronized
and unsynchronized tests at the same.
2017-12-03 11:42:18 +01:00
Thibault Saunier
f16b900643 validate:launcher: Launch tests in _TestsLauncher not in TestsManagaer
So that Test from several TestManager can run in parallel and thus avoid
waiting for tests from one TestManager to run the following one.,

Also by design TestsLauncher should always have been the responsible for
... launching tests.
2017-12-02 09:42:19 -03:00
Thibault Saunier
2c6c25d4bf validate:launcher: Use the number of failed test as exit code
We used to always return 0, which was not right!
2017-08-18 11:37:28 -03:00
Thibault Saunier
cb99482b9a launcher: Avoid exceptions when inspecting renders files
We were a bit to strict on the Exception types which lead to
the launcher failling itself when it shouldn't
2017-08-14 16:57:50 -03:00
Thibault Saunier
1a28e7b043 validate: Factor out a method to set properties on elements in utils
Make sure to use it where appropriate and add some logging when
setting an object property from an action.

And use the valgrind.conf to set all the properties instead of having
a mixture of a config scenario and the config file (making sure the
max-lateness is set on any sink)
2017-07-19 11:34:19 -04:00
Thibault Saunier
2177d8589c validate:launcher: Error out if no testsuite could be loaded 2017-07-18 12:09:13 -04:00
Thibault Saunier
2cf93f491b validate: launcher: Namespace test name with the testsuite name
Also allowing users to pass test names directly
2017-07-18 10:54:53 -04:00
Thibault Saunier
f9ef2bc56a validate: launcher: Properly use TestsLauncher.list_test to load tests
Otherwise we might skip check_defined_tests.
2017-07-18 10:54:21 -04:00
Thibault Saunier
40a08e6080 validate: launcher: Add some missing env variables in command to launch test 2017-07-11 10:11:49 -04:00
Thibault Saunier
c540601ed0 validate:launcher: Do not check ModuleNotFound exception
It is a subclass of ImportError and is avalaible only since 3.6
https://ci.gstreamer.net/job/pitivi-flatpak/626/console
2017-06-28 15:54:13 -04:00
Thibault Saunier
8ea68e9e06 validate:launcher: Speed up xml parsing using lxml if avalaible 2017-06-27 16:50:35 -04:00
Thibault Saunier
2f35acf558 validate:launcher: Enhance command printing when using a server
So it can be copy pasted and work
2017-06-20 15:46:15 -04:00
Thibault Saunier
335c14bf5b validate:launcher: Fix launching testsuite with relative paths 2017-06-16 17:36:34 -04:00
Thibault Saunier
9a45cd41a9 validate:launcher: Handle test that can't be run in parralel 2017-06-07 15:23:26 -04:00
Thibault Saunier
0e163ffae8 validate:launcher: Handle optional tests
For example RTSP tests might not be avalaible if gst-rtsp-server-example-uri is not avalaible
2017-06-07 11:17:53 -04:00
Thibault Saunier
0beb8647ce validate:launcher: Print logfiles when printing test result
It was often annoying to check wrong logs because the result are
not printed near the reference to logs.
2017-06-07 11:17:53 -04:00
Thibault Saunier
37de89e6db validate:launcher: Do not run scenarios that need prerolling on RTSP stream 2017-06-07 11:17:53 -04:00
Thibault Saunier
3c62c315a9 validate: Implement RTSP support 2017-06-07 11:17:53 -04:00
Thibault Saunier
02d716d6d8 validate:launcher: Handle not redirecting valgrind output 2017-06-01 16:57:50 -04:00
Edward Hervey
400d1f9326 validate: Implement Scenario.__repr__
Allows better debugging when looking at logs
2017-05-25 15:50:23 +02:00
Edward Hervey
9188968f5c validate: Add live-related features to scenarios and medias
Note: The notion of "live" here is in the *content* sense and not in the
GStreamer sense.
Ex:
  * A rtsp stream is always "live" in the GStreamer sense but might not always
    provide live content.
  * HLS/DASH streams are not "live" in the GStreamer sense but might
    provide "live" content.

Some scenarios might:
* require live content
* not be compatible with live content

This patch adds two new properties for scenarios:
* live_content_required (default False) for scenarios that can only work with
  live content.
* live_content_compatible (default False) for scenarios that can work with
  both live and non-live content.
This patch adds support for reading a "live" property from stream_info
2017-05-25 14:04:00 +02:00
Thibault Saunier
40a31b05b6 launcher: Allow using the base launcher as a test manager
No reason to force people to subclass it in simple cases.
2017-04-30 14:36:16 -03:00
Edward Hervey
325c1b37dd baseclasses: Add method to set a specific list of scenarios
https://bugzilla.gnome.org/show_bug.cgi?id=781314
2017-04-14 16:55:06 +02:00
Thibault Saunier
49271bc721 validate:launcher: Add information on media info files parsing failures 2017-02-27 12:12:38 -03:00