Commit graph

140 commits

Author SHA1 Message Date
Edward Hervey 9188968f5c validate: Add live-related features to scenarios and medias
Note: The notion of "live" here is in the *content* sense and not in the
GStreamer sense.
Ex:
  * A rtsp stream is always "live" in the GStreamer sense but might not always
    provide live content.
  * HLS/DASH streams are not "live" in the GStreamer sense but might
    provide "live" content.

Some scenarios might:
* require live content
* not be compatible with live content

This patch adds two new properties for scenarios:
* live_content_required (default False) for scenarios that can only work with
  live content.
* live_content_compatible (default False) for scenarios that can work with
  both live and non-live content.
This patch adds support for reading a "live" property from stream_info
2017-05-25 14:04:00 +02:00
Thibault Saunier 40a31b05b6 launcher: Allow using the base launcher as a test manager
No reason to force people to subclass it in simple cases.
2017-04-30 14:36:16 -03:00
Edward Hervey 325c1b37dd baseclasses: Add method to set a specific list of scenarios
https://bugzilla.gnome.org/show_bug.cgi?id=781314
2017-04-14 16:55:06 +02:00
Thibault Saunier 49271bc721 validate:launcher: Add information on media info files parsing failures 2017-02-27 12:12:38 -03:00
Thibault Saunier fbde653a0c validate: launcher: Fix the way we retrieve command name
We are now using a list of args for subprocess so just using it is simple now
2017-02-27 12:12:38 -03:00
Thibault Saunier 9caee62bc6 validate: Fix call to decode() on a string 2017-02-27 12:12:38 -03:00
Thibault Saunier f2b42cbaf8 validate:launcher: Various fixes to make the Test class directly usable
And the launcher installed with meson usable
2017-02-21 13:40:28 -03:00
Thibault Saunier 54a1fc8b30 validate:launcher: Take ValidateAction execution as a marker of test update 2017-02-17 15:29:41 -03:00
Thibault Saunier 40016b9956 validate:scenario: Report action done with the execution duration 2017-02-17 15:29:41 -03:00
Thibault Saunier 7773ca7765 validate:launcher: Add a way to specify a set of tests to run under the leak tracer
https://bugzilla.gnome.org/show_bug.cgi?id=767856
2017-02-10 15:56:11 -03:00
Thibault Saunier addd76fc85 validate: Mark tests as SKIPPED when installation is missing a GStreamer plugin 2017-02-07 13:12:09 -03:00
Thibault Saunier 823ce95d70 validate:launcher: Add an option for user to modify timeout values
Allowing to expand the test timeout when running on slow platforms
2017-02-06 12:16:41 -03:00
Thibault Saunier 5d1d8e6971 validate:launcher: Verify that Gst supression file could be found 2017-02-03 11:02:49 -03:00
Thibault Saunier de2cbda854 validate: Make sure to update children environment from current env
And minor fix in function call
2017-01-06 12:12:21 -03:00
Thibault Saunier ca1f8a03e0 validate:launcher: Make the IPC server socket blocking
0.0 does not mean blocking in python3 and makes everything failling
on windows.
2017-01-03 16:26:20 -03:00
Thibault Saunier f24e22446b validate: Properly kill subprocesses on windows 2017-01-03 16:24:19 -03:00
Thibault Saunier f6d0636466 validate:launcher: Fix running on windows 2017-01-03 15:34:39 -03:00
Thibault Saunier 2863986ea3 validate: Respect active testers when listing tests 2017-01-03 14:52:38 -03:00
Thibault Saunier 0ee2147137 Revert "Revert "validate:launcher: Add an app handler for unit tests described in meson""
This reverts commit 5656e2a1b2.
2017-01-03 13:01:31 -03:00
Edward Hervey 5656e2a1b2 Revert "validate:launcher: Add an app handler for unit tests described in meson"
This reverts commit a4aa5c60bb.

Does not work outside of gst-build (i.e. validate won't work when used either
installed, in a prefix, or in gst-uninstalled).
2017-01-02 17:21:14 +01:00
Thibault Saunier 1bc6cb3615 validate:launcher: Take into account test duration when filtering them
Otherwise running -t 'some.*test' will run long tests (longer than hard
timeout) which is not what the user expect.
2016-12-23 14:59:03 -03:00
Thibault Saunier d013e26cbc validate: Whitlist blacklisted test only when explicitely
Otherwise running -t '.*reverse.*' will also run blacklisted
tests which is probably not what use wants.
2016-12-22 10:29:18 -03:00
Thibault Saunier 5fc11cf389 launcher: Make --update-media-info check if generating frames info or not 2016-12-22 10:29:18 -03:00
Thibault Saunier a4aa5c60bb validate:launcher: Add an app handler for unit tests described in meson
This way we can run all tests with the launcher which brings in many
features.

And add a testsuite for GStreamer unit tests.
2016-12-22 10:29:18 -03:00
Thibault Saunier 1f9c67e6ff validate:launcher: Allow specifying a timeout factor 2016-12-22 10:08:27 -03:00
Thibault Saunier 085ebf03ba validate:launcher: Print the iteration number when running forever 2016-12-22 10:08:26 -03:00
Thibault Saunier d5de0b702c validate:launcher: Do not list tests on unneeded testers 2016-12-22 10:08:23 -03:00
Thibault Saunier 6504b9152c validate:launcher: Allow specifying several testsuite dirs 2016-12-22 10:08:21 -03:00
Thibault Saunier 26692e749c validate:launcher: Stop running test subprocesses in a shell
And instead properly use a list of argument for the subprocesses.
2016-12-22 10:07:58 -03:00
Thibault Saunier a842f9d511 validate:launcher: Handle missing media info file 2016-11-30 14:07:04 -03:00
Thibault Saunier b0784ffcb3 validate:launcher: Make sure that the IPC server is shutdown before closing the socket 2016-11-30 07:40:05 -03:00
Thibault Saunier 2959af0761 validate:launcher: Make sure to check string when verifying expected failures
The value can potentially be None and we should handle that
2016-11-29 14:47:35 -03:00
Thibault Saunier acfdd04e01 validate:launcher: Allow specifying timeout as a known issue
And minor cleanups
2016-11-28 19:37:08 -03:00
Thibault Saunier 03453961e8 validate:launcher: Properly report stack trace as such in the xunit file 2016-11-28 18:40:00 -03:00
Thibault Saunier aec8693258 validate:launcher: Remove now useless validatelog
We are now doing IPC to communicate with the launcher
so let it simply go to stdout.
2016-11-28 18:40:00 -03:00
Thibault Saunier e132c11a95 validate:launcher: Handle checking bug status for expected failures 2016-11-28 18:40:00 -03:00
Thibault Saunier 2857eaf2ad validate: launcher: Allow checking if bugs linked to blacklist is fixed 2016-11-16 17:39:45 -03:00
Thibault Saunier 615fab620a validate:launcher: Consider wanted tests as whitelisted 2016-11-16 12:02:32 -03:00
Thibault Saunier f5828b2b5c validate:launcher: Allow running the testsuite N number of times 2016-11-14 13:32:07 -03:00
Thibault Saunier 5452672e89 validate:launcher: Try to generate a backtrace on segfaults 2016-11-09 10:19:45 -03:00
Thibault Saunier 1e51aeb942 validate:launcher: Port to Python3
And sync logging.py with Pitivi version
2016-11-09 10:13:42 -03:00
Edward Hervey a29f4c390c baseclasses: Also check for minimum number of video-tracks
Some scenarios might only be for video files and are meaningless for
audio-only files
2016-10-26 17:34:49 +02:00
Thibault Saunier 41be7bb2a9 validate:launcher: Minor fix in returncode check 2016-09-26 13:36:48 -03:00
Thibault Saunier e376da4c17 validate:launcher: Do not use unset sent_eos variable
And rename class member to sent_eos_time as it is more accurate
2016-09-12 14:21:30 -03:00
Thibault Saunier 9c9fe14293 meson:validate:test: Properly set paths to run launcher based tests
Adding a --validate-tools-path option to the launcher, allowing
to pass it from meson.
2016-09-09 12:24:54 -03:00
Thibault Saunier 3ece6a065e validate: launcher: Always clean all tests at the end
Making sure that if an exception of anything happens we will
properly clean all the tests, or at least try to.
2016-09-08 13:10:29 -03:00
Thibault Saunier 03548cd63d validate:launcher: Allow specifying expected tests errors
In the future instead of blacklisting tests we should define
what error is expected, and this way when the bug is closed,
we will notice, also, it will allow us to check GstValidate
error reporting itself.
2016-09-08 13:10:29 -03:00
Thibault Saunier 2fff14e469 validate: Pass information about GstValidate execution over a socket
Instead of trying to parsing stdout, generate json messages and
send them over a socket so that gst-validate-launcher can properly
have informations about gst-validate subprocess execution.
2016-09-08 13:10:29 -03:00
Thibault Saunier 23203ca0bd validate:launcher: Use a xunit reporter only when explicitely specified 2016-09-08 10:39:56 -03:00
Thibault Saunier 8973fa04eb validate:launcher: Report error and exit when a testsuite could not load
Otherwise the user might end up seeing a lot of meaningless logs about
'removed' tests.
2016-09-08 10:39:56 -03:00