Commit graph

202 commits

Author SHA1 Message Date
Thibault Saunier
40a08e6080 validate: launcher: Add some missing env variables in command to launch test 2017-07-11 10:11:49 -04:00
Thibault Saunier
c540601ed0 validate:launcher: Do not check ModuleNotFound exception
It is a subclass of ImportError and is avalaible only since 3.6
https://ci.gstreamer.net/job/pitivi-flatpak/626/console
2017-06-28 15:54:13 -04:00
Thibault Saunier
8ea68e9e06 validate:launcher: Speed up xml parsing using lxml if avalaible 2017-06-27 16:50:35 -04:00
Thibault Saunier
2f35acf558 validate:launcher: Enhance command printing when using a server
So it can be copy pasted and work
2017-06-20 15:46:15 -04:00
Thibault Saunier
335c14bf5b validate:launcher: Fix launching testsuite with relative paths 2017-06-16 17:36:34 -04:00
Thibault Saunier
9a45cd41a9 validate:launcher: Handle test that can't be run in parralel 2017-06-07 15:23:26 -04:00
Thibault Saunier
0e163ffae8 validate:launcher: Handle optional tests
For example RTSP tests might not be avalaible if gst-rtsp-server-example-uri is not avalaible
2017-06-07 11:17:53 -04:00
Thibault Saunier
0beb8647ce validate:launcher: Print logfiles when printing test result
It was often annoying to check wrong logs because the result are
not printed near the reference to logs.
2017-06-07 11:17:53 -04:00
Thibault Saunier
37de89e6db validate:launcher: Do not run scenarios that need prerolling on RTSP stream 2017-06-07 11:17:53 -04:00
Thibault Saunier
3c62c315a9 validate: Implement RTSP support 2017-06-07 11:17:53 -04:00
Thibault Saunier
02d716d6d8 validate:launcher: Handle not redirecting valgrind output 2017-06-01 16:57:50 -04:00
Edward Hervey
400d1f9326 validate: Implement Scenario.__repr__
Allows better debugging when looking at logs
2017-05-25 15:50:23 +02:00
Edward Hervey
9188968f5c validate: Add live-related features to scenarios and medias
Note: The notion of "live" here is in the *content* sense and not in the
GStreamer sense.
Ex:
  * A rtsp stream is always "live" in the GStreamer sense but might not always
    provide live content.
  * HLS/DASH streams are not "live" in the GStreamer sense but might
    provide "live" content.

Some scenarios might:
* require live content
* not be compatible with live content

This patch adds two new properties for scenarios:
* live_content_required (default False) for scenarios that can only work with
  live content.
* live_content_compatible (default False) for scenarios that can work with
  both live and non-live content.
This patch adds support for reading a "live" property from stream_info
2017-05-25 14:04:00 +02:00
Thibault Saunier
40a31b05b6 launcher: Allow using the base launcher as a test manager
No reason to force people to subclass it in simple cases.
2017-04-30 14:36:16 -03:00
Edward Hervey
325c1b37dd baseclasses: Add method to set a specific list of scenarios
https://bugzilla.gnome.org/show_bug.cgi?id=781314
2017-04-14 16:55:06 +02:00
Thibault Saunier
49271bc721 validate:launcher: Add information on media info files parsing failures 2017-02-27 12:12:38 -03:00
Thibault Saunier
fbde653a0c validate: launcher: Fix the way we retrieve command name
We are now using a list of args for subprocess so just using it is simple now
2017-02-27 12:12:38 -03:00
Thibault Saunier
9caee62bc6 validate: Fix call to decode() on a string 2017-02-27 12:12:38 -03:00
Thibault Saunier
f2b42cbaf8 validate:launcher: Various fixes to make the Test class directly usable
And the launcher installed with meson usable
2017-02-21 13:40:28 -03:00
Thibault Saunier
54a1fc8b30 validate:launcher: Take ValidateAction execution as a marker of test update 2017-02-17 15:29:41 -03:00
Thibault Saunier
40016b9956 validate:scenario: Report action done with the execution duration 2017-02-17 15:29:41 -03:00
Thibault Saunier
7773ca7765 validate:launcher: Add a way to specify a set of tests to run under the leak tracer
https://bugzilla.gnome.org/show_bug.cgi?id=767856
2017-02-10 15:56:11 -03:00
Thibault Saunier
addd76fc85 validate: Mark tests as SKIPPED when installation is missing a GStreamer plugin 2017-02-07 13:12:09 -03:00
Thibault Saunier
823ce95d70 validate:launcher: Add an option for user to modify timeout values
Allowing to expand the test timeout when running on slow platforms
2017-02-06 12:16:41 -03:00
Thibault Saunier
5d1d8e6971 validate:launcher: Verify that Gst supression file could be found 2017-02-03 11:02:49 -03:00
Thibault Saunier
de2cbda854 validate: Make sure to update children environment from current env
And minor fix in function call
2017-01-06 12:12:21 -03:00
Thibault Saunier
ca1f8a03e0 validate:launcher: Make the IPC server socket blocking
0.0 does not mean blocking in python3 and makes everything failling
on windows.
2017-01-03 16:26:20 -03:00
Thibault Saunier
f24e22446b validate: Properly kill subprocesses on windows 2017-01-03 16:24:19 -03:00
Thibault Saunier
f6d0636466 validate:launcher: Fix running on windows 2017-01-03 15:34:39 -03:00
Thibault Saunier
2863986ea3 validate: Respect active testers when listing tests 2017-01-03 14:52:38 -03:00
Thibault Saunier
0ee2147137 Revert "Revert "validate:launcher: Add an app handler for unit tests described in meson""
This reverts commit 5656e2a1b2.
2017-01-03 13:01:31 -03:00
Edward Hervey
5656e2a1b2 Revert "validate:launcher: Add an app handler for unit tests described in meson"
This reverts commit a4aa5c60bb.

Does not work outside of gst-build (i.e. validate won't work when used either
installed, in a prefix, or in gst-uninstalled).
2017-01-02 17:21:14 +01:00
Thibault Saunier
1bc6cb3615 validate:launcher: Take into account test duration when filtering them
Otherwise running -t 'some.*test' will run long tests (longer than hard
timeout) which is not what the user expect.
2016-12-23 14:59:03 -03:00
Thibault Saunier
d013e26cbc validate: Whitlist blacklisted test only when explicitely
Otherwise running -t '.*reverse.*' will also run blacklisted
tests which is probably not what use wants.
2016-12-22 10:29:18 -03:00
Thibault Saunier
5fc11cf389 launcher: Make --update-media-info check if generating frames info or not 2016-12-22 10:29:18 -03:00
Thibault Saunier
a4aa5c60bb validate:launcher: Add an app handler for unit tests described in meson
This way we can run all tests with the launcher which brings in many
features.

And add a testsuite for GStreamer unit tests.
2016-12-22 10:29:18 -03:00
Thibault Saunier
1f9c67e6ff validate:launcher: Allow specifying a timeout factor 2016-12-22 10:08:27 -03:00
Thibault Saunier
085ebf03ba validate:launcher: Print the iteration number when running forever 2016-12-22 10:08:26 -03:00
Thibault Saunier
d5de0b702c validate:launcher: Do not list tests on unneeded testers 2016-12-22 10:08:23 -03:00
Thibault Saunier
6504b9152c validate:launcher: Allow specifying several testsuite dirs 2016-12-22 10:08:21 -03:00
Thibault Saunier
26692e749c validate:launcher: Stop running test subprocesses in a shell
And instead properly use a list of argument for the subprocesses.
2016-12-22 10:07:58 -03:00
Thibault Saunier
a842f9d511 validate:launcher: Handle missing media info file 2016-11-30 14:07:04 -03:00
Thibault Saunier
b0784ffcb3 validate:launcher: Make sure that the IPC server is shutdown before closing the socket 2016-11-30 07:40:05 -03:00
Thibault Saunier
2959af0761 validate:launcher: Make sure to check string when verifying expected failures
The value can potentially be None and we should handle that
2016-11-29 14:47:35 -03:00
Thibault Saunier
acfdd04e01 validate:launcher: Allow specifying timeout as a known issue
And minor cleanups
2016-11-28 19:37:08 -03:00
Thibault Saunier
03453961e8 validate:launcher: Properly report stack trace as such in the xunit file 2016-11-28 18:40:00 -03:00
Thibault Saunier
aec8693258 validate:launcher: Remove now useless validatelog
We are now doing IPC to communicate with the launcher
so let it simply go to stdout.
2016-11-28 18:40:00 -03:00
Thibault Saunier
e132c11a95 validate:launcher: Handle checking bug status for expected failures 2016-11-28 18:40:00 -03:00
Thibault Saunier
2857eaf2ad validate: launcher: Allow checking if bugs linked to blacklist is fixed 2016-11-16 17:39:45 -03:00
Thibault Saunier
615fab620a validate:launcher: Consider wanted tests as whitelisted 2016-11-16 12:02:32 -03:00
Thibault Saunier
f5828b2b5c validate:launcher: Allow running the testsuite N number of times 2016-11-14 13:32:07 -03:00
Thibault Saunier
5452672e89 validate:launcher: Try to generate a backtrace on segfaults 2016-11-09 10:19:45 -03:00
Thibault Saunier
1e51aeb942 validate:launcher: Port to Python3
And sync logging.py with Pitivi version
2016-11-09 10:13:42 -03:00
Edward Hervey
a29f4c390c baseclasses: Also check for minimum number of video-tracks
Some scenarios might only be for video files and are meaningless for
audio-only files
2016-10-26 17:34:49 +02:00
Thibault Saunier
41be7bb2a9 validate:launcher: Minor fix in returncode check 2016-09-26 13:36:48 -03:00
Thibault Saunier
e376da4c17 validate:launcher: Do not use unset sent_eos variable
And rename class member to sent_eos_time as it is more accurate
2016-09-12 14:21:30 -03:00
Thibault Saunier
9c9fe14293 meson:validate:test: Properly set paths to run launcher based tests
Adding a --validate-tools-path option to the launcher, allowing
to pass it from meson.
2016-09-09 12:24:54 -03:00
Thibault Saunier
3ece6a065e validate: launcher: Always clean all tests at the end
Making sure that if an exception of anything happens we will
properly clean all the tests, or at least try to.
2016-09-08 13:10:29 -03:00
Thibault Saunier
03548cd63d validate:launcher: Allow specifying expected tests errors
In the future instead of blacklisting tests we should define
what error is expected, and this way when the bug is closed,
we will notice, also, it will allow us to check GstValidate
error reporting itself.
2016-09-08 13:10:29 -03:00
Thibault Saunier
2fff14e469 validate: Pass information about GstValidate execution over a socket
Instead of trying to parsing stdout, generate json messages and
send them over a socket so that gst-validate-launcher can properly
have informations about gst-validate subprocess execution.
2016-09-08 13:10:29 -03:00
Thibault Saunier
23203ca0bd validate:launcher: Use a xunit reporter only when explicitely specified 2016-09-08 10:39:56 -03:00
Thibault Saunier
8973fa04eb validate:launcher: Report error and exit when a testsuite could not load
Otherwise the user might end up seeing a lot of meaningless logs about
'removed' tests.
2016-09-08 10:39:56 -03:00
Mathieu Duponchelle
ec1d8d7d8c launcher: Add --dump-on-failure switch
When the test fails, it can be useful to have the log files dumped
to stdout.

https://bugzilla.gnome.org/show_bug.cgi?id=741092
2016-09-01 03:39:18 +02:00
Thibault Saunier
5a745829dc validate: Add jpeg as known format 2016-07-29 15:54:02 -04:00
Thibault Saunier
bcc7a1fcca validate:launcher: Add h265, opus and vp9 as known formats 2016-07-15 08:56:02 -04:00
Guillaume Desmottes
2ddbcb666f validate: generate valgrind suppression traces
Makes fixing easier as then we can just re-use the generated trace.

Reviewed-by: Thibault Saunier <tsaunier@gnome.org>
Differential Revision: https://phabricator.freedesktop.org/D953
2016-05-24 10:28:26 -04:00
Thibault Saunier
83c652ba76 validate: launcher: Make sure to properly setup all testsuites
When a first testsuite will set paths, it does not mean that we should
just register following testsuite test manager default tests.

So we need to make a difference between the media paths the user passed
with --media-path and the ones defined by the testsuite.
2016-03-23 20:10:51 +01:00
Thibault Saunier
9190bcf62a validate:launcher: Handle testslist files even running several testsuites
Only if those testsuites do not use the same Tester as we
currently can't know to what testsuite a test belongs.
2016-03-23 20:10:01 +01:00
Thibault Saunier
621bda9b62 validate: Use GstValidate logging system to print buffering avancement
So it can be used in the launcher.
And make sure to properly parse the info in the launcher.
2016-03-21 14:34:27 +01:00
Thibault Saunier
9708c7eb19 validate:launcher: Strip os.pathsep from extra env variables
We might be working with something that is not a path
2016-03-01 15:10:15 +01:00
Thibault Saunier
39a0be8e0e validate:launcher: Add a way to fail if test have been removed/added 2016-02-23 11:47:15 +01:00
Vineeth T M
e2d2a6865f validate:launcher: Add proper check for is_seekable
The seekable variable in media_info file is of type string. When checking if the file
is seekable using is_seekable, it just returns the string, resulting in it always being true.
It should actually be comparing the string and returning true or false based on comparison

https://bugzilla.gnome.org/show_bug.cgi?id=755854
2016-02-17 22:33:16 +01:00
Thibault Saunier
8899fd8dd1 validate: Implement support to run tests inside gdb
Making debugging races leading to crashes easier to debug
2015-11-10 18:17:54 +01:00
Thibault Saunier
aef41ba72a validate: Add support for prores 2015-10-14 12:01:15 +01:00
Vineeth TM
aa2c93c3d4 validate:launcher: throw valgrind error only for definite loss
errors-for-leak-kinds should be set to definite, because almost every test case
, will have possibly lost memory, which may or may not be a leak.
And throwing error for all these cases doesn't seem to be correct.

https://bugzilla.gnome.org/show_bug.cgi?id=752754
2015-10-02 17:46:51 +02:00
Vineeth T M
e7b65fe5df validate: launcher: Fix media_check class name and add double quotes for valgrind logs
When creating the class names for media check, uri is being used,
instead of the path. Hence converting the uri using uri2path and creating
class name.
Add double quotes for valgrind logs, to support special characters like space

https://bugzilla.gnome.org/show_bug.cgi?id=752808
2015-10-02 17:43:50 +02:00
Thibault Saunier
91b3001f77 validate:launcher: Treat min-media-duration as a float 2015-09-30 18:13:28 +02:00
Thibault Saunier
bb2532aa54 launcher: Handle override files for media files
And make sure to create a new dict for extra_env_vars when instanciating
GstValidateTest
2015-08-15 18:30:53 +02:00
Thibault Saunier
367e6cc4f4 launcher: Make sure MediaDescriptor is set in the GstValidateTest class itself
It is used there but was set in each and every subclasses
2015-08-15 18:30:53 +02:00
Vineeth TM
94c684d401 validate:launcher: escape the characters to remove bad range in regex
When media file name consists of some special characters of the format
[b-a].mp3, then it fails with 'bad character range' error and exits.
call re.escape to escape the characters before using it in findall

https://bugzilla.gnome.org/show_bug.cgi?id=752650
2015-08-05 16:56:18 -04:00
Vineeth TM
b37c550b2b validate:launcher: skip setting up test suite for --medias-paths
in validate.py, some mixer test generators are being added by default.
When passing --media-paths, i would not want to test these.
So instead of setting up the validate test suite, just call tester.register_defaults().

https://bugzilla.gnome.org/show_bug.cgi?id=752518
2015-08-05 16:12:47 -04:00
Thibault Saunier
3e978a46b4 validate:launcher: Avoid printing twice env variables
When printing test command.
2015-06-12 11:31:13 +02:00
Thibault Saunier
d89b4884c6 validate:launcher: extra_env_variables is a dictionnary 2015-05-14 17:44:17 +02:00
Thibault Saunier
dc2f4f11c6 validate:launcher: Always set the protocol when creating a GstValidateMediaDescriptor
Summary: Depends on D174

Reviewers: Mathieu_Du, gdesmott

Differential Revision: http://phabricator.freedesktop.org/D175
2015-05-14 15:56:57 +02:00
Thibault Saunier
4ada36dbfc validate:launcher: Allow passing extra env variables to the tests
Summary: Depends on D173

Reviewers: Mathieu_Du, gdesmott

Differential Revision: http://phabricator.freedesktop.org/D174
2015-05-14 15:56:57 +02:00
Guillaume Desmottes
4927c65710 validate: disable QOS features when running with valgrind
Reviewers: thiblahute

Differential Revision: http://phabricator.freedesktop.org/D156
2015-05-11 21:55:24 +02:00
Guillaume Desmottes
f6681ff003 validate: rename get_valgrind_suppression_file()
Summary:
This function is actually not specific to valgrind so we can make it more
generic.

Reviewers: thiblahute

Differential Revision: http://phabricator.freedesktop.org/D162
2015-05-11 21:55:24 +02:00
Guillaume Desmottes
b947e1c2ef use the setup_sink_props_max_lateness config scenario with valgrind
Reviewers: thiblahute

Differential Revision: http://phabricator.freedesktop.org/D141
2015-05-04 14:13:46 +02:00
Guillaume Desmottes
133c415ade validate: display the URL of ignored Valgrind bugs
Summary:
We don't want to forget about those so best to remind it when starting tests
as we do with blacklisted tests.

Reviewers: thiblahute

Differential Revision: http://phabricator.freedesktop.org/D131
2015-05-01 15:15:26 +02:00
Guillaume Desmottes
c5c39d88b1 validate: move look_for_file_in_source_dir and get_valgrind_suppression_file to utils
Reviewers: thiblahute

Differential Revision: http://phabricator.freedesktop.org/D130
2015-05-01 15:15:26 +02:00
Thibault Saunier
753ba408dd validate:launcher: Concider unset MediaDescriptor duration has 'infinite' 2015-04-28 16:44:42 +02:00
Guillaume Desmottes
044ab241ee validate: use -debug versions of bins when running from source
Summary:
Those versions are using rpath instead of libtool's wrappers and so will be
faster to start and won't confuse valgrind.

Reviewers: thiblahute

Differential Revision: http://phabricator.freedesktop.org/D116
2015-04-21 11:19:17 +02:00
Guillaume Desmottes
8c2684c9a7 validate: add gstvalidate.supp valgrind suppression file
Differential Revision: http://phabricator.freedesktop.org/D115
2015-04-20 15:29:28 +02:00
Thibault Saunier
adbe811175 validate:launcher: Pass GST_VALIDATE_SCENARIO to the subprocess env only
Summary:
And make sure to remove it from the env if the user has it in its main
environment.

Without that commit we ended up passing scenarios from previous tests
to the following ones where None were specified.

Reviewers: Mathieu_Du

Differential Revision: http://phabricator.freedesktop.org/D98
2015-04-16 12:41:25 +02:00
Guillaume Desmottes
8a6494ddf8 validate: set GST_GL_XINITTHREADS
This ensure that XInitThreads is called and so gl contexts are properly
initialized.

https://bugzilla.gnome.org/show_bug.cgi?id=747840

Signed-off-by: Guillaume Desmottes <guillaume.desmottes@collabora.co.uk>
2015-04-14 12:56:32 +02:00
Thibault Saunier
2cf082c4be validate:launcher: Set more env variable in the launcher command desc
Summary:
Adding if present:
  * LD_PRELOAD
  * DISPLAY
  * GST_VALIDATE_CONFIG
  * GST_VALIDATE_OVERRIDE

+  enhance the add_env_variable method to more easily set envvar from
current value

Reviewers: Mathieu_Du

Differential Revision: http://phabricator.freedesktop.org/D78
2015-04-10 16:53:41 +02:00
Thibault Saunier
bbd04eef4c validate:launcher: Use GST_VALIDATE_SCENARIO envvar to set scenarios
Summary:
Instead of concidering all apps will have a --set-scenario argument
which is not going to be the case as soon as we run the tests through
LD_PRELOAD

Reviewers: Mathieu_Du

Differential Revision: http://phabricator.freedesktop.org/D70
2015-04-10 16:53:40 +02:00
Thibault Saunier
2fe03f1993 validate:launcher: Move get_current_position from GstValidatePipelineTest to GstValidateTest
This is where it belongs

Reviewers: Mathieu_Du

Differential Revision: http://phabricator.freedesktop.org/D69
2015-04-10 16:53:40 +02:00
Young Han Lee
4725f4ec09 validate:launcher: Show timeout seconds for timeout result message
Current timeout message doesn't show how many seconds a test took and
it is timeouted by normal timeout or hard timeout.

This patch changes the message like following.

1. normal timeout
old : validate.http.playback.reverse_playback.raw_video_mov: Timeout (Application timed out)
new : validate.http.playback.reverse_playback.raw_video_mov: Timeout (Application timed out: 120 secs)

2. hard timeout
old : validate.http.playback.reverse_playback.raw_video_mov: Timeout (Application timed out)
new : validate.http.playback.reverse_playback.raw_video_mov: Timeout (Hard timeout reached: 600 secs)

https://bugzilla.gnome.org/show_bug.cgi?id=746957
2015-03-30 13:32:57 +02:00
Young Han Lee
4c7ec6b8b1 validate:launcher: Fix wrong test number with -j option
When '-j n' option is given, first n tests print test number 0.
This is caused by test_num part of 919db98605.

https://bugzilla.gnome.org/show_bug.cgi?id=747006
2015-03-30 13:02:39 +02:00