Commit graph

126 commits

Author SHA1 Message Date
Thibault Saunier ca1f8a03e0 validate:launcher: Make the IPC server socket blocking
0.0 does not mean blocking in python3 and makes everything failling
on windows.
2017-01-03 16:26:20 -03:00
Thibault Saunier f24e22446b validate: Properly kill subprocesses on windows 2017-01-03 16:24:19 -03:00
Thibault Saunier f6d0636466 validate:launcher: Fix running on windows 2017-01-03 15:34:39 -03:00
Thibault Saunier 2863986ea3 validate: Respect active testers when listing tests 2017-01-03 14:52:38 -03:00
Thibault Saunier 0ee2147137 Revert "Revert "validate:launcher: Add an app handler for unit tests described in meson""
This reverts commit 5656e2a1b2.
2017-01-03 13:01:31 -03:00
Edward Hervey 5656e2a1b2 Revert "validate:launcher: Add an app handler for unit tests described in meson"
This reverts commit a4aa5c60bb.

Does not work outside of gst-build (i.e. validate won't work when used either
installed, in a prefix, or in gst-uninstalled).
2017-01-02 17:21:14 +01:00
Thibault Saunier 1bc6cb3615 validate:launcher: Take into account test duration when filtering them
Otherwise running -t 'some.*test' will run long tests (longer than hard
timeout) which is not what the user expect.
2016-12-23 14:59:03 -03:00
Thibault Saunier d013e26cbc validate: Whitlist blacklisted test only when explicitely
Otherwise running -t '.*reverse.*' will also run blacklisted
tests which is probably not what use wants.
2016-12-22 10:29:18 -03:00
Thibault Saunier 5fc11cf389 launcher: Make --update-media-info check if generating frames info or not 2016-12-22 10:29:18 -03:00
Thibault Saunier a4aa5c60bb validate:launcher: Add an app handler for unit tests described in meson
This way we can run all tests with the launcher which brings in many
features.

And add a testsuite for GStreamer unit tests.
2016-12-22 10:29:18 -03:00
Thibault Saunier 1f9c67e6ff validate:launcher: Allow specifying a timeout factor 2016-12-22 10:08:27 -03:00
Thibault Saunier 085ebf03ba validate:launcher: Print the iteration number when running forever 2016-12-22 10:08:26 -03:00
Thibault Saunier d5de0b702c validate:launcher: Do not list tests on unneeded testers 2016-12-22 10:08:23 -03:00
Thibault Saunier 6504b9152c validate:launcher: Allow specifying several testsuite dirs 2016-12-22 10:08:21 -03:00
Thibault Saunier 26692e749c validate:launcher: Stop running test subprocesses in a shell
And instead properly use a list of argument for the subprocesses.
2016-12-22 10:07:58 -03:00
Thibault Saunier a842f9d511 validate:launcher: Handle missing media info file 2016-11-30 14:07:04 -03:00
Thibault Saunier b0784ffcb3 validate:launcher: Make sure that the IPC server is shutdown before closing the socket 2016-11-30 07:40:05 -03:00
Thibault Saunier 2959af0761 validate:launcher: Make sure to check string when verifying expected failures
The value can potentially be None and we should handle that
2016-11-29 14:47:35 -03:00
Thibault Saunier acfdd04e01 validate:launcher: Allow specifying timeout as a known issue
And minor cleanups
2016-11-28 19:37:08 -03:00
Thibault Saunier 03453961e8 validate:launcher: Properly report stack trace as such in the xunit file 2016-11-28 18:40:00 -03:00
Thibault Saunier aec8693258 validate:launcher: Remove now useless validatelog
We are now doing IPC to communicate with the launcher
so let it simply go to stdout.
2016-11-28 18:40:00 -03:00
Thibault Saunier e132c11a95 validate:launcher: Handle checking bug status for expected failures 2016-11-28 18:40:00 -03:00
Thibault Saunier 2857eaf2ad validate: launcher: Allow checking if bugs linked to blacklist is fixed 2016-11-16 17:39:45 -03:00
Thibault Saunier 615fab620a validate:launcher: Consider wanted tests as whitelisted 2016-11-16 12:02:32 -03:00
Thibault Saunier f5828b2b5c validate:launcher: Allow running the testsuite N number of times 2016-11-14 13:32:07 -03:00
Thibault Saunier 5452672e89 validate:launcher: Try to generate a backtrace on segfaults 2016-11-09 10:19:45 -03:00
Thibault Saunier 1e51aeb942 validate:launcher: Port to Python3
And sync logging.py with Pitivi version
2016-11-09 10:13:42 -03:00
Edward Hervey a29f4c390c baseclasses: Also check for minimum number of video-tracks
Some scenarios might only be for video files and are meaningless for
audio-only files
2016-10-26 17:34:49 +02:00
Thibault Saunier 41be7bb2a9 validate:launcher: Minor fix in returncode check 2016-09-26 13:36:48 -03:00
Thibault Saunier e376da4c17 validate:launcher: Do not use unset sent_eos variable
And rename class member to sent_eos_time as it is more accurate
2016-09-12 14:21:30 -03:00
Thibault Saunier 9c9fe14293 meson:validate:test: Properly set paths to run launcher based tests
Adding a --validate-tools-path option to the launcher, allowing
to pass it from meson.
2016-09-09 12:24:54 -03:00
Thibault Saunier 3ece6a065e validate: launcher: Always clean all tests at the end
Making sure that if an exception of anything happens we will
properly clean all the tests, or at least try to.
2016-09-08 13:10:29 -03:00
Thibault Saunier 03548cd63d validate:launcher: Allow specifying expected tests errors
In the future instead of blacklisting tests we should define
what error is expected, and this way when the bug is closed,
we will notice, also, it will allow us to check GstValidate
error reporting itself.
2016-09-08 13:10:29 -03:00
Thibault Saunier 2fff14e469 validate: Pass information about GstValidate execution over a socket
Instead of trying to parsing stdout, generate json messages and
send them over a socket so that gst-validate-launcher can properly
have informations about gst-validate subprocess execution.
2016-09-08 13:10:29 -03:00
Thibault Saunier 23203ca0bd validate:launcher: Use a xunit reporter only when explicitely specified 2016-09-08 10:39:56 -03:00
Thibault Saunier 8973fa04eb validate:launcher: Report error and exit when a testsuite could not load
Otherwise the user might end up seeing a lot of meaningless logs about
'removed' tests.
2016-09-08 10:39:56 -03:00
Mathieu Duponchelle ec1d8d7d8c launcher: Add --dump-on-failure switch
When the test fails, it can be useful to have the log files dumped
to stdout.

https://bugzilla.gnome.org/show_bug.cgi?id=741092
2016-09-01 03:39:18 +02:00
Thibault Saunier 5a745829dc validate: Add jpeg as known format 2016-07-29 15:54:02 -04:00
Thibault Saunier bcc7a1fcca validate:launcher: Add h265, opus and vp9 as known formats 2016-07-15 08:56:02 -04:00
Guillaume Desmottes 2ddbcb666f validate: generate valgrind suppression traces
Makes fixing easier as then we can just re-use the generated trace.

Reviewed-by: Thibault Saunier <tsaunier@gnome.org>
Differential Revision: https://phabricator.freedesktop.org/D953
2016-05-24 10:28:26 -04:00
Thibault Saunier 83c652ba76 validate: launcher: Make sure to properly setup all testsuites
When a first testsuite will set paths, it does not mean that we should
just register following testsuite test manager default tests.

So we need to make a difference between the media paths the user passed
with --media-path and the ones defined by the testsuite.
2016-03-23 20:10:51 +01:00
Thibault Saunier 9190bcf62a validate:launcher: Handle testslist files even running several testsuites
Only if those testsuites do not use the same Tester as we
currently can't know to what testsuite a test belongs.
2016-03-23 20:10:01 +01:00
Thibault Saunier 621bda9b62 validate: Use GstValidate logging system to print buffering avancement
So it can be used in the launcher.
And make sure to properly parse the info in the launcher.
2016-03-21 14:34:27 +01:00
Thibault Saunier 9708c7eb19 validate:launcher: Strip os.pathsep from extra env variables
We might be working with something that is not a path
2016-03-01 15:10:15 +01:00
Thibault Saunier 39a0be8e0e validate:launcher: Add a way to fail if test have been removed/added 2016-02-23 11:47:15 +01:00
Vineeth T M e2d2a6865f validate:launcher: Add proper check for is_seekable
The seekable variable in media_info file is of type string. When checking if the file
is seekable using is_seekable, it just returns the string, resulting in it always being true.
It should actually be comparing the string and returning true or false based on comparison

https://bugzilla.gnome.org/show_bug.cgi?id=755854
2016-02-17 22:33:16 +01:00
Thibault Saunier 8899fd8dd1 validate: Implement support to run tests inside gdb
Making debugging races leading to crashes easier to debug
2015-11-10 18:17:54 +01:00
Thibault Saunier aef41ba72a validate: Add support for prores 2015-10-14 12:01:15 +01:00
Vineeth TM aa2c93c3d4 validate:launcher: throw valgrind error only for definite loss
errors-for-leak-kinds should be set to definite, because almost every test case
, will have possibly lost memory, which may or may not be a leak.
And throwing error for all these cases doesn't seem to be correct.

https://bugzilla.gnome.org/show_bug.cgi?id=752754
2015-10-02 17:46:51 +02:00
Vineeth T M e7b65fe5df validate: launcher: Fix media_check class name and add double quotes for valgrind logs
When creating the class names for media check, uri is being used,
instead of the path. Hence converting the uri using uri2path and creating
class name.
Add double quotes for valgrind logs, to support special characters like space

https://bugzilla.gnome.org/show_bug.cgi?id=752808
2015-10-02 17:43:50 +02:00