Commit graph

215 commits

Author SHA1 Message Date
Thibault Saunier
8973fa04eb validate:launcher: Report error and exit when a testsuite could not load
Otherwise the user might end up seeing a lot of meaningless logs about
'removed' tests.
2016-09-08 10:39:56 -03:00
Thibault Saunier
f7f600e730 validate: launcher: Fix the condition to check if we need an http server
We could be checking if a string was in None

And use gs_string_assign when assigning the first string
without using printf like format.
2016-09-08 10:39:56 -03:00
Mathieu Duponchelle
ec1d8d7d8c launcher: Add --dump-on-failure switch
When the test fails, it can be useful to have the log files dumped
to stdout.

https://bugzilla.gnome.org/show_bug.cgi?id=741092
2016-09-01 03:39:18 +02:00
Thibault Saunier
da327d8625 validate: Add support for Meson as alternative/parallel build system
https://github.com/mesonbuild/meson
2016-08-25 15:12:31 -03:00
Edward Hervey
fcff932e09 validate: Un-blacklist tests that are fixed
the bug reports to which they report have been closed and I can't make
them fail locally.
2016-08-13 16:56:18 +02:00
Edward Hervey
7d99e780a7 validate: Blacklist more ogg files
https://bugzilla.gnome.org/show_bug.cgi?id=769545
2016-08-13 15:39:18 +02:00
Edward Hervey
37771192a6 validate: Blacklist scrub_forward_seeking.op2b-mpeg2-wave_hd_mxf
See https://bugzilla.gnome.org/show_bug.cgi?id=764025
2016-08-12 12:33:30 +02:00
Thibault Saunier
5a745829dc validate: Add jpeg as known format 2016-07-29 15:54:02 -04:00
Thibault Saunier
bcc7a1fcca validate:launcher: Add h265, opus and vp9 as known formats 2016-07-15 08:56:02 -04:00
Edward Hervey
1e1475d352 validate: Fix usage for non-standard http server port
Translate the various stream_info URI from the standard port to the
specified port
Make the port option an integer option
2016-07-04 16:19:09 +02:00
Edward Hervey
b77797da4b validate: Use decodebin3/uridecodebin3 when specified
Set the USE_PLAYBIN3 environment variable to use those elements instead
of legacy ones.
2016-06-30 14:58:03 +02:00
Reynaldo H. Verdejo Pinochet
07fa1b3ca5 validate: fix some recurring typos 2016-06-09 14:31:48 -07:00
Vineeth TM
99f9f3f408 validate:launcher: Add support for relative path while providing file path
Instead of providing full absolute path while validating the file, should be
able to provide the relative path with respect to the present directory.

https://bugzilla.gnome.org/show_bug.cgi?id=753494
2016-06-04 12:51:35 +01:00
Guillaume Desmottes
2ddbcb666f validate: generate valgrind suppression traces
Makes fixing easier as then we can just re-use the generated trace.

Reviewed-by: Thibault Saunier <tsaunier@gnome.org>
Differential Revision: https://phabricator.freedesktop.org/D953
2016-05-24 10:28:26 -04:00
Guillaume Desmottes
4322138c8f validate: fix typo if Xvfb is not installed
Reviewed-by: Thibault Saunier <tsaunier@gnome.org>
Differential Revision: https://phabricator.freedesktop.org/D375
2016-05-24 10:28:23 -04:00
Thiago Santos
f4cdbd006b launcher: set gsettings-backend to prevent weird deadlocks
It seems like some sort of forking/dconf/gtype combination can
deadlock occasionally. Setting the gsettings backend to memory
makes it go away.

Same issue: https://cgit.freedesktop.org/gstreamer/gst-plugins-good/commit/tests/check/Makefile.am?id=8e2c1d1de56bddbff22170f8b17473882e0e63f9
2016-05-15 06:34:21 -03:00
Thiago Santos
d0a7d9b094 validate: add non-seekable fragment file seek tests to the blacklist
The file has no index and is not easily seekable, seeking in these
kind of files isn't implemented.
2016-05-06 21:27:53 -03:00
Thibault Saunier
83c652ba76 validate: launcher: Make sure to properly setup all testsuites
When a first testsuite will set paths, it does not mean that we should
just register following testsuite test manager default tests.

So we need to make a difference between the media paths the user passed
with --media-path and the ones defined by the testsuite.
2016-03-23 20:10:51 +01:00
Thibault Saunier
9190bcf62a validate:launcher: Handle testslist files even running several testsuites
Only if those testsuites do not use the same Tester as we
currently can't know to what testsuite a test belongs.
2016-03-23 20:10:01 +01:00
Thibault Saunier
187df093eb launcher: Avoid caching all the debug logs in memory
And just write the temporary XML file on disc
2016-03-23 12:41:49 +01:00
Thibault Saunier
23ad66c552 validate: launcher: Blacklist dash and HLS failling seeking tests
As described in https://bugzilla.gnome.org/show_bug.cgi?id=764020
2016-03-22 19:02:27 +01:00
Thibault Saunier
a566b2c4b9 validate:testsuite: Blacklist validate.hls.playback.scrub_forward_seeking.hls_bibbop as it is racy 2016-03-22 17:46:50 +01:00
Thibault Saunier
621bda9b62 validate: Use GstValidate logging system to print buffering avancement
So it can be used in the launcher.
And make sure to properly parse the info in the launcher.
2016-03-21 14:34:27 +01:00
Thibault Saunier
9b93cc884f validate:launcher: Remove dependency on wget 2016-03-21 14:09:26 +01:00
Vineeth T M
65560d8b03 Validate: Fix consider, launch spelling mistakes
https://bugzilla.gnome.org/show_bug.cgi?id=763289
2016-03-08 09:28:11 +02:00
Vineeth T M
7d9e9b3b79 validate: Fix wrong condition check when adding tests
When listing tests, checking whether uri is present or not and displaying error.
But uri does notneed to be present in case of pipeline generator. So the condition check is wrong.
This results in validateelements testsuite not working. Hence modifying the condition to
not error out on valid cases.

https://bugzilla.gnome.org/show_bug.cgi?id=762422
2016-03-05 12:56:14 +01:00
Thibault Saunier
9708c7eb19 validate:launcher: Strip os.pathsep from extra env variables
We might be working with something that is not a path
2016-03-01 15:10:15 +01:00
Thibault Saunier
39a0be8e0e validate:launcher: Add a way to fail if test have been removed/added 2016-02-23 11:47:15 +01:00
Vineeth T M
bc66079d2a validate:launcher: handle spaces in transcode output path
When there are spaces in transcoding output path, then it fails. Hence adding the
path in double quotes

https://bugzilla.gnome.org/show_bug.cgi?id=756217
2016-02-17 22:33:32 +01:00
Vineeth T M
e2d2a6865f validate:launcher: Add proper check for is_seekable
The seekable variable in media_info file is of type string. When checking if the file
is seekable using is_seekable, it just returns the string, resulting in it always being true.
It should actually be comparing the string and returning true or false based on comparison

https://bugzilla.gnome.org/show_bug.cgi?id=755854
2016-02-17 22:33:16 +01:00
Thibault Saunier
ade0fb3fed validate: launcher: Set scenario manager config before discovering testsuites 2015-11-16 16:45:13 +01:00
Vineeth T M
0f7105d317 validate: launcher: Print error when media-info files not present
When there are no media-info files present and --generate-media-info
option is not given, then it just fails without printing error.
Printing an error stating, use --generate-media-info if there are no
media info files.
When there are neither media files and media info files, print error
stating the same

https://bugzilla.gnome.org/show_bug.cgi?id=755087
2015-11-11 10:12:58 +01:00
Thibault Saunier
8899fd8dd1 validate: Implement support to run tests inside gdb
Making debugging races leading to crashes easier to debug
2015-11-10 18:17:54 +01:00
Sebastian Dröge
3bd31854a8 validate: Un-blacklist seeking HLS tests
They are reliable enough after 36b80edb72
in gst-plugins-base now.
2015-10-27 15:51:44 +02:00
Thibault Saunier
aef41ba72a validate: Add support for prores 2015-10-14 12:01:15 +01:00
Vineeth TM
aa2c93c3d4 validate:launcher: throw valgrind error only for definite loss
errors-for-leak-kinds should be set to definite, because almost every test case
, will have possibly lost memory, which may or may not be a leak.
And throwing error for all these cases doesn't seem to be correct.

https://bugzilla.gnome.org/show_bug.cgi?id=752754
2015-10-02 17:46:51 +02:00
Vineeth T M
e7b65fe5df validate: launcher: Fix media_check class name and add double quotes for valgrind logs
When creating the class names for media check, uri is being used,
instead of the path. Hence converting the uri using uri2path and creating
class name.
Add double quotes for valgrind logs, to support special characters like space

https://bugzilla.gnome.org/show_bug.cgi?id=752808
2015-10-02 17:43:50 +02:00
Vineeth T M
1b700f13b4 validate: launcher: Support relative path for folder names
Even though relative paths are supported, right now it does not
work when we give the path as 'media/' present in the current directory.
Adding support for the same.

https://bugzilla.gnome.org/show_bug.cgi?id=754100
2015-10-02 17:20:18 +02:00
Thibault Saunier
91b3001f77 validate:launcher: Treat min-media-duration as a float 2015-09-30 18:13:28 +02:00
Thibault Saunier
2b5db0c092 validate: Put the GstValidate testsuite version in the launcher config
This way testsuite implementation can have the information
2015-08-21 10:36:56 +02:00
Sebastian Dröge
5b81bb4bc9 validate/launcher: Blacklist some HLS seeking tests again 2015-08-16 17:53:28 +02:00
Thiago Santos
b77f56a234 validate/launcher: Un-blacklist validate.dash.playback.reverse_playback.*
Fixed now
2015-08-16 08:50:36 -03:00
Sebastian Dröge
f898f3ec8b validate/launcher: Un-blacklist validate.hls.playback.reverse_playback.*
It apparently succeeds now.
2015-08-16 12:26:16 +02:00
Sebastian Dröge
fea86d3556 validate/launcher: De-blacklist some HLS tests 2015-08-15 19:07:02 +02:00
Sebastian Dröge
1f8a5483d3 validate/launcher: Blacklist some DASH tests 2015-08-15 19:04:14 +02:00
Thibault Saunier
bb2532aa54 launcher: Handle override files for media files
And make sure to create a new dict for extra_env_vars when instanciating
GstValidateTest
2015-08-15 18:30:53 +02:00
Thibault Saunier
367e6cc4f4 launcher: Make sure MediaDescriptor is set in the GstValidateTest class itself
It is used there but was set in each and every subclasses
2015-08-15 18:30:53 +02:00
Sebastian Dröge
233c99fbb7 validate/launcher: Treat DASH like HLS in another place 2015-08-15 16:19:46 +02:00
Vineeth TM
e4d490fd45 validate:launcher: Fix documentation
Fix some trivial spelling mistakes in documentation
and document about --update-media-info.

https://bugzilla.gnome.org/show_bug.cgi?id=752748
2015-08-05 17:17:41 -04:00
Vineeth TM
94c684d401 validate:launcher: escape the characters to remove bad range in regex
When media file name consists of some special characters of the format
[b-a].mp3, then it fails with 'bad character range' error and exits.
call re.escape to escape the characters before using it in findall

https://bugzilla.gnome.org/show_bug.cgi?id=752650
2015-08-05 16:56:18 -04:00
Vineeth TM
ac172a9ac6 validate:launcher: improve uri generation for --medias-path
When --medias-paths option is being used, right now we have to
specify the full path, like /home/user/gst/master/media/
But when inside master directory, would like to specify only
media/ and expect it to work. Using os.path.abspath and create uri based on that.
This way we can either just pass media/ or pass the full path as parameters.

https://bugzilla.gnome.org/show_bug.cgi?id=752518
2015-08-05 16:12:47 -04:00
Vineeth TM
b37c550b2b validate:launcher: skip setting up test suite for --medias-paths
in validate.py, some mixer test generators are being added by default.
When passing --media-paths, i would not want to test these.
So instead of setting up the validate test suite, just call tester.register_defaults().

https://bugzilla.gnome.org/show_bug.cgi?id=752518
2015-08-05 16:12:47 -04:00
Thibault Saunier
1fd14635ed validate: launcher: Add a --force-sync option
Which should put the testsuite in a clean state (basically using git
reset --hard for git based testsuite for example)
2015-08-04 15:23:54 +02:00
Nicolas Dufresne
bc297a8295 validate: Enable flac reverse playback tests
This is now supported and works as expected.
2015-08-04 08:53:17 -04:00
Vineeth T M
aa6aebe41e validate:launcher: add quotes to the file path
When folder name contains space or other special characters,
it fails to recognise the same and error is thrown. Adding the path
inside  to recognise the same

https://bugzilla.gnome.org/show_bug.cgi?id=752611
2015-07-22 16:40:07 +01:00
Vineeth T M
074406bc2c validate:launcher: unquote the path to remove special characters
When getting path from url using, url2path, it is returning
special characters (%20 for space etc..), instead of returning
plain path. path.unquote replaces the same..

https://bugzilla.gnome.org/show_bug.cgi?id=752611
2015-07-22 16:39:43 +01:00
Vineeth TM
6bf6fe7dec validate:launcher: handle file path in --medias-paths
right now --medias-paths accepts only directories. Added support to
accept file path as well.

https://bugzilla.gnome.org/show_bug.cgi?id=752692
2015-07-22 16:35:49 +01:00
Vineeth T M
6c48e89aab validate:launcher: skip default media path for --media-paths option
when --media-paths is specified, then no need to check the default media.
And add Force argument to let testsuite force the inclusion of
default media directory.

https://bugzilla.gnome.org/show_bug.cgi?id=752461
2015-07-16 16:31:09 +02:00
Vineeth T M
1ef0fdd66c validate:main.py: trivial document fixes
gst-validate-launch is being used instead of gst-validate-launcher
in a couple of places.

https://bugzilla.gnome.org/show_bug.cgi?id=752455
2015-07-16 09:35:17 +02:00
Thibault Saunier
3e978a46b4 validate:launcher: Avoid printing twice env variables
When printing test command.
2015-06-12 11:31:13 +02:00
Thibault Saunier
d89b4884c6 validate:launcher: extra_env_variables is a dictionnary 2015-05-14 17:44:17 +02:00
Thibault Saunier
dc2f4f11c6 validate:launcher: Always set the protocol when creating a GstValidateMediaDescriptor
Summary: Depends on D174

Reviewers: Mathieu_Du, gdesmott

Differential Revision: http://phabricator.freedesktop.org/D175
2015-05-14 15:56:57 +02:00
Thibault Saunier
4ada36dbfc validate:launcher: Allow passing extra env variables to the tests
Summary: Depends on D173

Reviewers: Mathieu_Du, gdesmott

Differential Revision: http://phabricator.freedesktop.org/D174
2015-05-14 15:56:57 +02:00
Guillaume Desmottes
4927c65710 validate: disable QOS features when running with valgrind
Reviewers: thiblahute

Differential Revision: http://phabricator.freedesktop.org/D156
2015-05-11 21:55:24 +02:00
Guillaume Desmottes
f6681ff003 validate: rename get_valgrind_suppression_file()
Summary:
This function is actually not specific to valgrind so we can make it more
generic.

Reviewers: thiblahute

Differential Revision: http://phabricator.freedesktop.org/D162
2015-05-11 21:55:24 +02:00
Guillaume Desmottes
b947e1c2ef use the setup_sink_props_max_lateness config scenario with valgrind
Reviewers: thiblahute

Differential Revision: http://phabricator.freedesktop.org/D141
2015-05-04 14:13:46 +02:00
Guillaume Desmottes
db366e0a37 Fix 'stoped' typo
Reviewers: thiblahute

Differential Revision: http://phabricator.freedesktop.org/D137
2015-05-01 18:19:39 +02:00
Guillaume Desmottes
133c415ade validate: display the URL of ignored Valgrind bugs
Summary:
We don't want to forget about those so best to remind it when starting tests
as we do with blacklisted tests.

Reviewers: thiblahute

Differential Revision: http://phabricator.freedesktop.org/D131
2015-05-01 15:15:26 +02:00
Guillaume Desmottes
c5c39d88b1 validate: move look_for_file_in_source_dir and get_valgrind_suppression_file to utils
Reviewers: thiblahute

Differential Revision: http://phabricator.freedesktop.org/D130
2015-05-01 15:15:26 +02:00
Thibault Saunier
753ba408dd validate:launcher: Concider unset MediaDescriptor duration has 'infinite' 2015-04-28 16:44:42 +02:00
Guillaume Desmottes
044ab241ee validate: use -debug versions of bins when running from source
Summary:
Those versions are using rpath instead of libtool's wrappers and so will be
faster to start and won't confuse valgrind.

Reviewers: thiblahute

Differential Revision: http://phabricator.freedesktop.org/D116
2015-04-21 11:19:17 +02:00
Guillaume Desmottes
8c2684c9a7 validate: add gstvalidate.supp valgrind suppression file
Differential Revision: http://phabricator.freedesktop.org/D115
2015-04-20 15:29:28 +02:00
Thibault Saunier
31d11ff8c3 validate:launcher: Use full HD screen as default screen size in xvfb 2015-04-19 11:56:29 +02:00
Thibault Saunier
a0b3287d40 validate:launcher: Add a way to run tests without displaying the output
Summary:
Adding a --no-display option and running Xvfb virtual frame buffer X
server.

Reviewers: Mathieu_Du

Differential Revision: http://phabricator.freedesktop.org/D105
2015-04-17 21:51:29 +02:00
Thibault Saunier
76eedece5c validate:launcher: Error out if valgrind is not available on the system
Summary:
When the user wants to use valgrind, make sure it is present on the
system before doing anything

Reviewers: gdesmott

Differential Revision: http://phabricator.freedesktop.org/D104
2015-04-17 19:58:18 +02:00
Thibault Saunier
adbe811175 validate:launcher: Pass GST_VALIDATE_SCENARIO to the subprocess env only
Summary:
And make sure to remove it from the env if the user has it in its main
environment.

Without that commit we ended up passing scenarios from previous tests
to the following ones where None were specified.

Reviewers: Mathieu_Du

Differential Revision: http://phabricator.freedesktop.org/D98
2015-04-16 12:41:25 +02:00
Guillaume Desmottes
8a6494ddf8 validate: set GST_GL_XINITTHREADS
This ensure that XInitThreads is called and so gl contexts are properly
initialized.

https://bugzilla.gnome.org/show_bug.cgi?id=747840

Signed-off-by: Guillaume Desmottes <guillaume.desmottes@collabora.co.uk>
2015-04-14 12:56:32 +02:00
Thibault Saunier
0fcee6ed84 validate:launcher: Make validate the only default testsuite
Reviewers: Mathieu_Du

Differential Revision: http://phabricator.freedesktop.org/D93
2015-04-13 11:27:08 +02:00
Thibault Saunier
5a6d6b9475 validate:launcher: Let the responsibility to update asset to the testsuite
Summary:
It makes it easier to make sure that the assets needed for a specific
testsuite are available when needed

Reviewers: Mathieu_Du

Differential Revision: http://phabricator.freedesktop.org/D92
2015-04-13 11:27:08 +02:00
Thibault Saunier
2cf082c4be validate:launcher: Set more env variable in the launcher command desc
Summary:
Adding if present:
  * LD_PRELOAD
  * DISPLAY
  * GST_VALIDATE_CONFIG
  * GST_VALIDATE_OVERRIDE

+  enhance the add_env_variable method to more easily set envvar from
current value

Reviewers: Mathieu_Du

Differential Revision: http://phabricator.freedesktop.org/D78
2015-04-10 16:53:41 +02:00
Thibault Saunier
bbd04eef4c validate:launcher: Use GST_VALIDATE_SCENARIO envvar to set scenarios
Summary:
Instead of concidering all apps will have a --set-scenario argument
which is not going to be the case as soon as we run the tests through
LD_PRELOAD

Reviewers: Mathieu_Du

Differential Revision: http://phabricator.freedesktop.org/D70
2015-04-10 16:53:40 +02:00
Thibault Saunier
2fe03f1993 validate:launcher: Move get_current_position from GstValidatePipelineTest to GstValidateTest
This is where it belongs

Reviewers: Mathieu_Du

Differential Revision: http://phabricator.freedesktop.org/D69
2015-04-10 16:53:40 +02:00
Guillaume Desmottes
1bb98b49ca validate: GstValidateMediaCheckTest should inherit from GstValidateTest 2015-03-31 09:45:06 +02:00
Young Han Lee
75205828c7 validate:launcher: Install config.py for non-development mode
Running installed gst-validate-launcher aborted with the following error.

File "lib/gst-validate-launcher/python/launcher/baseclasses.py", line 28, in <module>
  import config
ImportError: No module named config

This is because config.py is added but not installed
in ba6d209b3f.

https://bugzilla.gnome.org/show_bug.cgi?id=747087
2015-03-31 09:29:16 +02:00
Young Han Lee
4725f4ec09 validate:launcher: Show timeout seconds for timeout result message
Current timeout message doesn't show how many seconds a test took and
it is timeouted by normal timeout or hard timeout.

This patch changes the message like following.

1. normal timeout
old : validate.http.playback.reverse_playback.raw_video_mov: Timeout (Application timed out)
new : validate.http.playback.reverse_playback.raw_video_mov: Timeout (Application timed out: 120 secs)

2. hard timeout
old : validate.http.playback.reverse_playback.raw_video_mov: Timeout (Application timed out)
new : validate.http.playback.reverse_playback.raw_video_mov: Timeout (Hard timeout reached: 600 secs)

https://bugzilla.gnome.org/show_bug.cgi?id=746957
2015-03-30 13:32:57 +02:00
Young Han Lee
4c7ec6b8b1 validate:launcher: Fix wrong test number with -j option
When '-j n' option is given, first n tests print test number 0.
This is caused by test_num part of 919db98605.

https://bugzilla.gnome.org/show_bug.cgi?id=747006
2015-03-30 13:02:39 +02:00
Thibault Saunier
ba6d209b3f validate:launcher: Avoid depending on PyGObject
Summary:
And rely on our knowledge of the configuration to figure out where the
suppression file has been installed

Reviewers: gdesmott

Differential Revision: http://phabricator.freedesktop.org/D61
2015-03-28 23:35:07 +01:00
Guillaume Desmottes
4f805422a7 validate: check VALGRIND_ERROR_CODE in Test as well
We were doing it only in GstValidateTest which was overriding the default
implementation.
2015-03-26 13:59:30 +01:00
Guillaume Desmottes
9ad79ffcbc validate: don't increase hard_timeout is if it's None
Some tests, like the media check ones, have None as hard_timeout.
2015-03-26 13:57:34 +01:00
Guillaume Desmottes
daaa32676c validate: increase VALGRIND_TIMEOUT_FACTOR
5 wasn't enough for my poor laptop.
2015-03-26 11:29:26 +01:00
Guillaume Desmottes
66a675cde8 validate: increase the normal timeout as well when using valgrind 2015-03-26 11:29:06 +01:00
Guillaume Desmottes
f1985bc738 validate: raise an error if valgrind detected issues
Differential Revision: http://phabricator.freedesktop.org/D53
2015-03-26 09:18:27 +01:00
Guillaume Desmottes
c23bac2912 validate: store valgrind logs to its own file 2015-03-23 13:41:03 +01:00
Guillaume Desmottes
cb8348c7b1 launcher: try using gst.supp as valgrind suppressions file
https://bugzilla.gnome.org/show_bug.cgi?id=746465
2015-03-20 10:34:49 +01:00
Guillaume Desmottes
271f9f3c8e launcher: add valgrind support
Add a --valgrind option to gst-validate-launcher to run the tests inside
Valgrind and tune GLib's memory allocator accordingly.

Fix https://bugzilla.gnome.org/show_bug.cgi?id=746465
2015-03-20 10:34:49 +01:00
Guillaume Desmottes
e7762d13cc validate:launcher: Fix small typo 2015-03-19 12:22:39 +01:00
Thibault Saunier
ad5b5f5369 validate:launcher: Make sure to show apps specific options in the help 2015-03-18 17:06:14 +01:00
Thibault Saunier
e9da098e33 validate:launcher: Fix test number printing 2015-03-18 17:05:57 +01:00
Thibault Saunier
7079d1910d validate:launcher Rename _other_testsuite_for_tester
To _check_tester_has_other_testsuite
2015-03-17 12:41:26 +01:00
Thibault Saunier
919db98605 validate:launcher: Cache all the tests in the runner
This way we do not have to re ask all the test managers
what tests should be run.
2015-03-17 12:41:26 +01:00