Philippe Normand
ac88851587
validate: launcher: Simplify fakesink handling
...
Now the function returns either a fakeaudiosink or a fakevideosink, depending on
the media type.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/246 >
2021-05-25 10:46:47 +00:00
Stéphane Cerveau
1fd8c1501a
validate: add config file support
...
Each test can now use a config file for
the given media file used to test.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/243 >
2021-05-05 10:58:10 +02:00
Brady J. Garvin
2b8eef30e7
validate:launcher: Ensure a positive job count.
...
The default number of jobs to use is half of the available cores
rounded down, but in situations where only one core is available (such
as under some VMs), this means that `gst-validate-launcher` defaults
to using zero jobs, a case that the test-running code is not prepared
to handle.
This change makes the code match the documentation for the `--jobs` option,
guards against negative values both in the default setting and in argument
parsing, and introduces some defensive programming to prevent other situations
where the code might try to use zero jobs.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/154 >
2021-01-30 10:11:57 -06:00
Stéphane Cerveau
88b320cc2b
validate: add sync-version
...
Be able by the command line to change the sync version
which is usually the GST_VALIDATE_TESTSUITE_VERSION
from the test suite
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/227 >
2020-12-08 15:25:30 +00:00
Thibault Saunier
bf56c5ab92
validate:launcher: Bump hard timeouts for all transcodin tests
...
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/172 >
2020-10-23 12:18:40 -03:00
Thibault Saunier
d811e1b39f
launcher: Avoid variable framerate when encoding to theora
...
It is not supported by theoraenc.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/172 >
2020-10-23 12:18:40 -03:00
Thibault Saunier
6c08b2c570
validate: Scale down even more to speed up encoding in VP9
...
From 3min to 50secs to execute here.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/172 >
2020-10-23 12:18:40 -03:00
Thibault Saunier
5b7ac97502
validate:transcoding: Port to GstTranscoder
...
Remove flag to force EOS on sigintr, making it the only choice
Also add support for variable framerate
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/172 >
2020-10-23 12:18:40 -03:00
Thibault Saunier
06793d1d34
validate:launcher: Add a way to define test timeout from dicts
...
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/172 >
2020-10-23 12:18:21 -03:00
Tim-Philipp Müller
c5d4d42164
Revert "Revert "launcher: add webrtcbin datachannel tests to valgrind skip list""
...
This reverts commit ff79af843b
.
Looks like most of these are still problematic and flaky in valgrind,
so re-add to skiplist for now.
https://gitlab.freedesktop.org/thaytan/gst-plugins-base/-/jobs/4275045
2020-08-26 15:47:23 +01:00
Tim-Philipp Müller
ff79af843b
Revert "launcher: add webrtcbin datachannel tests to valgrind skip list"
...
This reverts commit 369c74941f
.
Fixes https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/issues/1383
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/220 >
2020-08-24 23:59:14 +01:00
Thibault Saunier
e995c5ccc3
validate:check: Blacklist new ges valgrind test
...
I can't find what the issue is and the test is very long anyway
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/219 >
2020-08-14 11:29:06 -04:00
Tim-Philipp Müller
369c74941f
launcher: add webrtcbin datachannel tests to valgrind skip list
...
They were previously not run because the sctp plugin wasn't built
but they will be run now that we bundle libusrsctp.
https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1465
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/218 >
2020-08-14 00:55:59 +01:00
Thibault Saunier
19550bec3d
launcher: Keep running tests forever on KNOWN_ERROR
...
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207 >
2020-06-22 17:20:32 +00:00
Thibault Saunier
f0db9bc907
validate:launcher: Ensure that the main directory exists
...
Fixes https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/issues/53 part 1
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/209 >
2020-06-15 18:42:03 -04:00
Edward Hervey
2dd165b4ec
launcher: Limit copies of massive debug logs in markdown file
...
When debugging is activated, we could end up with log files ranging in the
multi-megabyte or even gigabyte range. Copying those is expensive from a cpu/io
point of view in addition to clobbering the storage.
Instead of always copying those files, check if they are smaller than 500kB. If
not, don't copy them and instead provide a link to their location.
Fixes #52
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/203 >
2020-06-03 09:39:11 +02:00
Thibault Saunier
b1cf1ffebd
validateflow: Add a way to configure when to generate expectations
...
By default, generate them whenever the file is missing but adding a way
to override that with `validateflow,generate-expectations=true` to force
regenerating them or setting `validateflow,generate-expectations=false`
to disallow generating them (on CI servers for example)
Also update the validateflow documentation to take that into account
and remove references to pipeline.json file which is now gone!
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/200 >
2020-05-28 17:17:10 -04:00
Thibault Saunier
e49876bb05
validate:launcher: Stop wrongly erroring on testsuite loading failure
...
When the testsuite was actually already loaded as the .py
file was explicitly passed in
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197 >
2020-05-26 19:16:27 -04:00
Thibault Saunier
91ac75aa4c
launcher: check: Properly set PLUGIN_PATH and registry when running in gst-build
...
This makes registry
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197 >
2020-05-26 19:16:26 -04:00
Thibault Saunier
90cc65f7ee
launcher: Ensure that -j tests run in parallel when running forever
...
So that you can reproduce the issue you want faster!
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197 >
2020-05-26 17:19:36 -04:00
Thibault Saunier
766f339fb8
launcher: Add support for running tests inside rr
...
Allowing us to easily run the tests forever and then replay the
failures!
https://rr-project.org/
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197 >
2020-05-26 17:19:36 -04:00
Thibault Saunier
884c45ff3e
validate:launcher:check: Add support for gst-tester-1.0
...
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/189 >
2020-05-07 09:39:16 -04:00
Thibault Saunier
b669bb0327
validate: Add support for known-issues in the .validatetest
...
And add some tests about remaining actions failures
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/189 >
2020-05-07 09:36:03 -04:00
Thibault Saunier
e00ed70925
validate: Fix rendering destination directory path creation
...
We were ending up creating file:/some/path in cwd
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187 >
2020-05-06 22:54:59 +00:00
Thibault Saunier
af3006dc12
validate: Plug some leaks
...
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187 >
2020-05-06 22:54:59 +00:00
Thibault Saunier
b65b2bc2fe
validate: Add a mechanism to mark tests as skipped
...
And use it when a plugin is missing and the user didn't ask for
failure when it happens
And use the TAP[0] synthax to report it
[0]: https://testanything.org
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187 >
2020-05-06 22:54:59 +00:00
Thibault Saunier
a99cbecd98
validate:launcher: Mark some more tests as too long for valgrind
2020-04-29 08:24:18 -04:00
Thibault Saunier
bf952d3c8b
validate: Introduce the concept of "Test files"
...
This way we can have a single file that wraps scenarios,
`gst-validate-1.0` arguments, as well as a configuration.
It changes the name of `description` of scenarios to use `meta`
The goal is to replace tests describes in python with dictionary
to fully self contained `.validatetest` files which look like:
```
meta,
handles-states=true,
ignore-eos=true,
gst-validate-args = {
"videotestsrc pattern=blue ! video/x-raw,format=I420,framerate=1/1 ! timeoverlay ! $(videosink) name=videosink allocation-meta-flags=0",
},
configs = {
"$(validateflow), pad=videosink:sink, buffers-checksum=true, ignored-fields={\"buffers=meta\", }",
}
play
seek, start=0.0, stop=5.0, flags=accurate+flush, rate=1.0
crank-clock, expected-elapsed-time=0.0
crank-clock, repeat=4, expected-elapsed-time=1.0
crank-clock, expected-elapsed-time=1.0
stop, on-message=eos
```
2020-04-27 16:03:33 -04:00
Nicolas Dufresne
856944c960
validate-launcher: Fix syntax error
...
This error prevents downloading assets from scratch. This regression was
introduced by MR !145 / commit 2581fef684
2020-04-13 15:38:05 -04:00
Thibault Saunier
1468b57430
validate:launcher: Fix reporting on failure when running forever/fatal
2020-03-12 12:35:45 -03:00
Thibault Saunier
0b41b8a8c0
validate:launcher: Add a generator to generate test for frame accurate seeking
2020-03-12 12:35:45 -03:00
Thibault Saunier
2e45db5ba4
validate:launcher: Enhance support for running ssim tests
...
Using a special 'ssim' variable in pipeline dicts to activate it
Similare to what we do for validateflow
2020-03-11 16:36:20 -03:00
Thibault Saunier
b4c1424ea2
validate:launcher: Add support for skipped media info files
...
Those are skipped to generate tests by default but are updated when
required, this will allow us to generate specific test on demand for
those
2020-03-11 16:36:16 -03:00
Thibault Saunier
48e007cb29
validate:launcher: Never try to load a testsuite with the same name from different locations
2020-03-06 16:05:38 -03:00
Thibault Saunier
080fdf8e2f
validate:launcher: Add support for the imagesequence protocol
2020-03-06 16:05:38 -03:00
Thibault Saunier
e4ea35d25c
validate:launcher: Fix reporting on failure when running forever/fatal
2020-03-06 15:59:17 -03:00
Thibault Saunier
e7b8ba2d68
validate:launcher: Cache GstValidateMediaDescriptor to avoid reparsing .media_info
...
Saving another second at startup
2020-03-06 15:59:17 -03:00
Thibault Saunier
bc622ec658
validate:launcher: Batch inspecting scenarios
...
Removing almost 1 second to start running tests with the default
testsuite
2020-03-06 15:36:10 -03:00
Thibault Saunier
ca93a4b704
validate:launcher: Fix printed test number
2020-03-06 15:36:10 -03:00
Thibault Saunier
d1c3607bc3
validate:launcher: Allow passing configs paths when loading from dictionary
...
The same way we allow it for scenarios
2020-03-06 15:36:10 -03:00
Thibault Saunier
3d3a5bee25
launcher: Do not dump output on known issues
...
And remove dead code
2020-02-26 12:05:39 -03:00
Thibault Saunier
2036d8292f
validate: Enhance error reporting when scenario or configs are invalid
2020-02-25 11:28:35 -03:00
Thibault Saunier
2f135d430b
validate:launcher: Handle validate report bigger than allowed by the socket
...
This almost never happens but I had a case where we had a report
with a GstSample in the caps that were reported leading to an
error printed.
2020-02-13 00:11:45 +00:00
Thibault Saunier
d17c2ff65c
validate:launcher: Reference exception in a var as we use it in the handling
2020-02-13 00:11:45 +00:00
Thibault Saunier
d3c6611b76
validate:launcher: Fix defining several scenario with a same config
...
When generating tests from dictionary the dict format allows passing
several scenario for a same config and pipelines, but this was breaking
the case where expected flow is different with each config, instead we
should generate one config per scenario, fixing the expectation files
generated.
2020-02-06 14:33:56 -03:00
Thibault Saunier
600c5a27ed
validate:launcher: Print the duration of the test run in the logs
2020-02-04 16:09:36 -03:00
Stéphane Cerveau
a2e926ff0a
gst-validate-launcher: separate known error from passed tests
...
Introduce known_error in statistics to keep in mind the expected
error result.
2020-01-24 14:52:49 +00:00
Stéphane Cerveau
2581fef684
gst-validate-launcher: update documentation
...
Use the new api to create your custom testsuite.
Fix some broken links and enhance the logging system.
2020-01-24 14:52:49 +00:00
Nicolas Dufresne
1bc8e92efc
launcher: Allow partionning the tests
...
This introduce new command line options, --parts and --part-index. When
--parts is set to a value larger then 1, the tests will be split in the
same number of group. The group number identified by --part-index will
be executed.
This is being added in orther to support gliblab CI parallel feature.
2020-01-15 10:04:11 -05:00
Thibault Saunier
e4ca67938e
validate:launcher: Try to dump logs with bat if avalaible
...
We got to many issues with mdv, it seems not to be maintained
bat is a very good replacement.
2020-01-06 13:14:59 +00:00