Thibault Saunier
e00ed70925
validate: Fix rendering destination directory path creation
...
We were ending up creating file:/some/path in cwd
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187 >
2020-05-06 22:54:59 +00:00
Thibault Saunier
af3006dc12
validate: Plug some leaks
...
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187 >
2020-05-06 22:54:59 +00:00
Thibault Saunier
b65b2bc2fe
validate: Add a mechanism to mark tests as skipped
...
And use it when a plugin is missing and the user didn't ask for
failure when it happens
And use the TAP[0] synthax to report it
[0]: https://testanything.org
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187 >
2020-05-06 22:54:59 +00:00
Thibault Saunier
a99cbecd98
validate:launcher: Mark some more tests as too long for valgrind
2020-04-29 08:24:18 -04:00
Thibault Saunier
bf952d3c8b
validate: Introduce the concept of "Test files"
...
This way we can have a single file that wraps scenarios,
`gst-validate-1.0` arguments, as well as a configuration.
It changes the name of `description` of scenarios to use `meta`
The goal is to replace tests describes in python with dictionary
to fully self contained `.validatetest` files which look like:
```
meta,
handles-states=true,
ignore-eos=true,
gst-validate-args = {
"videotestsrc pattern=blue ! video/x-raw,format=I420,framerate=1/1 ! timeoverlay ! $(videosink) name=videosink allocation-meta-flags=0",
},
configs = {
"$(validateflow), pad=videosink:sink, buffers-checksum=true, ignored-fields={\"buffers=meta\", }",
}
play
seek, start=0.0, stop=5.0, flags=accurate+flush, rate=1.0
crank-clock, expected-elapsed-time=0.0
crank-clock, repeat=4, expected-elapsed-time=1.0
crank-clock, expected-elapsed-time=1.0
stop, on-message=eos
```
2020-04-27 16:03:33 -04:00
Nicolas Dufresne
856944c960
validate-launcher: Fix syntax error
...
This error prevents downloading assets from scratch. This regression was
introduced by MR !145 / commit 2581fef684
2020-04-13 15:38:05 -04:00
Thibault Saunier
1468b57430
validate:launcher: Fix reporting on failure when running forever/fatal
2020-03-12 12:35:45 -03:00
Thibault Saunier
0b41b8a8c0
validate:launcher: Add a generator to generate test for frame accurate seeking
2020-03-12 12:35:45 -03:00
Thibault Saunier
2e45db5ba4
validate:launcher: Enhance support for running ssim tests
...
Using a special 'ssim' variable in pipeline dicts to activate it
Similare to what we do for validateflow
2020-03-11 16:36:20 -03:00
Thibault Saunier
b4c1424ea2
validate:launcher: Add support for skipped media info files
...
Those are skipped to generate tests by default but are updated when
required, this will allow us to generate specific test on demand for
those
2020-03-11 16:36:16 -03:00
Thibault Saunier
48e007cb29
validate:launcher: Never try to load a testsuite with the same name from different locations
2020-03-06 16:05:38 -03:00
Thibault Saunier
080fdf8e2f
validate:launcher: Add support for the imagesequence protocol
2020-03-06 16:05:38 -03:00
Thibault Saunier
e4ea35d25c
validate:launcher: Fix reporting on failure when running forever/fatal
2020-03-06 15:59:17 -03:00
Thibault Saunier
e7b8ba2d68
validate:launcher: Cache GstValidateMediaDescriptor to avoid reparsing .media_info
...
Saving another second at startup
2020-03-06 15:59:17 -03:00
Thibault Saunier
bc622ec658
validate:launcher: Batch inspecting scenarios
...
Removing almost 1 second to start running tests with the default
testsuite
2020-03-06 15:36:10 -03:00
Thibault Saunier
ca93a4b704
validate:launcher: Fix printed test number
2020-03-06 15:36:10 -03:00
Thibault Saunier
d1c3607bc3
validate:launcher: Allow passing configs paths when loading from dictionary
...
The same way we allow it for scenarios
2020-03-06 15:36:10 -03:00
Thibault Saunier
3d3a5bee25
launcher: Do not dump output on known issues
...
And remove dead code
2020-02-26 12:05:39 -03:00
Thibault Saunier
2036d8292f
validate: Enhance error reporting when scenario or configs are invalid
2020-02-25 11:28:35 -03:00
Thibault Saunier
2f135d430b
validate:launcher: Handle validate report bigger than allowed by the socket
...
This almost never happens but I had a case where we had a report
with a GstSample in the caps that were reported leading to an
error printed.
2020-02-13 00:11:45 +00:00
Thibault Saunier
d17c2ff65c
validate:launcher: Reference exception in a var as we use it in the handling
2020-02-13 00:11:45 +00:00
Thibault Saunier
d3c6611b76
validate:launcher: Fix defining several scenario with a same config
...
When generating tests from dictionary the dict format allows passing
several scenario for a same config and pipelines, but this was breaking
the case where expected flow is different with each config, instead we
should generate one config per scenario, fixing the expectation files
generated.
2020-02-06 14:33:56 -03:00
Thibault Saunier
600c5a27ed
validate:launcher: Print the duration of the test run in the logs
2020-02-04 16:09:36 -03:00
Stéphane Cerveau
a2e926ff0a
gst-validate-launcher: separate known error from passed tests
...
Introduce known_error in statistics to keep in mind the expected
error result.
2020-01-24 14:52:49 +00:00
Stéphane Cerveau
2581fef684
gst-validate-launcher: update documentation
...
Use the new api to create your custom testsuite.
Fix some broken links and enhance the logging system.
2020-01-24 14:52:49 +00:00
Nicolas Dufresne
1bc8e92efc
launcher: Allow partionning the tests
...
This introduce new command line options, --parts and --part-index. When
--parts is set to a value larger then 1, the tests will be split in the
same number of group. The group number identified by --part-index will
be executed.
This is being added in orther to support gliblab CI parallel feature.
2020-01-15 10:04:11 -05:00
Thibault Saunier
e4ca67938e
validate:launcher: Try to dump logs with bat if avalaible
...
We got to many issues with mdv, it seems not to be maintained
bat is a very good replacement.
2020-01-06 13:14:59 +00:00
Brady J. Garvin
06822b519b
validate:launcher: Support mixed str/bytes control sequences.
...
It is not safe for `_preformat_levels` to assume that all of the fields in a
`TerminalController` have the same type; at least in my environment, some of
these fields are populated with `bytes` while others remain strings.
This change conditionally applies decoding to each control sequence separately
using a helper function `_as_string`. As a side-effect, it also eliminates some
code repetition in `_preformat_levels`.
Closes #50 .
2020-01-05 14:09:07 -06:00
Thibault Saunier
195d3a3edc
validate:launcher: Enhance progress reporting using a progress bar
...
This also allows us to properly report progress on the CI
2019-12-30 16:52:22 +00:00
Thibault Saunier
d543375948
validate:launcher: Take our timeout factor into account for gstcheck
2019-12-30 15:56:02 +00:00
Mathieu Duponchelle
27677d4e10
check: unblacklist removed systemclock tests
...
See https://gitlab.freedesktop.org/gstreamer/gstreamer/merge_requests/348
2019-12-23 10:27:11 +01:00
Mathieu Duponchelle
48faa5944b
check: unblacklist gstreamer.pipelines_parse_launch.delayed_link
...
It should not be flaky anymore after
https://gitlab.freedesktop.org/gstreamer/gstreamer/merge_requests/343
2019-12-14 10:45:30 +01:00
Thibault Saunier
1f67876928
validate:launcher: Use python dict for pipeline description
2019-12-10 13:02:16 +00:00
Thibault Saunier
562750213f
validate: launcher: Make encoding extra check use common code path
...
Reusing the reporting infrastructure instead of shurtcuting it
2019-11-20 18:07:48 -03:00
Thibault Saunier
8a6ee4841b
validate: Add vp9 transcoding tests
...
Making sure to encode small frames as vp9enc is slow.
2019-11-20 18:07:47 -03:00
Thibault Saunier
80de180cc7
validate:launcher: Avoid clashes when importing testsuite
...
This introduce an hard dependency on python >= 3.5, same as meson
2019-11-11 18:57:27 -03:00
Jordan Petridis
6b907ba036
check: blacklist gst-plugins-good.elements_splitmux.test_splitmuxsink$
...
https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/626
2019-11-11 14:25:16 +01:00
Stéphane Cerveau
998185acbc
gstvalidate: fix GstValidateRTSPMediaDescriptor typo
2019-11-06 18:21:11 +01:00
Víctor Manuel Jáquez Leal
37a0dbfebf
validate: blacklist gstreamer-vaapi checks
...
They still can be checked by running the tests explicitly.
2019-10-25 12:26:58 +02:00
Tim-Philipp Müller
280321dee1
validate: remove autotools build
2019-10-15 00:24:00 +01:00
Thibault Saunier
2153d82f34
validate:launcher: Fix the 'can-happen-several-times' known issue field
2019-10-02 19:22:31 +00:00
Thibault Saunier
461e479802
validate:launcher: Do not hardcode pathsep
2019-08-28 15:34:31 -04:00
Philippe Normand
3351807107
validate/launcher: Ensure the HTTP server is started when a pipeline needs it
...
Pipelines declared in gst-integration-testsuites can rely on the validate HTTP
server, so when an URI pointing to it is detected, advertise the server as
needed before starting the test.
For this to work the test scenario should explicitely declare the pipeline uri,
as shown in this example:
"some_playbin3":
{
"pipeline": "playbin3 uri=%(uri)s video-sink=%(videosink)s",
"config": [
"%(validateflow)s, pad=sink:sink"
],
"scenarios": ["play_15s"],
"uri": "http://127.0.0.1:%(http-server-port)s/defaults/html/foo.html "
}
2019-08-21 08:37:38 +01:00
Thibault Saunier
8e01e03364
validate:launcher: Allow passing any extra_data in json test definition
...
This means that we can now pass any extra key that `populate_tests`
expects, meaning any key expected by FakeMediaDescriptor and
a few other keys supported by the methods such as
`expected-issues` and `extra_env_vars`
2019-08-20 13:55:52 +00:00
Thibault Saunier
556bc0bb4f
launcher: Raise an exception when provided scenario can't be found
2019-08-19 10:33:20 +00:00
Mathieu Duponchelle
5a068bff72
validate: Update blacklisting reason for fast forward rtsp
...
While https://gitlab.freedesktop.org/gstreamer/gst-rtsp-server/issues/14
was merged, the client side (in particular rtpbasedepayload) still
isn't expected to work appropriately
2019-08-01 21:04:12 +02:00
Mathieu Duponchelle
93114b55ab
json tests: assume all user pipelines can seek reverse
...
When doing a targeted test, it is up to the user to make sure
their pipeline + scenario behaves correctly.
2019-07-26 02:26:20 +02:00
Thibault Saunier
ba4275ad00
validate:launcher: Pass the right timeout_factor is passed to subprojects
2019-07-08 23:37:31 -04:00
Thibault Saunier
3343f166da
validate:launcher: Move get_fakesink_for_media_type to utils
...
So it can be reused in other apps like GES
2019-06-26 12:09:03 -04:00
Thibault Saunier
7d471ee25e
validate: Set 'LOGSDIR' variable in scenarios and config files
...
Implementing support for variables in config files.
2019-06-23 03:10:34 -04:00