By default, generate them whenever the file is missing but adding a way
to override that with `validateflow,generate-expectations=true` to force
regenerating them or setting `validateflow,generate-expectations=false`
to disallow generating them (on CI servers for example)
Also update the validateflow documentation to take that into account
and remove references to pipeline.json file which is now gone!
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/200>
This way we can have a single file that wraps scenarios,
`gst-validate-1.0` arguments, as well as a configuration.
It changes the name of `description` of scenarios to use `meta`
The goal is to replace tests describes in python with dictionary
to fully self contained `.validatetest` files which look like:
```
meta,
handles-states=true,
ignore-eos=true,
gst-validate-args = {
"videotestsrc pattern=blue ! video/x-raw,format=I420,framerate=1/1 ! timeoverlay ! $(videosink) name=videosink allocation-meta-flags=0",
},
configs = {
"$(validateflow), pad=videosink:sink, buffers-checksum=true, ignored-fields={\"buffers=meta\", }",
}
play
seek, start=0.0, stop=5.0, flags=accurate+flush, rate=1.0
crank-clock, expected-elapsed-time=0.0
crank-clock, repeat=4, expected-elapsed-time=1.0
crank-clock, expected-elapsed-time=1.0
stop, on-message=eos
```
When generating tests from dictionary the dict format allows passing
several scenario for a same config and pipelines, but this was breaking
the case where expected flow is different with each config, instead we
should generate one config per scenario, fixing the expectation files
generated.
Pipelines declared in gst-integration-testsuites can rely on the validate HTTP
server, so when an URI pointing to it is detected, advertise the server as
needed before starting the test.
For this to work the test scenario should explicitely declare the pipeline uri,
as shown in this example:
"some_playbin3":
{
"pipeline": "playbin3 uri=%(uri)s video-sink=%(videosink)s",
"config": [
"%(validateflow)s, pad=sink:sink"
],
"scenarios": ["play_15s"],
"uri": "http://127.0.0.1:%(http-server-port)s/defaults/html/foo.html"
}
This means that we can now pass any extra key that `populate_tests`
expects, meaning any key expected by FakeMediaDescriptor and
a few other keys supported by the methods such as
`expected-issues` and `extra_env_vars`
Instead of having the issues centered on the test classes, they
are now focusing on the "bug".
And harmise names on `expected_issue` not `expected_failures`
validateflow can be used to check the buffers and events flowing through
a custom pipeline match an expectation file. This can be used to test
non-regular-playback use cases like demuxers handling adaptive streaming
fragment pushing.
This patch includes also new actions used for these cases:
`appsrc-push`, `appsrc-eos` and `flush` (plus `checkpoint`, which is
only available with validateflow).
When defining pipelines_descriptions to run test on in a `.json` file, you might
need to point to paths in the testsuite directory (for media files URIs
for example), you can now do
`"pipeline": "filesrc location="$(config_path)s/../medias/some/file.mkv...`
Introducing the `.media_info.push` media info extension, which is meant
to let the launcher know that those file should run with the "pushfile://"
protocol.
And allow symlinking "normal" `.media_info` to their `.pushfile` variant
so that both can share the exact same content.
This is useful when you want to check only the demuxer output.
- Keep the information in the media file so that we can launch media-check
with the proper arguments in the launcher. Update it accordingly.
- Refactor compare_streams to simplify it, which in the end leads to
reporting all the issues instead of exiting on the first one.
We will run a simple pipeline with the IQA element to run ssim (dssim)
tests on the rendered files, comparing it with a reference file.
For now we use the very empiric 1.0 value as a ssim error threshold and
the goal is basically to detect completely broken renderings.
The issue is closed upstream (because of concentrating on decodebin3
instead), and initial forever testing seems to show the issue doesn't
happen anymore
Since we now check position/status of pipeline at regular intevals,
we no longer need to impose a different timeout based on the
protocol used.
Avoids having 4min long timeouts for no reason (30s is enough)
