Commit graph

30 commits

Author SHA1 Message Date
Thibault Saunier
124153fe06 validateflow: Allow specifying checksum type
And add an extra mode 'checksum-as-id' which basically numerate
buffers checksums as they are being received so that it is simpler
to compare expectations when you are tracking buffers from both
sinkpads and srcpads.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/202>
2020-05-31 17:17:22 -04:00
Thibault Saunier
012288910b validate: Update documentation
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/200>
2020-05-28 17:17:10 -04:00
Thibault Saunier
b1cf1ffebd validateflow: Add a way to configure when to generate expectations
By default, generate them whenever the file is missing but adding a way
to override that with `validateflow,generate-expectations=true` to force
regenerating them or setting `validateflow,generate-expectations=false`
to disallow generating them (on CI servers for example)

Also update the validateflow documentation to take that into account
and remove references to pipeline.json file which is now gone!

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/200>
2020-05-28 17:17:10 -04:00
Thibault Saunier
c9318639d7 docs: Update gst-validate-launcher documentation
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>
2020-05-26 17:19:36 -04:00
Thibault Saunier
b669bb0327 validate: Add support for known-issues in the .validatetest
And add some tests about remaining actions failures

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/189>
2020-05-07 09:36:03 -04:00
Thibault Saunier
b65b2bc2fe validate: Add a mechanism to mark tests as skipped
And use it when a plugin is missing and the user didn't ask for
failure when it happens

And use the TAP[0] synthax to report it

[0]: https://testanything.org

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187>
2020-05-06 22:54:59 +00:00
Thibault Saunier
ec2a139246 validate: Allow overidding issue severity from configs
Refactoring sensibly to allow getting configs outside the `core` namespace
and outside plugin names.

The `GST_VALIDATE_OVERRIDE` env variable should probably be removed
all together at some point.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/185>
2020-04-29 10:13:51 -04:00
Thibault Saunier
9840417a87 validate: Add a way to run a TestClock in scenarios
A TestClock will be used automatically when a scenario has a
`crank-clock` action.

And make `validate` and `debug-viewer` options features in meson,
no reason they weren't and now we require gst-check to build validate

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/182>
2020-04-27 17:00:12 -04:00
Thibault Saunier
bf952d3c8b validate: Introduce the concept of "Test files"
This way we can have a single file that wraps scenarios,
`gst-validate-1.0` arguments, as well as a configuration.

It changes the name of `description` of scenarios to use `meta`
The goal is to replace tests describes in python with dictionary
to fully self contained `.validatetest` files which look like:

```
meta,
    handles-states=true,
    ignore-eos=true,
    gst-validate-args = {
         "videotestsrc pattern=blue ! video/x-raw,format=I420,framerate=1/1 ! timeoverlay ! $(videosink) name=videosink allocation-meta-flags=0",
    },
    configs = {
         "$(validateflow), pad=videosink:sink, buffers-checksum=true, ignored-fields={\"buffers=meta\", }",
    }

play
seek, start=0.0, stop=5.0, flags=accurate+flush, rate=1.0
crank-clock, expected-elapsed-time=0.0
crank-clock, repeat=4, expected-elapsed-time=1.0
crank-clock, expected-elapsed-time=1.0
stop, on-message=eos
```
2020-04-27 16:03:33 -04:00
Thibault Saunier
a055c3272e validate:ssim: Allow specifying file framerate to use frame numbers during comparison 2020-03-12 12:35:45 -03:00
Thibault Saunier
db2acf1cdf validate:flow: Make field filtering in what is logged more generic
Instead of forcing it on event, allow specifying filters on anything
we log, meaning also buffers
2020-02-13 00:11:45 +00:00
Thibault Saunier
39d6c7760e validateflow: Add a logged-event-fields configuration 2020-02-13 00:11:45 +00:00
Thibault Saunier
5b82274f17 validate:flow: Add a way to set the types of events to log/ignore
Added two properties to the plugin:
     * ignored-event-types: A list of event types to be ignored when logging events
     * logged-event-types: A list of event types to be logged when logging events

This commits also moves the "ignored-event-fields" property to using a proper
GstValueList for the list of event fields to be taken into account, instead
of the home grown separated by comas list of string, making the API more
uniform.

This also adds a simple helper method: `gst_validate_utils_get_strv`
2020-02-04 16:09:36 -03:00
Stéphane Cerveau
2581fef684 gst-validate-launcher: update documentation
Use the new api to create your custom testsuite.
Fix some broken links and enhance the logging system.
2020-01-24 14:52:49 +00:00
Thibault Saunier
71fe16fd1c doc: Minor typo fixes. 2019-06-27 00:41:03 +00:00
Thibault Saunier
bdb7990d64 docs: Document validate core configuration 2019-06-26 12:09:03 -04:00
Thibault Saunier
25a7173b22 validateflow: Add buffers-checksum option to log buffers data checksum 2019-06-26 12:09:03 -04:00
Thibault Saunier
ade8ba3fcb validate: Add SCENARIO_NAME and CONFIG_NAME vars in configs/scenarios 2019-06-26 12:09:03 -04:00
Thibault Saunier
7d471ee25e validate: Set 'LOGSDIR' variable in scenarios and config files
Implementing support for variables in config files.
2019-06-23 03:10:34 -04:00
Thibault Saunier
b11c5ba185 scenario: Set SCENARIO_PATH/DIR variables in scenarios
And add some documentation about it
2019-06-23 01:54:26 -04:00
Thibault Saunier
0e0928b0b7 scenario: Add a TMPDIR global variables in scenarios
This also adds the notion of global variables which will be useable
in config files too.

And add some documentation about default variables in scenarios
2019-06-23 01:34:41 -04:00
Thibault Saunier
db487b2732 docs: Update validate action types
Include minor fixes in the action types and markdown generator
2019-06-18 18:15:30 -04:00
Thibault Saunier
4a250437a7 docs: Document GES validate action types 2019-06-17 09:53:07 -04:00
Thibault Saunier
7c84ca55d3 docs: Minor fix about ssim plugin 2019-05-23 11:34:19 -04:00
Thibault Saunier
85282e53ca docs: Fix docstrings 2019-05-13 17:00:00 -04:00
Thibault Saunier
1d52eb0260 docs: Document the validateflow plugin 2019-05-13 11:37:38 -04:00
Thibault Saunier
aeb9520029 docs: Document validate action types
Just did:

```
 gst-validate-1.0 --inspect-action-type all >   docs/gst-validate-action-types.md
```
2019-05-13 11:37:38 -04:00
Thibault Saunier
0a7b23f566 doc: Port to hotdoc
Ideally we want a GstValidate hotdoc plugin... not for now.
2019-05-13 11:37:38 -04:00
Thibault Saunier
b61cb56e20 validate-flow: Add an ignored-event-fields configuration
It replaces `record-stream-id` as it is a more generic way of doing
the same thing.
2019-03-11 18:14:23 -03:00
Thibault Saunier
8f75606bf7 docs: Document the validateflow plugin 2019-03-11 18:11:38 -03:00