Commit graph

2518 commits

Author SHA1 Message Date
Thibault Saunier
bb0d81fd4c validate: Add details on all g_log message reports.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187>
2020-05-06 22:54:59 +00:00
Thibault Saunier
b65b2bc2fe validate: Add a mechanism to mark tests as skipped
And use it when a plugin is missing and the user didn't ask for
failure when it happens

And use the TAP[0] synthax to report it

[0]: https://testanything.org

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187>
2020-05-06 22:54:59 +00:00
Thibault Saunier
b1e9e409fd validate: Enhance issue reporting from structures
And properly bail out when required
This is preparatory work for gst-test in core which will use the TAP
protocol

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187>
2020-05-06 22:54:59 +00:00
Thibault Saunier
a1a2efc970 validate:flow: Handle some more segment fields filtering
Those slept through when implementing filtering

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/188>
2020-05-05 23:09:31 -04:00
Thibault Saunier
04a648e345 ssim: Minor improvements on the way we attach to pads
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>
2020-04-30 12:40:13 -04:00
Edward Hervey
fb71bf8fab validate-scenario: Refactor seek handling
* Store all seek values into a list of pending seeks instead
  of hardcoding some values
* Store all segments that sinks received
* Match segments to seeks when all sinks received segments with
  the same seqnum
* Detect when a seek did *not* result in segments with identical
  matching seqnums

Should allow checking for all types of seek handling, including
flush-less seeks

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>
2020-04-30 12:40:12 -04:00
Edward Hervey
5dafe4bb64 validate-report: Add new check for top-level seek
This issue is to detect seeks that don't result in segments
with identical seqnums. This check can be done at the top-level
scenario

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>
2020-04-30 12:40:12 -04:00
Edward Hervey
e05d4ac8f5 validate-pad-monitor: Post GstBaseSink SEGMENT on the bus
Allows higher-level bin or app (like validate-scenario) to know
what each sink currently has in terms of SEGMENT.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>
2020-04-30 12:40:12 -04:00
Edward Hervey
854826501c validate-element-monitor: Detect basesink elements
And add a macro

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>
2020-04-30 12:40:12 -04:00
Edward Hervey
4080cd9749 validate-scenario: Add logging for scenario lock taking/releasing
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>
2020-04-30 12:40:12 -04:00
Thibault Saunier
1e8a0dac2a pad-monitor: Reliably track pending seeks
Instead of overriding all values when receiving a seek, store
them as a list of expected values.

This allows handling several seeks in a row, like non-flushing
seeks.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>
2020-04-30 12:40:12 -04:00
Thibault Saunier
ec2a139246 validate: Allow overidding issue severity from configs
Refactoring sensibly to allow getting configs outside the `core` namespace
and outside plugin names.

The `GST_VALIDATE_OVERRIDE` env variable should probably be removed
all together at some point.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/185>
2020-04-29 10:13:51 -04:00
Thibault Saunier
0e85d15da0 validate: Add an option to describe issue types
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/185>
2020-04-29 10:13:51 -04:00
Thibault Saunier
66d29a31fd validate: Show the exact file line when error out in structure files
And minor stdout enhancements

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/185>
2020-04-29 08:33:21 -04:00
Thibault Saunier
a99cbecd98 validate:launcher: Mark some more tests as too long for valgrind 2020-04-29 08:24:18 -04:00
Bilal Elmoussaoui
91bf685fd5 Apply suggestion to debug-viewer/org.freedesktop.GstDebugViewer.appdata.xml.in
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/155>
2020-04-28 15:05:40 +00:00
Bilal Elmoussaoui
5a8cf59451 Metainfo: backport flathub fixes
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/155>
2020-04-28 15:05:40 +00:00
Thibault Saunier
640469ce0c validate: Do not check pulling thread when thread is paused
With decodebin3 we have cases where a task has been started in
the `typefind` element but the demuxer is the one pulling (from
its own thread)

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/183>
2020-04-28 10:17:02 -04:00
Thibault Saunier
75d30ae2d3 validate: Add a check-position action type
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/182>
2020-04-27 17:00:12 -04:00
Thibault Saunier
47f59ad731 validate:scenario: Enhance failures messages
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/182>
2020-04-27 17:00:12 -04:00
Thibault Saunier
9840417a87 validate: Add a way to run a TestClock in scenarios
A TestClock will be used automatically when a scenario has a
`crank-clock` action.

And make `validate` and `debug-viewer` options features in meson,
no reason they weren't and now we require gst-check to build validate

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/182>
2020-04-27 17:00:12 -04:00
Thibault Saunier
bf952d3c8b validate: Introduce the concept of "Test files"
This way we can have a single file that wraps scenarios,
`gst-validate-1.0` arguments, as well as a configuration.

It changes the name of `description` of scenarios to use `meta`
The goal is to replace tests describes in python with dictionary
to fully self contained `.validatetest` files which look like:

```
meta,
    handles-states=true,
    ignore-eos=true,
    gst-validate-args = {
         "videotestsrc pattern=blue ! video/x-raw,format=I420,framerate=1/1 ! timeoverlay ! $(videosink) name=videosink allocation-meta-flags=0",
    },
    configs = {
         "$(validateflow), pad=videosink:sink, buffers-checksum=true, ignored-fields={\"buffers=meta\", }",
    }

play
seek, start=0.0, stop=5.0, flags=accurate+flush, rate=1.0
crank-clock, expected-elapsed-time=0.0
crank-clock, repeat=4, expected-elapsed-time=1.0
crank-clock, expected-elapsed-time=1.0
stop, on-message=eos
```
2020-04-27 16:03:33 -04:00
Thibault Saunier
b7202e2f16 validate:flow: Sensibly improve stdout 2020-04-27 10:52:48 -04:00
Thibault Saunier
9aff82d6bb validate: Fix criticals around NULL structure usage 2020-04-27 10:52:47 -04:00
Thibault Saunier
7d3ada4f4e validate: Fix 'repeat' parameter on non ASYNC action types 2020-04-27 10:52:47 -04:00
Thibault Saunier
05815d4b4f validate:scenario: Do not consider action with 'on-message' as on addition 2020-04-23 19:57:54 -04:00
Thibault Saunier
2b32a68df1 validate: report: Handle when reporting NULL action
And add information about the action repeat state.
2020-04-23 19:57:54 -04:00
Thibault Saunier
449674459a validate: Handle comments in multiline expressions
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/179>
2020-04-22 13:02:29 -04:00
Thibault Saunier
bf24fd3d74 validate: Fix multi variable in a single structure field
We were keeping using the GMatchInfo even after modifying the string
which is explicitly stated as invalid in the GRegex documentation

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/179>
2020-04-22 12:47:20 -04:00
Thibault Saunier
58de8e5330 validate:scenario: Fix 'on-message' actions execution
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/177>
2020-04-21 20:45:51 -04:00
Thibault Saunier
4f7217b0b3 validate: Add { and [ as line continuation markers
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/177>
2020-04-21 15:28:00 -04:00
Nicolas Dufresne
856944c960 validate-launcher: Fix syntax error
This error prevents downloading assets from scratch. This regression was
introduced by MR !145 / commit 2581fef684
2020-04-13 15:38:05 -04:00
Nicolas Dufresne
4bc9f73650 utils: Fix double free in error case
This was detected by Coverity. The content point would have been freed gain in
the done: label.

CID 1461289
2020-04-07 18:33:08 -04:00
Thibault Saunier
81770eda7e validate: Use gst_print for validate report messages 2020-03-23 21:28:45 -03:00
Nicolas Dufresne
3762c1a6d2 ci: Port from only: to rules
This fixed CI breakage introduced by gst-ci!247
2020-03-17 11:53:36 -04:00
Thibault Saunier
9b260a1ec6 validate: Plug some leaks
And add some valgrind suppression for fontconfig
2020-03-12 12:35:45 -03:00
Thibault Saunier
a055c3272e validate:ssim: Allow specifying file framerate to use frame numbers during comparison 2020-03-12 12:35:45 -03:00
Thibault Saunier
f24ca37794 validate:scenario: Make the action->prepare function return a GstValidateExecuteActionReturn
Implementers might want to report the error themselves
2020-03-12 12:35:45 -03:00
Thibault Saunier
7898d5b347 validate: Add a way to check last frame number
This introduces a new 'timecode-frame-number' in the 'check-last-sample'
action type se we can verify the number of output frames.
2020-03-12 12:35:45 -03:00
Thibault Saunier
ee5e0b396e validate: Enhance debug message on invalid expression function call 2020-03-12 12:35:45 -03:00
Thibault Saunier
987d125c7e validate:utils: Allow plain string in gst_validate_utils_get_strv 2020-03-12 12:35:45 -03:00
Thibault Saunier
06f6fd8de1 validatessim: Avoid having ':' in file names
This is unsupported on windows
2020-03-12 12:35:45 -03:00
Thibault Saunier
e4f687e0e7 validate:ssim: Flush cairo surface before getting pixels 2020-03-12 12:35:45 -03:00
Thibault Saunier
1468b57430 validate:launcher: Fix reporting on failure when running forever/fatal 2020-03-12 12:35:45 -03:00
Thibault Saunier
4ae29ab7ed validate:ssim: Enhance printing position
Adding a new `gst_validate_print_position` method which also
sends messages to the runner if required.
2020-03-12 12:35:45 -03:00
Thibault Saunier
0b41b8a8c0 validate:launcher: Add a generator to generate test for frame accurate seeking 2020-03-12 12:35:45 -03:00
Thibault Saunier
30ceb4ef40 validate:ssim: Use stream time to reference frames 2020-03-11 16:36:20 -03:00
Thibault Saunier
289c87de5d validate:ssim: Don't check neighbor frames when comparing exact same frame 2020-03-11 16:36:20 -03:00
Thibault Saunier
d816853838 validate:ssim: Enhance debugging message when similarity do not match
Printing out where the diff image file is
2020-03-11 16:36:20 -03:00
Thibault Saunier
2e45db5ba4 validate:launcher: Enhance support for running ssim tests
Using a special 'ssim' variable in pipeline dicts to activate it
Similare to what we do for validateflow
2020-03-11 16:36:20 -03:00