Commit graph

2560 commits

Author SHA1 Message Date
Thibault Saunier
ec41001c80 scenario: Add an option to set properties on all instances
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/205>
2020-06-10 17:30:33 -04:00
Thibault Saunier
83ebecef5c validate:flow: Add a way to dump buffer content as hex
Useful in unit tests with very small buffers

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/205>
2020-06-10 17:30:33 -04:00
Thibault Saunier
3b69b73299 validate: Cleanup implementation of appsrc-push
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/205>
2020-06-10 17:30:33 -04:00
Thibault Saunier
70634b1a3f validate: Remove useless condition
CID 1462652

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/192>
2020-06-07 15:47:36 -04:00
Thibault Saunier
eb6c126068 validate:ssim: Avoid dereferencing NULL pointer
CID 1462650

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/192>
2020-06-06 06:00:11 +00:00
Thibault Saunier
66972abafc validate: Give a proper argv[0] when running test files
Fixes CID 1462613

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/192>
2020-06-06 06:00:11 +00:00
Thibault Saunier
fdfe797d4e validate:transcoding: Exit after printing the help
Fixes CID 1455575

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/192>
2020-06-06 06:00:11 +00:00
Thibault Saunier
f6e45c8c90 validate: Ensure Scenario isn't NULL when preparing action
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/192>
2020-06-06 06:00:11 +00:00
Thibault Saunier
7ee3421cad validate: Update documentation now core plugins are integrated
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/204>
2020-06-03 14:29:22 -04:00
Thibault Saunier
88fcc580be validate: Make extracheck and flow part of core instead plugins
It makes things more complex and doesn't bring anything!

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/204>
2020-06-03 11:07:15 -04:00
Edward Hervey
2dd165b4ec launcher: Limit copies of massive debug logs in markdown file
When debugging is activated, we could end up with log files ranging in the
multi-megabyte or even gigabyte range. Copying those is expensive from a cpu/io
point of view in addition to clobbering the storage.

Instead of always copying those files, check if they are smaller than 500kB. If
not, don't copy them and instead provide a link to their location.

Fixes #52

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/203>
2020-06-03 09:39:11 +02:00
Thibault Saunier
124153fe06 validateflow: Allow specifying checksum type
And add an extra mode 'checksum-as-id' which basically numerate
buffers checksums as they are being received so that it is simpler
to compare expectations when you are tracking buffers from both
sinkpads and srcpads.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/202>
2020-05-31 17:17:22 -04:00
Thibault Saunier
6a305b1e23 validate: Add a parameter to force waiting on the test clock
This allows to wait for a new buffer to reach the sink without
actually cranking that buffer, allowing to quite the test without
waiting for EOS in a 100% reproducible way

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/201>
2020-05-29 21:12:53 -04:00
Thibault Saunier
774a4a8704 validate: Error out when a config hasn't been used at all
It probably means a plugin is not available

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/201>
2020-05-29 18:16:23 -04:00
Tim-Philipp Müller
cd4836951e Ship validate as part of a gst-devtools tarball
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/199>
2020-05-29 14:30:26 +00:00
Tim-Philipp Müller
5a8fcfa663 meson: make debug_viewer a feature option
... and disable by default.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/199>
2020-05-29 14:30:26 +00:00
Thibault Saunier
ca8cdfd736 validate: Fix mixup in quarks usages
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/200>
2020-05-28 17:17:10 -04:00
Thibault Saunier
0f1dffdcc0 validate: Add missing return func when GLib < 2.50
CID 1463854

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/200>
2020-05-28 17:17:10 -04:00
Thibault Saunier
012288910b validate: Update documentation
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/200>
2020-05-28 17:17:10 -04:00
Thibault Saunier
b1cf1ffebd validateflow: Add a way to configure when to generate expectations
By default, generate them whenever the file is missing but adding a way
to override that with `validateflow,generate-expectations=true` to force
regenerating them or setting `validateflow,generate-expectations=false`
to disallow generating them (on CI servers for example)

Also update the validateflow documentation to take that into account
and remove references to pipeline.json file which is now gone!

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/200>
2020-05-28 17:17:10 -04:00
Thibault Saunier
3264de6751 scenario: Minor cleanup around pipeline change state management
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/198>
2020-05-27 19:20:16 +00:00
Thibault Saunier
e1ad840494 scenario: Mark seek as done only when reaching next state
There is a race where following actions could generate a
flush-start/flush-stop dance but the state change resulting from the
seek hasn't been committed yet, leading to the ASYNC_START being
ignored by GstBin since its pending_state is not VOID when receiving
the ASYNC_START message.

Conceptually it is totally correct to consider an action done when
the state change of the pipeline is stabilized..

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/198>
2020-05-27 19:20:16 +00:00
Thibault Saunier
e49876bb05 validate:launcher: Stop wrongly erroring on testsuite loading failure
When the testsuite was actually already loaded as the .py
file was explicitly passed in

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>
2020-05-26 19:16:27 -04:00
Thibault Saunier
9f11ba5937 validate: Fix loading configs from a caps
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>
2020-05-26 19:16:26 -04:00
Thibault Saunier
cf0f1df314 flow: Use bat to color diffs when possible
Adding a function to check if can output colored logs

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>
2020-05-26 19:16:26 -04:00
Thibault Saunier
91ac75aa4c launcher: check: Properly set PLUGIN_PATH and registry when running in gst-build
This makes registry

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>
2020-05-26 19:16:26 -04:00
Thibault Saunier
90cc65f7ee launcher: Ensure that -j tests run in parallel when running forever
So that you can reproduce the issue you want faster!

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>
2020-05-26 17:19:36 -04:00
Thibault Saunier
c9318639d7 docs: Update gst-validate-launcher documentation
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>
2020-05-26 17:19:36 -04:00
Thibault Saunier
766f339fb8 launcher: Add support for running tests inside rr
Allowing us to easily run the tests forever and then replay the
failures!

https://rr-project.org/

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>
2020-05-26 17:19:36 -04:00
Guillaume Desmottes
f78997ec2e validate: add missing gir annotation
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/194>
2020-05-25 14:42:01 +00:00
Edward Hervey
5e2df60422 validate-scenario: Initialize variable correctly
ret needs to be always reset to FALSE *before* checking attempting to load the
individual files. Otherwise there's the possibility it would silently accept an
invalid scenario name

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/196>
2020-05-22 18:00:04 +02:00
Edward Hervey
8697663d8c validate-scenario: Remove unused variable
We only need to check whether the field is present and of a given type

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/196>
2020-05-22 17:56:40 +02:00
Thibault Saunier
fa95de073b validate: Stop using g_file_peek_path
It was introduced in 2.56 so is too recent

Fixes https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/issues/51

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/195>
2020-05-15 11:28:10 -04:00
Thibault Saunier
0ffcacf325 validate: Add including support in the structure file parser
Adding proper error reporting support

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/191>
2020-05-11 17:03:49 -04:00
Thibault Saunier
d203f4251b validate: utils: Do not try to replace vars in debug info fields
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/190>
2020-05-08 20:21:16 -04:00
Thibault Saunier
dcb719d6cc validate: Fix parsing validate tests files with vars on windows
And ensure that we escape windows path in variables

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/190>
2020-05-08 20:21:16 -04:00
Thibault Saunier
884c45ff3e validate:launcher:check: Add support for gst-tester-1.0
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/189>
2020-05-07 09:39:16 -04:00
Thibault Saunier
b669bb0327 validate: Add support for known-issues in the .validatetest
And add some tests about remaining actions failures

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/189>
2020-05-07 09:36:03 -04:00
Thibault Saunier
da390689c9 validate: Ensure a meta structure is found in test files
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/189>
2020-05-07 00:31:45 -04:00
Thibault Saunier
619efa56a7 validate:scenario: Check remaining action on stop, not EOS
When ignoring EOS, on addition action could end up not being executed
and no error was reported which was wrong.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/189>
2020-05-07 00:31:45 -04:00
Thibault Saunier
e00ed70925 validate: Fix rendering destination directory path creation
We were ending up creating file:/some/path in cwd

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187>
2020-05-06 22:54:59 +00:00
Thibault Saunier
af3006dc12 validate: Plug some leaks
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187>
2020-05-06 22:54:59 +00:00
Thibault Saunier
bb0d81fd4c validate: Add details on all g_log message reports.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187>
2020-05-06 22:54:59 +00:00
Thibault Saunier
b65b2bc2fe validate: Add a mechanism to mark tests as skipped
And use it when a plugin is missing and the user didn't ask for
failure when it happens

And use the TAP[0] synthax to report it

[0]: https://testanything.org

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187>
2020-05-06 22:54:59 +00:00
Thibault Saunier
b1e9e409fd validate: Enhance issue reporting from structures
And properly bail out when required
This is preparatory work for gst-test in core which will use the TAP
protocol

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187>
2020-05-06 22:54:59 +00:00
Thibault Saunier
a1a2efc970 validate:flow: Handle some more segment fields filtering
Those slept through when implementing filtering

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/188>
2020-05-05 23:09:31 -04:00
Thibault Saunier
04a648e345 ssim: Minor improvements on the way we attach to pads
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>
2020-04-30 12:40:13 -04:00
Edward Hervey
fb71bf8fab validate-scenario: Refactor seek handling
* Store all seek values into a list of pending seeks instead
  of hardcoding some values
* Store all segments that sinks received
* Match segments to seeks when all sinks received segments with
  the same seqnum
* Detect when a seek did *not* result in segments with identical
  matching seqnums

Should allow checking for all types of seek handling, including
flush-less seeks

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>
2020-04-30 12:40:12 -04:00
Edward Hervey
5dafe4bb64 validate-report: Add new check for top-level seek
This issue is to detect seeks that don't result in segments
with identical seqnums. This check can be done at the top-level
scenario

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>
2020-04-30 12:40:12 -04:00
Edward Hervey
e05d4ac8f5 validate-pad-monitor: Post GstBaseSink SEGMENT on the bus
Allows higher-level bin or app (like validate-scenario) to know
what each sink currently has in terms of SEGMENT.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>
2020-04-30 12:40:12 -04:00