Commit graph

2443 commits

Author SHA1 Message Date
Thibault Saunier 5a2e2d5001 validateflow: Print some indication that the flow checking is happening 2019-02-11 08:34:36 -03:00
Edward Hervey 4467c9925d validate: Add autotools support for flow plugin 2019-02-11 11:09:46 +01:00
Thibault Saunier 3d3264b342 Revert "validate:launcher: Cache the result of meson introspect"
This reverts commit 05ce6d3b92.

We can't do that as it breaks meson logic to set envvars
2019-02-10 01:25:52 -03:00
Thibault Saunier 5a6307f913 validate:launcher: Do not sort tests all the time
Do it once only once it is fully populated
2019-02-09 17:25:03 -03:00
Thibault Saunier 05ce6d3b92 validate:launcher: Cache the result of meson introspect
Running it takes quite some time and we can easily cache it.
2019-02-09 17:24:13 -03:00
Thibault Saunier a88156a416 validate:launcher: Add a simple way to profile app 2019-02-09 17:23:28 -03:00
Guillaume Desmottes 0e03b38cc2 validate: scenario: not need to use an atomic to handle dropped count
It's all handled from the same thread.
2019-02-08 14:39:22 +01:00
Guillaume Desmottes 78a8306955 validate: scenario: fix dropped checking when terminating scenario
We want to early return if either no max value has been set for the
scenario or if we didn't receive any QoS information.
2019-02-08 14:25:52 +01:00
Guillaume Desmottes 916bf5cdf1 validate: properly namespace config related checks 2019-02-08 13:32:12 +01:00
Alicia Boya García e96f2ca714 New validate plugin: validateflow
validateflow can be used to check the buffers and events flowing through
a custom pipeline match an expectation file. This can be used to test
non-regular-playback use cases like demuxers handling adaptive streaming
fragment pushing.

This patch includes also new actions used for these cases:
`appsrc-push`, `appsrc-eos` and `flush` (plus `checkpoint`, which is
only available with validateflow).
2019-02-07 18:19:22 +00:00
Guillaume Desmottes 0c83ff56dd baseclass: add_validate_config: don't use self.proc_env
self.proc_env is created when starting the test but this API can be call
by generator when creating the test.
2019-02-07 17:13:02 +01:00
Guillaume Desmottes 921a073bdf validate: allow scenarios to define a max nb of dropped buffers
The 'max-dropped' description field can now be used to specify the max
number of buffers than can be dropped by the QoS system.
2019-02-07 14:12:59 +00:00
Thibault Saunier e8162cc957 validate:launcher: Error out in the check testsuite if rebuilding failed 2019-02-06 21:14:13 +00:00
Guillaume Desmottes 2d8ef6e81c validate: allow scenarios to define max pipeline latency
The 'max-latency' description field can now be used to specify the max
latency allowed for the running pipeline.
2019-02-06 16:07:02 +01:00
Mathieu Duponchelle 90c2d32030 Scenarios: override seek flags for RTSP tests
Our RTSP server is not accurate, it makes no sense to perform
accuracy checks on the client-side segments.
2019-02-02 02:08:58 +01:00
Thibault Saunier 49102316df validate:scenario: Set variables on all action fields 2019-02-01 23:46:21 +00:00
Thibault Saunier e50d41420e validate: Handle G_LOG_ERROR in our glog handler 2019-02-01 23:46:21 +00:00
Thibault Saunier 2f957ab932 validate:scenario: Rename action 'define-consts' to 'set-vars'
Those are not consts are they can be modified at runtime
2019-02-01 23:46:21 +00:00
Thibault Saunier d0ee26e950 validate: Cleanup flags/enum_from_string 2019-02-01 23:46:21 +00:00
Mathieu Duponchelle f9190236af scenario: prioritize SCENARIOS_PATH when including 2019-02-01 23:08:16 +01:00
Thibault Saunier 71d2e3f418 validate: Let people know the testsuite starts running
Now that we do not print infos about successful tests when redirecting.
2019-02-01 01:24:33 -03:00
Thibault Saunier ba9b2c5ec5 validate: Keep trying to print stack traces when our log hanlder is removed
There are cases where a crash happens after the program ends
2019-02-01 00:08:45 -03:00
Thibault Saunier 68573ae919 validate:scenario: Give pipeline position when failling on EOS 2019-01-31 23:47:11 -03:00
Thibault Saunier 35003bc25b validate:launcher: Do not print passing tests if not running in a tty 2019-01-31 23:47:11 -03:00
Thibault Saunier 25e0694d1e validate:scenario: Explicitely mark generated stop() action as such 2019-01-31 22:37:17 -03:00
Thibault Saunier 78781de913 validate:launcher:punittest: Raise an exception if a testsuite can't be loaded 2019-01-30 15:57:13 -03:00
Edward Hervey d1319a1b6d pad-monitor: Remove unused pad getrange override 2019-01-30 14:30:52 +01:00
Edward Hervey de59b3ad00 validate: Use g_object_{get|set}_qdata where applicable
This provides a substantial speedup compared to using strings
2019-01-30 14:29:48 +01:00
Mathieu Duponchelle d5e3f4d31c TestsManager: stop displaying blacklisted tests on stdout
It's basically spam, better suited to the debug logs
2019-01-30 00:27:32 +00:00
Thibault Saunier 440055214a validate:launcher: Never print lines larger than the terminal 2019-01-29 16:09:30 -03:00
Thibault Saunier 7e70bf8729 validate: scenario: Mark action as being executed right before calling ->execute
And make sure that we do not try to execute the following action

In the case the action type leads to the GMainContext to be ieterated
2019-01-29 13:09:36 -03:00
Jordan Petridis 31164cf0a6 validate: Report the full test name in the xunit file
Looks like gitlab prefers this way of representing tests as it
displays only the name field in its junit reports.

Close #32
2019-01-26 16:55:23 +00:00
Thibault Saunier 2551e8750e validate:launcher: Make baseclasses.py pep8 compliant 2019-01-26 15:54:00 +00:00
Thibault Saunier 9b69bcad08 validate:launcher: Handle launching launching a sub launcher
If you use validate-launcher in a meson testsuite, those test now
gets integrated as one unique testsuite (with a pretty long namespace).
2019-01-26 15:54:00 +00:00
Thibault Saunier 6665652cff validate:launcher: Refactor the "main" function
- Move the parser code into a `LauncherConfig.create_parser()` method
- Remove the need to pass libsdir to the _TestsLauncher object
- Extract out a `setup_launcher_from_args` function
2019-01-26 15:54:00 +00:00
Thibault Saunier 1b3867b82d launcher: Move http serveur and xvfb server to the main test runner object
No good reason for it to be in the main function
2019-01-26 15:54:00 +00:00
Thibault Saunier 3f4f815500 launcher: Cleanup the way we find python test command line
By setting it before the test base class adds the current testsuite name in the classname
2019-01-26 15:54:00 +00:00
Thibault Saunier 9cf213dee0 launcher: Fix test listing in meson VS gstcheck tests
You might select tests that match Meson but not gstcheck in which
case the 'meson only' variant is exposed but those should never exist.
2019-01-26 15:54:00 +00:00
Thibault Saunier a3bab7c66e validate:launcher: Add %(config_path)s in the pipeline desc vars
When defining pipelines_descriptions to run test on in a `.json` file, you might
need to point to paths in the testsuite directory (for media files URIs
for example), you can now do

 `"pipeline": "filesrc location="$(config_path)s/../medias/some/file.mkv...`
2019-01-26 10:11:24 -03:00
Tim-Philipp Müller 2dd8395ee8 Release 1.15.1 2019-01-17 10:01:50 +00:00
Tim-Philipp Müller ce007a7dcb Fix distcheck
Work around broken disthook check in release.mak so we don't
have to update the common submodules for that (applies only
to this module because the version number is in the top-level
meson.build but the package/dist directory is a subdir). This
only became a problem now because the common submodule hadn't
been updated for the last few years.
2019-01-17 10:00:46 +00:00
Tim-Philipp Müller 3c6769b9f5 win32: update .def file for new API
Fixes distcheck
2019-01-17 09:38:13 +00:00
Thibault Saunier 7befe3d033 validate:launcher: Display unix nickname of signals leading to test failure 2019-01-15 16:55:32 -03:00
Thibault Saunier 5692f8d74d validate:launcher: Fix error message about 'crashed' test 2019-01-15 16:05:41 -03:00
Alicia Boya García 438393293e pre-commit-python: Allow line breaks between binary operators
pre-commit-python overrides the list of ignored Python style errors.
Unfortunately, before this patch the list did not exclude W503 and
W504 (which are otherwise ignored by default).

The consequence of having those two warnings enabled at the same time is
that it's not possible to break lines on binary operators, which is an
unreasonable unintentional restriction:

  'validateflow': "validateflow, expectations-dir=\"" +
                  expectations_dir + "\", actual-results-dir=\"" +
                  actual_results_dir + "\"",

W504 line break after binary operator

  'validateflow': "validateflow, expectations-dir=\""
                  + expectations_dir + "\", actual-results-dir=\""
                  + actual_results_dir + "\"",

W503 line break before binary operator

This patch excludes W503 so that there is a valid style for breaking
lines on binary operators.
2019-01-15 18:57:02 +01:00
Thibault Saunier 92483ae210 validate: Plug newly introduced leak 2019-01-12 15:25:53 -03:00
Thibault Saunier 31c1a7ac63 validate: Use filename instead of full path in dotfiles names 2019-01-12 10:00:42 -03:00
Thibault Saunier 27f1712671 validate:tests: Fix race in validate_padmonitor.buffer_outside_segment
We were using a fakesrc which data flow was potentially breaking the
test.
2019-01-07 00:07:34 +01:00
Thibault Saunier 9d11ade78a validate: Lower a ERROR message to INFO
This behaviour is totally valid when running unit tests
2019-01-07 00:07:34 +01:00
Thibault Saunier 9843562616 validate:tests: s/discount_buffer/discont_buffer/ 2019-01-03 15:27:33 +00:00