validate: Introduce the concept of "Test files"

This way we can have a single file that wraps scenarios,
`gst-validate-1.0` arguments, as well as a configuration.

It changes the name of `description` of scenarios to use `meta`
The goal is to replace tests describes in python with dictionary
to fully self contained `.validatetest` files which look like:

```
meta,
    handles-states=true,
    ignore-eos=true,
    gst-validate-args = {
         "videotestsrc pattern=blue ! video/x-raw,format=I420,framerate=1/1 ! timeoverlay ! $(videosink) name=videosink allocation-meta-flags=0",
    },
    configs = {
         "$(validateflow), pad=videosink:sink, buffers-checksum=true, ignored-fields={\"buffers=meta\", }",
    }

play
seek, start=0.0, stop=5.0, flags=accurate+flush, rate=1.0
crank-clock, expected-elapsed-time=0.0
crank-clock, repeat=4, expected-elapsed-time=1.0
crank-clock, expected-elapsed-time=1.0
stop, on-message=eos
```
This commit is contained in:
Thibault Saunier 2020-04-24 15:41:10 -04:00
parent b7202e2f16
commit bf952d3c8b
12 changed files with 683 additions and 162 deletions

View file

@ -31,8 +31,8 @@ the \$GST\_VALIDATE\_SCENARIOS\_PATH environment variable.
Each line in the `.scenario` file represent an action (you can also use
`\ ` at the end of a line write a single action on multiple lines).
Usually you should start you scenario with a `description` "config"
action in order for the user to have more information about the
Usually you should start you scenario with a `meta` structure
in order for the user to have more information about the
scenario. It can contain a `summary` field which is a string explaining
what the scenario does and then several info fields about the scenario.
You can find more info about it running:
@ -43,7 +43,7 @@ So a basic scenario file that will seek three times and stop would look
like:
```
description, summary="Seeks at 1.0 to 2.0 then at \
meta, summary="Seeks at 1.0 to 2.0 then at \
3.0 to 0.0 and then seeks at \
1.0 to 2.0 for 1.0 second (between 2.0 and 3.0).", \
seek=true, duration=5.0, min-media-duration=4.0

View file

@ -0,0 +1,86 @@
---
title: Test file
short-description: GstValidate test file
...
# GstValidate Test file
A `.validatetest` file describes a fully contained validate test case. It
includes the arguments of the tool supposed to be used to run the test as well
as possibly a [configuration](gst-validate-config.md) and a set of action to
describe the validate [scenario](gst-validate-scenarios.md).
# The file format
A validate test file requires a `meta` structure which contains the same
information as the [scenario](gst-validate-scenarios.md) `meta` with some
additional fields described below. The `meta` structure should be either the
first or the one following the `set-globals` structure. The `set-globals`
structures allows you to set global variables for the rest of the
`.validatetest` file and is a free form variables setter. For example you can
do:
``` yaml
set-globals, media_dir=$(test_dir)/../../media
```
The `meta` format:
## Tool arguments
In the case of [`gst-validate`](gst-validate.md) it **has to** contain a
`gst-validate-args` field with `gst-validate` argv arguments like:
``` yaml
# This is the default tool so it is not mandatory for the `gst-validate` tool
tool = "gst-validate-$(gst_api_version)",
args = {
# pipeline description
videotestrc num-buffers=2 ! $(videosink),
# Random extra argument
--set-media-info $(test-dir)/some.media_info
}
```
## configs
The `config` field is an array of string containing the same content as
usual [config](gst-validate-config.md) files contain.
For example:
``` json
configs = {
# Set videotestsrc0 pattern value to `blue`
"core, action=set-property, target-element-name=videotestsrc0, property-name=pattern, property-value=blue",
"$(validateflow), pad=sink1:sink, caps-properties={ width, height };",
}
```
# Variables
The same way
Validate testfile will define some variables to make those files relocable:
* `$(test_dir)`: The directory where the `.validatetest` file is in.
* `$(test_name)`: The name of the test file (without extension).
* `$(test_name_dir)`: The name of the test directory (test_name with folder
separator instead of `.`).
* `$(validateflow)`: The validateflow structure name with the default/right
values for the `expectations-dir` and `actual-results-dir`
fields. See [validateflow](plugins/validateflow.md) for more
information.
* `$(videosink)`: The GStreamer videosink to use if the test can work with
different sinks for the video. It allows the tool to use
fakesinks when the user doesn't want to have visual feedback
for example.
* `$(audiosink)`: The GStreamer audiosink to use if the test can work with
different sinks for the audio. It allows the tool to use
fakesinks when the user doesn't want to have audio feedback
for example.

View file

@ -4,6 +4,7 @@ index.md
gst-validate-media-check.md
gst-validate-launcher.md
gst-validate-scenarios.md
gst-validate-test-file.md
gst-validate-config.md
gst-validate-environment-variables.md
gst-validate-action-types.md

View file

@ -41,12 +41,14 @@ extern G_GNUC_INTERNAL GQuark _Q_VALIDATE_MONITOR;
extern G_GNUC_INTERNAL GType _gst_validate_action_type_type;
void init_scenarios (void);
void register_action_types (void);
/* FIXME 2.0 Remove that as this is only for backward compatibility
* as we used to have to print actions in the action execution function
* and this is done by the scenario itself now */
G_GNUC_INTERNAL gboolean _action_check_and_set_printed (GstValidateAction *action);
G_GNUC_INTERNAL gboolean gst_validate_action_is_subaction (GstValidateAction *action);
G_GNUC_INTERNAL gboolean gst_validate_scenario_check_and_set_needs_clock_sync (GList *structures, GstStructure **meta);
G_GNUC_INTERNAL void _priv_validate_override_registry_deinit (void);
G_GNUC_INTERNAL GstValidateReportingDetails gst_validate_runner_get_default_reporting_details (GstValidateRunner *runner);
@ -56,4 +58,8 @@ G_GNUC_INTERNAL void gst_validate_init_runner (void);
G_GNUC_INTERNAL void gst_validate_deinit_runner (void);
G_GNUC_INTERNAL void gst_validate_report_deinit (void);
G_GNUC_INTERNAL gboolean gst_validate_send (JsonNode * root);
G_GNUC_INTERNAL void gst_validate_set_test_file_globals (GstStructure* meta, const gchar* testfile, gboolean use_fakesinks);
G_GNUC_INTERNAL gboolean gst_validate_get_test_file_scenario (GList** structs, const gchar** scenario_name, gchar** original_name);
G_GNUC_INTERNAL GstValidateScenario* gst_validate_scenario_from_structs (GstValidateRunner* runner, GstElement* pipeline, GList* structures,
gchar* origin_file);
#endif

View file

@ -755,13 +755,29 @@ static void
gst_validate_pipeline_monitor_create_scenarios (GstValidateBinMonitor * monitor)
{
/* scenarios currently only make sense for pipelines */
const gchar *scenarios_names;
gchar **scenarios = NULL;
const gchar *scenarios_names, *scenario_name = NULL;
gchar **scenarios = NULL, *testfile = NULL;
GstObject *target =
gst_validate_monitor_get_target (GST_VALIDATE_MONITOR (monitor));
GstValidateRunner *runner =
gst_validate_reporter_get_runner (GST_VALIDATE_REPORTER (monitor));
GList *scenario_structs = NULL;
if (gst_validate_get_test_file_scenario (&scenario_structs, &scenario_name,
&testfile)) {
if (scenario_name) {
monitor->scenario =
gst_validate_scenario_factory_create (runner,
GST_ELEMENT_CAST (target), scenario_name);
goto done;
}
monitor->scenario =
gst_validate_scenario_from_structs (runner,
GST_ELEMENT_CAST (target), scenario_structs, testfile);
goto done;
}
if ((scenarios_names = g_getenv ("GST_VALIDATE_SCENARIO"))) {
gint i;

View file

@ -52,6 +52,7 @@
#include "gst-validate-reporter.h"
#include "gst-validate-report.h"
#include "gst-validate-utils.h"
#include "gst-validate-internal.h"
#include "validate.h"
#include <gst/validate/gst-validate-override.h>
#include <gst/validate/gst-validate-override-registry.h>
@ -3438,20 +3439,20 @@ _action_type_has_parameter (GstValidateActionType * atype,
}
static gboolean
_load_scenario_file (GstValidateScenario * scenario,
const gchar * scenario_file, gboolean * is_config)
gst_validate_scenario_load_structures (GstValidateScenario * scenario,
GList * structures, gboolean * is_config, gchar * origin_file)
{
gboolean ret = TRUE;
GList *structures, *tmp;
GList *tmp;
GstValidateScenarioPrivate *priv = scenario->priv;
GList *config;
*is_config = FALSE;
structures =
gst_validate_utils_structs_parse_from_filename (scenario_file, NULL);
if (structures == NULL)
goto failed;
if (!structures) {
GST_INFO_OBJECT (scenario, "No structures provided");
return FALSE;
}
for (tmp = structures; tmp; tmp = tmp->next) {
GstValidateAction *action;
@ -3460,7 +3461,7 @@ _load_scenario_file (GstValidateScenario * scenario,
GstStructure *structure = (GstStructure *) tmp->data;
type = gst_structure_get_name (structure);
if (!g_strcmp0 (type, "description")) {
if (!g_strcmp0 (type, "description") || !g_strcmp0 (type, "meta")) {
const gchar *pipeline_name;
gst_structure_get_boolean (structure, "is-config", is_config);
@ -3494,7 +3495,7 @@ _load_scenario_file (GstValidateScenario * scenario,
goto failed;
}
if (!gst_validate_scenario_load (scenario, location, scenario_file)) {
if (!gst_validate_scenario_load (scenario, location, origin_file)) {
GST_ERROR ("Failed including scenario %s", location);
goto failed;
}
@ -3527,8 +3528,12 @@ _load_scenario_file (GstValidateScenario * scenario,
}
action = gst_validate_action_new (scenario, action_type, structure, TRUE);
if (action->priv->state == GST_VALIDATE_EXECUTE_ACTION_ERROR)
if (action->priv->state == GST_VALIDATE_EXECUTE_ACTION_ERROR) {
GST_ERROR_OBJECT (scenario, "Newly created action: %" GST_PTR_FORMAT
" was in error state", structure);
goto failed;
}
action->action_number = priv->num_actions++;
}
@ -3557,6 +3562,16 @@ failed:
goto done;
}
static gboolean
_load_scenario_file (GstValidateScenario * scenario,
gchar * scenario_file, gboolean * is_config)
{
return gst_validate_scenario_load_structures (scenario,
gst_validate_utils_structs_parse_from_filename (scenario_file, NULL),
is_config, scenario_file);
}
static gboolean
gst_validate_scenario_load (GstValidateScenario * scenario,
const gchar * scenario_name, const gchar * relative_scenario)
@ -3958,28 +3973,27 @@ _element_added_cb (GstBin * bin, GstElement * element,
}
}
/**
* gst_validate_scenario_factory_create:
* @runner: The #GstValidateRunner to use to report issues
* @pipeline: The pipeline to run the scenario on
* @scenario_name: The name (or path) of the scenario to run
*
* Returns: (transfer full): A #GstValidateScenario or NULL
*/
GstValidateScenario *
gst_validate_scenario_factory_create (GstValidateRunner *
runner, GstElement * pipeline, const gchar * scenario_name)
static GstValidateScenario *
gst_validate_scenario_new (GstValidateRunner *
runner, GstElement * pipeline, gchar * scenario_name, GList * structures)
{
GList *config;
GstValidateScenario *scenario =
g_object_new (GST_TYPE_VALIDATE_SCENARIO, "validate-runner",
runner, NULL);
GST_LOG ("Creating scenario %s", scenario_name);
if (!gst_validate_scenario_load (scenario, scenario_name, NULL)) {
g_object_unref (scenario);
if (structures) {
gboolean is_config;
gst_validate_scenario_load_structures (scenario, structures, &is_config,
scenario_name);
} else {
return NULL;
GST_LOG ("Creating scenario %s", scenario_name);
if (!gst_validate_scenario_load (scenario, scenario_name, NULL)) {
g_object_unref (scenario);
return NULL;
}
}
if (scenario->priv->pipeline_name &&
@ -3994,6 +4008,10 @@ gst_validate_scenario_factory_create (GstValidateRunner *
return NULL;
}
gst_validate_printf (NULL,
"\n**-> Running scenario %s on pipeline %s**\n\n", scenario_name,
GST_OBJECT_NAME (pipeline));
g_weak_ref_init (&scenario->priv->ref_pipeline, pipeline);
gst_validate_reporter_set_name (GST_VALIDATE_REPORTER (scenario),
g_strdup (scenario_name));
@ -4038,10 +4056,6 @@ gst_validate_scenario_factory_create (GstValidateRunner *
_add_execute_actions_gsource (scenario);
}
gst_validate_printf (NULL,
"\n**-> Running scenario %s on pipeline %s**\n\n", scenario_name,
GST_OBJECT_NAME (pipeline));
scenario->priv->overrides =
gst_validate_override_registry_get_override_for_names
(gst_validate_override_registry_get (), "scenarios", NULL);
@ -4049,6 +4063,31 @@ gst_validate_scenario_factory_create (GstValidateRunner *
return scenario;
}
GstValidateScenario *
gst_validate_scenario_from_structs (GstValidateRunner * runner,
GstElement * pipeline, GList * structures, gchar * origin_file)
{
g_return_val_if_fail (structures, NULL);
return gst_validate_scenario_new (runner, pipeline, origin_file, structures);
}
/**
* gst_validate_scenario_factory_create:
* @runner: The #GstValidateRunner to use to report issues
* @pipeline: The pipeline to run the scenario on
* @scenario_name: The name (or path) of the scenario to run
*
* Returns: (transfer full): A #GstValidateScenario or NULL
*/
GstValidateScenario *
gst_validate_scenario_factory_create (GstValidateRunner *
runner, GstElement * pipeline, const gchar * scenario_name)
{
return gst_validate_scenario_new (runner, pipeline, (gchar *) scenario_name,
NULL);
}
static gboolean
_add_description (GQuark field_id, const GValue * value, KeyFileGroupName * kfg)
{
@ -4064,6 +4103,41 @@ _add_description (GQuark field_id, const GValue * value, KeyFileGroupName * kfg)
return TRUE;
}
gboolean
gst_validate_scenario_check_and_set_needs_clock_sync (GList * structures,
GstStructure ** meta)
{
gboolean needs_clock_sync = FALSE;
GList *tmp;
for (tmp = structures; tmp; tmp = tmp->next) {
GstStructure *_struct = (GstStructure *) tmp->data;
gboolean is_meta = gst_structure_has_name (_struct, "description")
|| gst_structure_has_name (_struct, "meta");
if (!is_meta) {
GstValidateActionType *type =
_find_action_type (gst_structure_get_name (_struct));
if (type && type->flags & GST_VALIDATE_ACTION_TYPE_NEEDS_CLOCK)
needs_clock_sync = TRUE;
continue;
}
if (!*meta)
*meta = gst_structure_copy (_struct);
}
if (needs_clock_sync) {
if (*meta)
gst_structure_set (*meta, "need-clock-sync", G_TYPE_BOOLEAN, TRUE, NULL);
else
*meta = gst_structure_from_string ("description, need-clock-sync=true;",
NULL);
}
return needs_clock_sync;
}
static gboolean
_parse_scenario (GFile * f, GKeyFile * kf)
@ -4072,40 +4146,23 @@ _parse_scenario (GFile * f, GKeyFile * kf)
gchar *path = g_file_get_path (f);
if (g_str_has_suffix (path, GST_VALIDATE_SCENARIO_SUFFIX)) {
gboolean needs_clock_sync = FALSE;
GstStructure *desc = NULL;
GstStructure *meta = NULL;
GList *tmp, *structures = gst_validate_structs_parse_from_gfile (f);
for (tmp = structures; tmp; tmp = tmp->next) {
GstStructure *_struct = (GstStructure *) tmp->data;
GstValidateActionType *type =
_find_action_type (gst_structure_get_name (_struct));
gst_validate_scenario_check_and_set_needs_clock_sync (structures, &meta);
for (tmp = structures; tmp; tmp = tmp->next)
gst_structure_remove_fields (tmp->data, "__lineno__", "__filename__",
NULL);
gst_structure_remove_fields (_struct, "__lineno__", "__filename__", NULL);
if (!desc && gst_structure_has_name (_struct, "description"))
desc = gst_structure_copy (_struct);
else if (type && type->flags & GST_VALIDATE_ACTION_TYPE_NEEDS_CLOCK)
needs_clock_sync = TRUE;
}
if (needs_clock_sync) {
if (desc)
gst_structure_set (desc, "need-clock-sync", G_TYPE_BOOLEAN, TRUE, NULL);
else
desc = gst_structure_from_string ("description, need-clock-sync=true;",
NULL);
}
if (desc) {
if (meta) {
KeyFileGroupName kfg;
kfg.group_name = g_file_get_path (f);
kfg.kf = kf;
gst_structure_foreach (desc,
gst_structure_foreach (meta,
(GstStructureForeachFunc) _add_description, &kfg);
gst_structure_free (desc);
gst_structure_free (meta);
} else {
g_key_file_set_string (kf, path, "noinfo", "nothing");
}
@ -4293,7 +4350,7 @@ check_last_sample_internal (GstValidateScenario * scenario,
&iframe_number)) {
GST_VALIDATE_REPORT_ACTION (scenario, action,
SCENARIO_ACTION_EXECUTION_ERROR,
"The 'checksum' or 'time-code-frame-number' parametters of the "
"The 'checksum' or 'time-code-frame-number' parameters of the "
"`check-last-sample` action type needs to be specified, none found");
res = GST_VALIDATE_EXECUTE_ACTION_ERROR_REPORTED;
@ -4306,7 +4363,7 @@ check_last_sample_internal (GstValidateScenario * scenario,
tc_meta = gst_buffer_get_video_time_code_meta (buffer);
if (!tc_meta) {
GST_VALIDATE_REPORT (scenario, SCENARIO_ACTION_EXECUTION_ERROR,
"Could not \"check-last-sample\" as the buffer doesn' contain a TimeCode"
"Could not \"check-last-sample\" as the buffer doesn't contain a TimeCode"
" meta");
res = GST_VALIDATE_EXECUTE_ACTION_ERROR_REPORTED;
goto done;
@ -5085,6 +5142,50 @@ void
init_scenarios (void)
{
GList *tmp;
register_action_types ();
for (tmp = gst_validate_plugin_get_config (NULL); tmp; tmp = tmp->next) {
const gchar *action_typename;
GstStructure *plug_conf = (GstStructure *) tmp->data;
if ((action_typename = gst_structure_get_string (plug_conf, "action"))) {
GstValidateAction *action;
GstValidateActionType *atype = _find_action_type (action_typename);
if (!atype) {
g_error ("[CONFIG ERROR] Action type %s not found", action_typename);
continue;
}
if (atype->flags & GST_VALIDATE_ACTION_TYPE_HANDLED_IN_CONFIG) {
GST_INFO ("Action type %s from configuration files"
" is handled.", action_typename);
continue;
}
if (!(atype->flags & GST_VALIDATE_ACTION_TYPE_CONFIG) &&
!(_action_type_has_parameter (atype, "as-config"))) {
g_error ("[CONFIG ERROR] Action '%s' is not a config action",
action_typename);
continue;
}
gst_structure_set (plug_conf, "as-config", G_TYPE_BOOLEAN, TRUE, NULL);
gst_structure_set_name (plug_conf, action_typename);
action = gst_validate_action_new (NULL, atype, plug_conf, FALSE);
gst_validate_action_unref (action);
}
}
}
void
register_action_types (void)
{
GST_DEBUG_CATEGORY_INIT (gst_validate_scenario_debug, "gstvalidatescenario",
GST_DEBUG_FG_YELLOW, "Gst validate scenarios");
@ -5092,7 +5193,7 @@ init_scenarios (void)
_gst_validate_action_type_type = gst_validate_action_type_get_type ();
/* *INDENT-OFF* */
REGISTER_ACTION_TYPE ("description", NULL,
REGISTER_ACTION_TYPE ("meta", NULL,
((GstValidateActionParameter []) {
{
.name = "summary",
@ -5218,7 +5319,7 @@ init_scenarios (void)
},
{NULL}
}),
"Allows to describe the scenario in various ways",
"Scenario metadata.\nNOTE: it used to be called \"description\"",
GST_VALIDATE_ACTION_TYPE_CONFIG);
REGISTER_ACTION_TYPE ("seek", _execute_seek,
@ -5798,43 +5899,6 @@ init_scenarios (void)
}),
"Request a video key unit", FALSE);
/* *INDENT-ON* */
for (tmp = gst_validate_plugin_get_config (NULL); tmp; tmp = tmp->next) {
const gchar *action_typename;
GstStructure *plug_conf = (GstStructure *) tmp->data;
if ((action_typename = gst_structure_get_string (plug_conf, "action"))) {
GstValidateAction *action;
GstValidateActionType *atype = _find_action_type (action_typename);
if (!atype) {
g_error ("[CONFIG ERROR] Action type %s not found", action_typename);
continue;
}
if (atype->flags & GST_VALIDATE_ACTION_TYPE_HANDLED_IN_CONFIG) {
GST_INFO ("Action type %s from configuration files"
" is handled.", action_typename);
continue;
}
if (!(atype->flags & GST_VALIDATE_ACTION_TYPE_CONFIG) &&
!(_action_type_has_parameter (atype, "as-config"))) {
g_error ("[CONFIG ERROR] Action '%s' is not a config action",
action_typename);
continue;
}
gst_structure_set (plug_conf, "as-config", G_TYPE_BOOLEAN, TRUE, NULL);
gst_structure_set_name (plug_conf, action_typename);
action = gst_validate_action_new (NULL, atype, plug_conf, FALSE);
gst_validate_action_unref (action);
}
}
}
void

View file

@ -39,6 +39,7 @@
#endif
#include "gst-validate-utils.h"
#include "gst-validate-internal.h"
#include <gst/gst.h>
#define PARSER_BOOLEAN_EQUALITY_THRESHOLD (1e-10)
@ -1035,7 +1036,6 @@ gst_validate_replace_variables_in_string (GstStructure * local_vars,
{
gint varname_len;
GMatchInfo *match_info = NULL;
const gchar *var_value = NULL;
gchar *tmpstring, *string = g_strdup (in_string);
if (!_variables_regex)
@ -1044,6 +1044,8 @@ gst_validate_replace_variables_in_string (GstStructure * local_vars,
gst_validate_set_globals (NULL);
while (g_regex_match (_variables_regex, string, 0, &match_info)) {
const gchar *var_value = NULL;
if (g_match_info_matches (match_info)) {
GRegex *replace_regex;
gchar *tmp, *varname, *pvarname = g_match_info_fetch (match_info, 0);
@ -1101,6 +1103,16 @@ _structure_set_variables (GQuark field_id, GValue * value,
{
gchar *str;
if (GST_VALUE_HOLDS_LIST (value)) {
gint i;
for (i = 0; i < gst_value_list_get_size (value); i++)
_structure_set_variables (0, (GValue *) gst_value_list_get_value (value,
i), local_variables);
return TRUE;
}
if (!G_VALUE_HOLDS_STRING (value))
return TRUE;
@ -1140,8 +1152,11 @@ gst_validate_set_globals (GstStructure * structure)
logsdir = g_get_tmp_dir ();
global_vars =
gst_structure_new ("vars", "TMPDIR", G_TYPE_STRING, g_get_tmp_dir (),
"LOGSDIR", G_TYPE_STRING, logsdir, NULL);
gst_structure_new ("vars",
"TMPDIR", G_TYPE_STRING, g_get_tmp_dir (),
"LOGSDIR", G_TYPE_STRING, logsdir,
"tmpdir", G_TYPE_STRING, g_get_tmp_dir (),
"logsdir", G_TYPE_STRING, logsdir, NULL);
}
if (!structure)
@ -1190,5 +1205,95 @@ gst_validate_utils_get_strv (GstStructure * str, const gchar * fieldname)
parsed_list[i] = g_value_dup_string (gst_value_list_get_value (value, i));
parsed_list[i] = NULL;
return parsed_list;
}
static void
strip_ext (char *fname)
{
char *end = fname + strlen (fname);
while (end > fname && *end != '.')
--end;
if (end > fname)
*end = '\0';
}
/* NOTE: vars == NULL implies that we are working on a testfile and the variables
* will be set globally */
void
gst_validate_structure_set_variables_from_struct_file (GstStructure * vars,
const gchar * struct_file)
{
gchar *config_dir;
gchar *config_fname;
gchar *config_name;
gchar *t, *config_name_dir;
gchar *validateflow, *expectations_dir, *actual_result_dir;
const gchar *logdir;
if (!struct_file)
return;
config_dir = g_path_get_dirname (struct_file);
config_fname = g_path_get_basename (struct_file);
config_name = g_strdup (config_fname);
gst_validate_set_globals (NULL);
logdir = gst_structure_get_string (global_vars, "logsdir");
g_assert (logdir);
strip_ext (config_name);
config_name_dir = g_strdup (config_name);
for (t = config_name_dir; *t != '\0'; t++) {
if (*t == '.')
*t = '/';
}
expectations_dir =
g_build_filename (config_dir, config_name, "flow-expectations", NULL);
actual_result_dir = g_build_filename (logdir, config_name_dir, NULL);
validateflow =
g_strdup_printf
("validateflow, expectations-dir=\"%s\", actual-results-dir=\"%s\"",
expectations_dir, actual_result_dir);
gst_structure_set (!vars ? global_vars : vars,
"gst_api_version", G_TYPE_STRING, GST_API_VERSION,
!vars ? "test_dir" : "CONFIG_DIR", G_TYPE_STRING, config_dir,
!vars ? "test_name" : "CONFIG_NAME", G_TYPE_STRING, config_name,
!vars ? "test_name_dir" : "CONFIG_NAME_DIR", G_TYPE_STRING,
config_name_dir, !vars ? "test_path" : "CONFIG_PATH", G_TYPE_STRING,
struct_file, "validateflow", G_TYPE_STRING, validateflow, NULL);
g_free (config_dir);
g_free (config_name_dir);
g_free (config_fname);
g_free (config_name);
g_free (validateflow);
g_free (actual_result_dir);
g_free (expectations_dir);
}
void
gst_validate_set_test_file_globals (GstStructure * meta, const gchar * testfile,
gboolean use_fakesinks)
{
gboolean needs_sync = FALSE;
const gchar *videosink, *audiosink;
if (!use_fakesinks) {
videosink = "autovideosink";
audiosink = "autoaudiosink";
} else if (gst_structure_get_boolean (meta, "need-clock-sync", &needs_sync)
&& needs_sync) {
videosink = "fakevideosink qos=true max-lateness=20000000";
audiosink = "fakesink sync=true";
} else {
videosink = "fakevideosink sync=false";
audiosink = "fakesink";
}
gst_structure_set (global_vars,
"videosink", G_TYPE_STRING, videosink,
"audiosink", G_TYPE_STRING, audiosink, NULL);
}

View file

@ -52,6 +52,8 @@ GST_VALIDATE_API
GList * gst_validate_utils_structs_parse_from_filename (const gchar * scenario_file,
gchar **file_path);
GST_VALIDATE_API
GstStructure * gst_validate_utils_test_file_get_meta (const gchar * testfile, gboolean use_fakesinks);
GST_VALIDATE_API
GList * gst_validate_structs_parse_from_gfile (GFile * scenario_file);
GST_VALIDATE_API
@ -75,6 +77,7 @@ gboolean gst_validate_element_matches_target (GstElement * element, GstStructure
gchar * gst_validate_replace_variables_in_string (GstStructure * local_vars, const gchar * in_string);
GST_VALIDATE_API
void gst_validate_structure_resolve_variables (GstStructure *structure, GstStructure *local_variables);
void gst_validate_set_globals (GstStructure *structure);
void gst_validate_structure_set_variables_from_struct_file(GstStructure* vars, const gchar* struct_file);
void gst_validate_set_globals(GstStructure* structure);
#endif

View file

@ -38,6 +38,8 @@
#include <sys/types.h>
#include <sys/stat.h>
#include <math.h>
#include "validate.h"
#include "gst-validate-utils.h"
#include "gst-validate-internal.h"
@ -54,6 +56,8 @@ static GMutex _gst_validate_registry_mutex;
static GstRegistry *_gst_validate_registry_default = NULL;
static GList *core_config = NULL;
static GList *testfile_structs = NULL;
static gchar *global_testfile = NULL;
static gboolean validate_initialized = FALSE;
static gboolean loaded_globals = FALSE;
GstClockTime _priv_start_time;
@ -134,11 +138,58 @@ _set_vars_func (GQuark field_id, const GValue * value, GstStructure * vars)
return TRUE;
}
static GstStructure *
get_test_file_meta (void)
{
GList *tmp;
for (tmp = testfile_structs; tmp; tmp = tmp->next) {
if (gst_structure_has_name (tmp->data, "meta"))
return tmp->data;
}
return NULL;
}
static GList *
get_config_from_structures (GList * structures, GstStructure * local_vars,
const gchar * suffix)
{
GList *tmp, *result = NULL;
for (tmp = structures; tmp; tmp = tmp->next) {
GstStructure *structure = tmp->data;
if (gst_structure_has_name (structure, suffix)) {
if (gst_structure_has_field (structure, "set-vars")) {
gst_structure_remove_field (structure, "set-vars");
if (!local_vars) {
GST_WARNING ("Unused `set-vars` config: %" GST_PTR_FORMAT, structure);
continue;
}
gst_structure_foreach (structure,
(GstStructureForeachFunc) _set_vars_func, local_vars);
} else {
gst_validate_structure_resolve_variables (structure, local_vars);
result = g_list_append (result, structure);
}
} else {
if (!loaded_globals && gst_structure_has_name (structure, "set-globals")) {
gst_validate_structure_resolve_variables (structure, local_vars);
gst_validate_set_globals (structure);
}
gst_structure_free (structure);
}
}
return result;
}
static GList *
create_config (const gchar * config, const gchar * suffix)
{
GstStructure *local_vars;
GList *structures = NULL, *tmp, *result = NULL;
GList *structures = NULL, *result = NULL;
gchar *config_file = NULL;
if (!suffix) {
@ -170,44 +221,11 @@ create_config (const gchar * config, const gchar * suffix)
}
}
if (config_file) {
gchar *config_dir = g_path_get_dirname (config_file);
gchar *config_fname = g_path_get_basename (config_file);
gchar **config_name =
g_regex_split_simple ("\\.config", config_fname, 0, 0);
gst_structure_set (local_vars,
"CONFIG_DIR", G_TYPE_STRING, config_dir,
"CONFIG_NAME", G_TYPE_STRING, config_name[0],
"CONFIG_PATH", G_TYPE_STRING, config_file, NULL);
g_free (config_dir);
g_free (config_fname);
g_strfreev (config_name);
}
gst_validate_structure_set_variables_from_struct_file (local_vars,
config_file);
g_free (config_file);
for (tmp = structures; tmp; tmp = tmp->next) {
GstStructure *structure = tmp->data;
if (gst_structure_has_name (structure, suffix)) {
if (gst_structure_has_field (structure, "set-vars")) {
gst_structure_remove_field (structure, "set-vars");
gst_structure_foreach (structure,
(GstStructureForeachFunc) _set_vars_func, local_vars);
} else {
gst_validate_structure_resolve_variables (structure, local_vars);
result = g_list_append (result, structure);
}
} else {
if (!loaded_globals && gst_structure_has_name (structure, "set-globals")) {
gst_validate_structure_resolve_variables (structure, local_vars);
gst_validate_set_globals (structure);
}
gst_structure_free (structure);
}
}
result = get_config_from_structures (structures, local_vars, suffix);
loaded_globals = TRUE;
g_list_free (structures);
@ -215,6 +233,48 @@ create_config (const gchar * config, const gchar * suffix)
return result;
}
static GList *
gst_validate_get_testfile_configs (const gchar * suffix)
{
GList *res = NULL;
gchar **config_strs = NULL, *filename = NULL;
gint current_lineno = -1;
GstStructure *meta = get_test_file_meta ();
if (!meta)
return NULL;
gst_structure_get (meta,
"__lineno__", G_TYPE_INT, &current_lineno,
"__filename__", G_TYPE_STRING, &filename, NULL);
config_strs = gst_validate_utils_get_strv (meta, "configs");
if (config_strs) {
gint i;
for (i = 0; config_strs[i]; i++) {
GstStructure *tmpstruct =
gst_structure_from_string (config_strs[i], NULL);
if (tmpstruct == NULL) {
g_error ("%s:%d: Invalid structure\n %d | %s\n %*c|",
filename, current_lineno, current_lineno, config_strs[i],
(gint) floor (log10 (abs ((current_lineno)))) + 1, ' ');
}
gst_structure_set (tmpstruct,
"__lineno__", G_TYPE_INT, current_lineno,
"__filename__", G_TYPE_STRING, filename, NULL);
res = g_list_append (res, tmpstruct);
}
}
g_free (filename);
g_strfreev (config_strs);
return get_config_from_structures (res, NULL, suffix);
}
/**
* gst_validate_plugin_get_config:
* @plugin: a #GstPlugin, or #NULL
@ -246,10 +306,10 @@ gst_validate_plugin_get_config (GstPlugin * plugin)
suffix = "core";
}
plugin_conf = gst_validate_get_testfile_configs (suffix);
config = g_getenv ("GST_VALIDATE_CONFIG");
if (!config) {
GST_DEBUG ("GST_VALIDATE_CONFIG not set");
return NULL;
return plugin_conf;
}
tmp = g_strsplit (config, G_SEARCHPATH_SEPARATOR_S, -1);
@ -335,6 +395,13 @@ gst_validate_init_plugins (void)
gst_registry_fork_set_enabled (TRUE);
}
void
gst_validate_init_debug (void)
{
GST_DEBUG_CATEGORY_INIT (gstvalidate_debug, "validate", 0,
"Validation library");
}
/**
* gst_validate_init:
*
@ -348,10 +415,7 @@ gst_validate_init (void)
if (validate_initialized) {
return;
}
GST_DEBUG_CATEGORY_INIT (gstvalidate_debug, "validate", 0,
"Validation library");
gst_validate_init_debug ();
_priv_start_time = gst_util_get_timestamp ();
_Q_VALIDATE_MONITOR = g_quark_from_static_string ("validate-monitor");
@ -383,6 +447,10 @@ gst_validate_deinit (void)
g_clear_object (&_gst_validate_registry_default);
g_list_free_full (testfile_structs, (GDestroyNotify) gst_structure_free);
testfile_structs = NULL;
g_clear_pointer (&global_testfile, g_free);
_priv_validate_override_registry_deinit ();
core_config = NULL;
validate_initialized = FALSE;
@ -397,3 +465,85 @@ gst_validate_is_initialized (void)
{
return validate_initialized;
}
gboolean
gst_validate_get_test_file_scenario (GList ** structs,
const gchar ** scenario_name, gchar ** original_name)
{
GList *res = NULL, *tmp;
GstStructure *meta = get_test_file_meta ();
if (!testfile_structs)
return FALSE;
if (meta && gst_structure_has_field (meta, "scenario")) {
*scenario_name = gst_structure_get_string (meta, "scenario");
return TRUE;
}
for (tmp = testfile_structs; tmp; tmp = tmp->next) {
GstStructure *structure = NULL;
if (gst_structure_has_name (tmp->data, "set-globals"))
continue;
structure = gst_structure_copy (tmp->data);
if (gst_structure_has_name (structure, "meta"))
gst_structure_remove_fields (structure, "configs", "gst-validate-args",
NULL);
res = g_list_append (res, structure);
}
*structs = res;
*original_name = global_testfile;
return TRUE;
}
GstStructure *
gst_validate_setup_test_file (const gchar * testfile, gboolean use_fakesinks)
{
const gchar *tool;
GstStructure *res = NULL;
if (global_testfile)
g_error ("A testfile was already loaded: %s", global_testfile);
gst_validate_set_globals (NULL);
gst_validate_structure_set_variables_from_struct_file (NULL, testfile);
testfile_structs =
gst_validate_utils_structs_parse_from_filename (testfile, NULL);
if (!testfile_structs)
g_error ("Could not load test file: %s", testfile);
res = testfile_structs->data;
if (gst_structure_has_name (testfile_structs->data, "set-globals")) {
GstStructure *globals = testfile_structs->data;
gst_validate_set_globals (globals);
res = testfile_structs->next->data;
}
if (!gst_structure_has_name (res, "meta"))
g_error ("First structure of a .validatetest file should be a `meta` or "
"`set-gobals` then `meta`, got: %s", gst_structure_to_string (res));
register_action_types ();
gst_validate_scenario_check_and_set_needs_clock_sync (testfile_structs, &res);
gst_validate_set_test_file_globals (res, testfile, use_fakesinks);
gst_validate_structure_resolve_variables (res, NULL);
tool = gst_structure_get_string (res, "tool");
if (!tool)
tool = "gst-validate-" GST_API_VERSION;
if (g_strcmp0 (tool, g_get_prgname ()))
g_error ("Validate test file: '%s' was made to be run with '%s' not '%s'",
testfile, tool, g_get_prgname ());
global_testfile = g_strdup (testfile);
return res;
}

View file

@ -23,11 +23,15 @@ G_BEGIN_DECLS
GST_VALIDATE_API
void gst_validate_init (void);
GST_VALIDATE_API
void gst_validate_init_debug (void);
GST_VALIDATE_API
void gst_validate_deinit (void);
GST_VALIDATE_API
GList * gst_validate_plugin_get_config (GstPlugin * plugin);
GST_VALIDATE_API
gboolean gst_validate_is_initialized (void);
GST_VALIDATE_API
GstStructure *gst_validate_setup_test_file(const gchar * testfile, gboolean use_fakesinks);
G_END_DECLS

View file

@ -182,6 +182,26 @@ class FakeMediaDescriptor(MediaDescriptor):
return self._infos.get('plays-reverse', False)
class GstValidateSimpleTestsGenerator(GstValidateTestsGenerator):
def __init__(self, name, test_manager, tests_dir):
self.tests_dir = tests_dir
super().__init__(name, test_manager)
def populate_tests(self, uri_minfo_special_scenarios, scenarios):
for root, _, files in os.walk(self.tests_dir):
for f in files:
name, ext = os.path.splitext(f)
if ext != ".validatetest":
continue
fpath = os.path.abspath(os.path.join(root, f))
pathname = os.path.abspath(os.path.join(root, name))
name = pathname.replace(os.path.commonpath([self.tests_dir, root]), '').replace('/', '.')
self.add_test(GstValidateSimpleTest(fpath, 'test' + name,
self.test_manager.options,
self.test_manager.reporter))
class GstValidatePipelineTestsGenerator(GstValidateTestsGenerator):
def __init__(self, name, test_manager, pipeline_template=None,
@ -191,7 +211,7 @@ class GstValidatePipelineTestsGenerator(GstValidateTestsGenerator):
@pipeline_template: A template pipeline to be used to generate actual pipelines
@pipelines_descriptions: A list of tuple of the form:
(test_name, pipeline_description, extra_data)
extra_data being a dictionnary with the follwing keys:
extra_data being a dictionary with the following keys:
'scenarios': ["the", "valide", "scenarios", "names"]
'duration': the_duration # in seconds
'timeout': a_timeout # in seconds
@ -515,11 +535,10 @@ class GstValidateCheckAccurateSeekingTestGenerator(GstValidatePipelineTestsGener
continue
config = [
'%(ssim)s, element-name="videoconvert", reference-images-dir="' + \
reference_frame_dir + '", framerate=%d/%d' % (framerate.numerator, framerate.denominator)
'%(ssim)s, element-name="videoconvert", reference-images-dir="'
+ reference_frame_dir + '", framerate=%d/%d' % (framerate.numerator, framerate.denominator)
]
pipelines[test_name] = {
"pipeline": "uridecodebin uri=" + media_info.get_uri() + " ! deinterlace ! videoconvert ! video/x-raw,interlace-mode=progressive,format=I420 ! videoconvert name=videoconvert ! %(videosink)s",
"media_info": media_info,
@ -651,6 +670,17 @@ class GstValidateMixerTestsGenerator(GstValidatePipelineTestsGenerator):
)
class GstValidateSimpleTest(GstValidateTest):
def __init__(self, test_file, *args, **kwargs):
self.test_file = test_file
super().__init__(GstValidateBaseTestManager.COMMAND, *args, **kwargs)
def build_arguments(self):
self.add_arguments('--set-test-file', self.test_file)
if self.options.mute:
self.add_arguments('--use-fakesinks')
class GstValidateLaunchTest(GstValidateTest):
def __init__(self, classname, options, reporter, pipeline_desc,
@ -960,7 +990,7 @@ class GstValidateTestManager(GstValidateBaseTestManager):
group = parser.add_argument_group("GstValidate tools specific options"
" and behaviours",
description="""When using --wanted-tests, all the scenarios can be used, even those which have
not been tested and explicitely activated if you set use --wanted-tests ALL""")
not been tested and explicitly activated if you set use --wanted-tests ALL""")
group.add_argument("--validate-check-uri", dest="validate_uris",
action="append", help="defines the uris to run default tests on")
group.add_argument("--validate-tools-path", dest="validate_tools_path",
@ -1035,7 +1065,7 @@ not been tested and explicitely activated if you set use --wanted-tests ALL""")
media_descriptor = GstValidateMediaDescriptor(media_info)
try:
# Just testing that the vairous mandatory infos are present
# Just testing that the various mandatory infos are present
caps = media_descriptor.get_caps()
if uri is None or media_descriptor.get_protocol() == Protocols.IMAGESEQUENCE:
uri = media_descriptor.get_uri()

View file

@ -45,6 +45,7 @@
static gint ret = 0;
static GMainLoop *mainloop;
static GstElement *pipeline;
static gboolean is_testfile;
static gboolean buffering = FALSE;
static gboolean is_live = FALSE;
@ -90,7 +91,7 @@ bus_callback (GstBus * bus, GstMessage * message, gpointer data)
break;
}
case GST_MESSAGE_EOS:
if (!g_getenv ("GST_VALIDATE_SCENARIO"))
if (!g_getenv ("GST_VALIDATE_SCENARIO") && !is_testfile)
g_main_loop_quit (loop);
break;
case GST_MESSAGE_ASYNC_DONE:
@ -198,7 +199,7 @@ bus_callback (GstBus * bus, GstMessage * message, gpointer data)
if (GST_IS_VALIDATE_SCENARIO (GST_MESSAGE_SRC (message))
&& state == GST_STATE_NULL) {
gst_validate_printf (GST_MESSAGE_SRC (message),
"State change request NULL, quiting mainloop\n");
"State change request NULL, quitting mainloop\n");
g_main_loop_quit (mainloop);
}
break;
@ -301,17 +302,47 @@ _register_playbin_actions (void)
/* *INDENT-ON* */
}
int main (int argc, gchar ** argv);
static int
run_test_from_file (const gchar * testfile, gboolean use_fakesinks)
{
gint argc, ret;
gchar **args, **argv;
GstStructure *meta = gst_validate_setup_test_file (testfile, use_fakesinks);
args = gst_validate_utils_get_strv (meta, "args");
if (!args)
g_error ("No 'args' in .validatetest meta structure: %s",
gst_structure_to_string (meta));
for (argc = 0; args[argc]; argc++);
argc++;
argv = g_new0 (char *, argc + 1);
argv[0] = argv[0];
memcpy (&argv[1], args, sizeof (char *) * (argc));
ret = main (argc, argv);
g_strfreev (args);
g_free (argv);
return ret;
}
int
main (int argc, gchar ** argv)
{
GError *err = NULL;
gchar *scenario = NULL, *configs = NULL, *media_info = NULL,
*verbosity = NULL;
*verbosity = NULL, *testfile = NULL;
gboolean list_scenarios = FALSE, monitor_handles_state,
inspect_action_type = FALSE;
GstStateChangeReturn sret;
gchar *output_file = NULL;
BusCallbackData bus_callback_data = { 0, };
gboolean use_fakesinks = FALSE;
#ifdef G_OS_UNIX
guint signal_watch_id;
@ -319,12 +350,17 @@ main (int argc, gchar ** argv)
int rep_err;
GOptionEntry options[] = {
{"set-test-file", '\0', 0, G_OPTION_ARG_FILENAME, &testfile,
"Let you set a all container testfile", NULL},
{"set-scenario", '\0', 0, G_OPTION_ARG_FILENAME, &scenario,
"Let you set a scenario, it can be a full path to a scenario file"
" or the name of the scenario (name of the file without the"
" '.scenario' extension).", NULL},
{"list-scenarios", 'l', 0, G_OPTION_ARG_NONE, &list_scenarios,
"List the available scenarios that can be run", NULL},
{"use-fakesinks", 'm', 0, G_OPTION_ARG_NONE, &use_fakesinks,
"Use fakesinks when possible. This will have effect when using"
" test files.", NULL},
{"verbosity", 'v', 0, G_OPTION_ARG_STRING, &verbosity,
"Set overall verbosity as defined by GstValidateVerbosityFlags"
" as a string", NULL},
@ -379,6 +415,15 @@ main (int argc, gchar ** argv)
exit (1);
}
gst_init (&argc, &argv);
gst_validate_init_debug ();
if (testfile) {
is_testfile = TRUE;
if (scenario)
g_error ("Can not specify scenario and testfile at the same time");
return run_test_from_file (testfile, use_fakesinks);
}
if (scenario || configs) {
gchar *scenarios;
@ -393,7 +438,6 @@ main (int argc, gchar ** argv)
g_free (configs);
}
gst_init (&argc, &argv);
gst_validate_init ();
if (list_scenarios || output_file) {
@ -431,19 +475,26 @@ main (int argc, gchar ** argv)
/* Create the pipeline */
argvn = g_new0 (char *, argc);
memcpy (argvn, argv + 1, sizeof (char *) * (argc - 1));
pipeline = (GstElement *) gst_parse_launchv ((const gchar **) argvn, &err);
g_free (argvn);
if (argc == 2) {
gst_validate_printf (NULL, "**-> Pipeline: '%s'**\n", argvn[0]);
pipeline = (GstElement *) gst_parse_launch (argvn[0], &err);
} else {
pipeline = (GstElement *) gst_parse_launchv ((const gchar **) argvn, &err);
}
if (!pipeline) {
g_print ("Failed to create pipeline: %s\n",
err ? err->message : "unknown reason");
g_clear_error (&err);
g_object_unref (runner);
g_free (argvn);
exit (1);
} else if (err) {
g_printerr ("Erroneous pipeline: %s\n",
err->message ? err->message : "unknown reason");
g_clear_error (&err);
g_free (argvn);
return 1;
}
@ -500,7 +551,12 @@ main (int argc, gchar ** argv)
g_signal_connect (bus, "message", (GCallback) bus_callback,
&bus_callback_data);
g_print ("Starting pipeline\n");
if (argc == 2)
g_print ("-> Starting pipeline");
else
g_print ("-> Starting pipeline\n");
g_free (argvn);
g_object_get (monitor, "handles-states", &monitor_handles_state, NULL);
if (monitor_handles_state == FALSE) {
sret = gst_element_set_state (pipeline, GST_STATE_PLAYING);
@ -523,7 +579,7 @@ main (int argc, gchar ** argv)
}
g_print ("Pipeline started\n");
} else {
g_print ("Letting scenario handle set state\n");
g_print ("-> Letting scenario handle set state\n");
}
g_main_loop_run (mainloop);