Type of analytics meta data Handle containing data required to use gst_analytics_cls_mtd APIs. This type is generally expected to be allocated on the stack. Instance identifier Instance of #GstAnalyticsRelationMeta where the analytics-metadata identified by @id is stored. index of the class associated with @quarks ( and label) or a negative value on failure. Instance handle Quark of the class Get index of class represented by @quark Number of classes in this classification instance Instance handle Get number of classes Get confidence level for class at @index confidence level for @index, <0.0 if the call failed. instance handle Object class index Quark of this class (label) associated with @index Instance handle index of the class Get quark of the class at @index Get an id identifying #GstAnalyticsMtd type. opaque id of #GstAnalyticsMtd type Passes to functions asking for a relation span when the span is infinite. A wildcard matching any type of analysis Handle containing data required to use gst_analytics_mtd API. This type is generally expected to be allocated on the stack. Instance identifier Instance of #GstAnalyticsRelationMeta where the analytics-metadata identified by @id is stored. Get instance id Id of @instance Instance of #GstAnalyticsMtd opaque id of the type Instance of #GstAnalyticsMtd Get analysis result type. Get instance size Size (in bytes) of this instance or 0 on failure. Instance of #GstAnalyticsMtd Gets the string version of the name of this type of analytics data the name The type of analytics data This structure must be provided when registering a new type of Mtd. It must have a static lifetime (never be freed). The name of the metadata type A pointer to a function that will be called when the containing meta is transform to potentially copy the data into a new Mtd into the new meta. A pointer to a function that will be called when the containing meta is cleared to potetially do cleanup (ex. _unref or release) resources it was using. Handle containing data required to use gst_analytics_od_mtd APIs. This type is generally expected to be allocated on the stack. Instance identifier Instance of #GstAnalyticsRelationMeta where the analytics-metadata identified by @id is stored. Retrieve location confidence level. TRUE on success, otherwise FALSE. instance Confidence on object location Retrieve location and location confidence level. TRUE on success, otherwise FALSE. instance x component of upper-left corner of the object location y component of upper-left corner of the object location bounding box width of the object location bounding box height of the object location Confidence on object location Quark of the class of object associated with this location. Quark different from on success and 0 on failure. Instance handle Get an id that represent object-detection metadata type Opaque id of the #GstAnalyticsMtd type No relation First analysis-meta is part of second analysis-meta First analysis-meta contain second analysis-meta. reserved Only use for criteria. An opaque #GstMeta that can be used to hold various types of results from analysis processes. The content should be accessed through the API. Add analytic classification metadata to @instance. Added successfully Instance of #GstAnalyticsRelationMeta where to add classification instance length of @confidence_levels confidence levels labels of this classification. Order define index, quark, labels relation. This array need to exist as long has this classification meta exist. Handle updated to newly added classification meta. Add a relatable metadata to @meta. This method is meant to be used by new struct sub-classing GstAnalyticsRelatableMtd. A pointer to a memory area of size @size where to put the data Instance Implementation of relatable (#GstAnalyticsRelatableMtd) Size required Updated handle Added successfully Instance of #GstAnalyticsRelationMeta where to add classification instance Quark of the object type x component of bounding box upper-left corner y component of bounding box upper-left corner bounding box width bounding box height confidence level on the object location Handle updated with newly added object detection meta. Add an object-detetion metadata to @instance. Add analytic classification metadata to @instance. Added successfully Instance of #GstAnalyticsRelationMeta where to add classification instance confidence levels labels of this classification. Order define index, quark, labels relation. This array need to exist as long has this classification meta exist. Handle updated to newly added classification meta. Add analytics segmentation metadata to @instance. The rectangle (@masks_loc_x, @mask_loc_y, @mask_loc_w, @mask_loc_h) define a area of the image that correspond to the segmentation masks stored in @buffer. For example if the segmentation masks stored in @buffer describe the segmented regions for the entire image the rectangular area will be (@masks_loc_x = 0, @masks_loc_y = 0, @masks_loc_w = image_width, @masks_loc_h = image_height). TRUE if added successfully, otherwise FALSE Instance of #GstAnalyticsRelationMeta where to add segmentation instance. Buffer containing segmentation masks. @buffer must have a #GstVideoMeta attached Segmentation type Number of regions in the masks Arrays of region ids present in the mask. Left coordinate of the rectangle corresponding to the masks in the image. Top coordinate of the rectangle corresponding to the masks in the image. Width of the rectangle corresponding to the masks in the image. Height of the rectangle corresponding to the masks in the image. Handle update with newly added segmentation meta. Added successfully Instance of GstAnalyticsRelationMeta where to add tracking mtd Tracking id Timestamp of first time the object was observed. Handle updated with newly added tracking meta. Add an analytic tracking metadata to @instance. Verify existence of relation(s) between @an_meta_first_d and @an_meta_second_id according to relation condition @cond_types. It optionally also return a shortest path of relations ( compliant with @cond_types) between @an_meta_first_id and @an_meta_second_id. TRUE if a relation between exit between @an_meta_first_id and @an_meta_second_id, otherwise FALSE. a #GstAnalyticsRelationMeta describing analysis-meta relation First analysis-meta Second analysis-meta Maximum number of relation between @an_meta_first_id and @an_meta_second_id. A value of 1 mean only only consider direct relation. condition on relation types. If not NULL this list will be filled with relation path between @an_meta_first_id and @an_meta_second_id. List value should be access with GSList API. Use GPOINTER_TO_INT(iter->data) where iter is a GSList element to get analysis-meta id on the relation path. Free this list with g_slist_free (@relations_path) after using. Fill @rlt if a analytics-meta with id == @an_meta_id exist in @meta instance, otherwise this method return FALSE and @rlt is invalid. TRUE if successful. Instance of #GstAnalyticsRelationMeta Id of #GstAnalyticsClsMtd instance to retrieve Will be filled with relatable meta TRUE if @rlt_mtd was updated, other wise FALSE GstAnalyticsRelationMeta instance where to query for GstAnalyticsRelatableMtd. Id of GstAnalyticsMtd involved in relation to query Type of relation to filter on. Type of GstAnalyticsMtd to filter on Opaque data to store state of the query. If @state point to NULL, the first analytics-metadata directly related to @an_meta_id will be set in @rlt_mtd. Doesn't need to be free. Handle updated to directly related relatable meta. Fill @rlt if a analytics-meta with id == @an_meta_id exist in @meta instance, otherwise this method return FALSE and @rlt is invalid. TRUE if successful. Instance of GstAnalyticsRelationMeta Id of GstAnalyticsMtd instance to retrieve Filter on a specific type of analysis, use %GST_ANALYTICS_MTD_TYPE_ANY to match any type Will be filled with relatable meta Analytics data pointer Instance of GstAnalyticsRelationMeta Id of GstAnalyticsMtd instance to retrieve Fill @rlt if a analytics-meta with id == @an_meta_id exist in @meta instance, otherwise this method return FALSE and @rlt is invalid. TRUE if successful. Instance of #GstAnalyticsRelationMeta Id of #GstAnalyticsODMtd instance to retrieve Will be filled with relatable meta Get relations between first and second analysis-meta. Ids (@an_meta_first_id and @an_meta_second_id) must be from a call to @gst_analytics_mtd_get_id (handle). relation description between first and second analysis-meta. a #GstAnalyticsRelationMeta Id of first analysis-meta Id of second analysis-meta Fill @rlt if a analytics-meta with id == @an_meta_id exist in @meta instance, otherwise this method return FALSE and @rlt is invalid. TRUE if successful. Instance of GstAnalyticsRelationMeta Id of GstAnalyticsMtd instance to retrieve Will be filled with relatable meta FALSE if end was reached and iteration is completed. Instance of GstAnalyticsRelationMeta Opaque data to store iteration state, initialize to NULL, no need to free it. Type of GstAnalyticsMtd to iterate on or use %GST_ANALYTICS_MTD_TYPE_ANY for any. Handle updated to iterated GstAnalyticsRelatableMtd. Sets the relation (#GstAnalyticsRelTypes) between @an_meta_first and @an_meta_second. Ids must have been obtained a call to @gst_analytics_mtd_get_id(handle). TRUE on success and FALSE on failure. Parameter to receive new maximum number of analysis-meta described by relation. a #GstAnalyticsRelTypes defining relation between two analysis-meta first meta id second meta id Get the meta info GstAnalyticsRelationMeta initialization parameters. Initial relations order. Buffer size in bytes to store relatable metadata This type of metadata holds information on which pixels belongs to a region of the image representing a type of object. It supports two types of segmentation, semantic or instance: * Semantic: All objects of the same type have the same id * Instance: Each instance of an object has a different id The results of the segmentation are stored in a #GstBuffer that has a #GstVideoMeta associated with it. This buffer is stored in the GstAnalyticsSegmentationMtd using #gst_analytics_relation_meta_add_segmentation_mtd(). The #GstBuffer containing the segmentation mask is image-like but the color values are arbitrary values, referred by region-id in this API, without meaning beyond specifying that two pixels in the original image with the same values in their corresponding mask value belong to the same region. To further describe a region, the #GstAnalyticsSegmentationMtd can be associated with other #GstAnalyticsMtd. Since region ids are generated by the segmentation process itself and are not always sequential, we use a map of indexes to region ids starting with 0 without discontinuity which facilitate N-to-N mapping with other #GstAnalyticsMtd. For example it can be associated with #GstAnalyticsClsMtd to describe the class of object matching the pixels of a segmented region. Example: Associate Instance Segmentation with Classification In the following example the segmentation process will fill segmask with values of 0 for background, 12 for the first region which correspond to a to a strawberry, 7 for the second region that also correspond to a strawberry in the image and 31 for the third region that correspond to a leaf in the image. region_ids is fill during segmentation post-processing region_ids: |region-index | region-id | |-------------|-----------| | 0 | 0 | | 1 | 12 | | 2 | 7 | | 3 | 31 | region_count = 4 ``` C GstAnalyticsSegmentationMtd segmtd; GstAnalyticsClassificationMtd clsmtd; GstBuffer *segmask, *img; guint *region_ids; gsize region_count, class_count; gfloat *class_confidence; GQuark *classes; ... (segmentation filling segmask based on img) gst_analytics_relation_meta_add_segmentation_mtd (rmeta, segmask, GST_SEGMENTATION_TYPE_INSTANCE, region_count, region_ids, &segmtd); class_count = region_count; ... (class-index must match and correspond to region-index) classes [0] = g_quark_from_string ("background"); classes [1] = g_quark_from_string ("strawberry"); classes [2] = g_quark_from_string ("strawberry"); classes [3] = g_quark_from_string ("leaf"); ... (set confidence level for each class associated with a region ... where -1.0 mean undefined.) class_confidence [0] = -1.0; class_confidence [1] = 0.6; class_confidence [2] = 0.9; class_confidence [3] = 0.8; gst_analytics_relation_meta_add_cls_mtd (rmeta, class_count, class_confidence, classes, &clsmtd); gst_analytics_relation_meta_set_relation (rmeta, GST_ANALYTICS_REL_TYPE_RELATE_TO, segmtd.id, clsmtd.id); ``` Example: Associate Semantic Segmentation with Classification Assuming the same context as for Instance Segmentation above but instead a semantic segmentation is performed, therefore region-id-12 and region-id-7 are now represented by the same region-id-12 region_ids: (here |region-index | region-id | |-------------|-----------| | 0 | 0 | | 1 | 12 | | 2 | 31 | Code remain the same except that we set all confidence level to undefined (-1.0). ``` ... (class-index must match and correspond to region-index) classes [0] = g_quark_from_string ("background"); classes [1] = g_quark_from_string ("strawberry"); classes [2] = g_quark_from_string ("leaf"); ... (set confidence level for each class associated with a region ... where -1.0 mean undefined.) class_confidence [0] = -1.0; class_confidence [1] = -1.0; class_confidence [2] = -1.0; gst_analytics_relation_meta_add_cls_mtd (rmeta, class_count, class_confidence, classes, &clsmtd); gst_analytics_relation_meta_set_relation (rmeta, GST_ANALYTICS_REL_TYPE_RELATE_TO, segmtd.id, clsmtd.id); ``` Example: Retrieving class associated with a segmentation region-id-12 This the typical case for an overlay as we visit the segmentation mask we we find region-id values ``` gsize idx; gst_analytics_segmentation_mtd_get_region_index (&segmtd, &idx, 12); gst_analytics_relation_meta_get_direct_related (rmeta, segmtd.id, GST_ANALYTICS_REL_TYPE_RELATE_TO, gst_analytics_cls_mtd_get_type (), NULL, &clsmtd); GQuark region_class = gst_analytics_cls_mtd_get_quark (&segmtd, idx) ... ``` Since: 1.26 Instance identifier Instance of #GstAnalyticsRelationMeta where the analytics-metadata identified by @id is stored. Get segmentation mask data. Segmentation mask data stored in a #GstBuffer Instance Left coordinate of the rectangle corresponding to the mask in the image. Top coordinate of the rectangle corresponding to the mask in the image. Width of the rectangle corresponding to the mask in the image. Height of the rectangle corresponding to the mask in the image. Get the regions count. Number of regions segmented Instance Get id of the region corresponding to @index, which should be smaller than the return value of gst_analytics_segmentation_mtd_get_region_count() The region ID Instance Region index Get region index of the region identified by @id. TRUE if a region with @id exist, otherwise FALSE Instance Region index Region id Get an instance of #GstAnalyticsMtdType that represent segmentation metadata type. A #GstAnalyticsMtdType type Enum value describing supported segmentation type Segmentation where the belonging of each pixel to a class of objects is identified. Segmentation where the belonging of each pixel to instance of an object is identified. Hold tensor data semantically identify the contents of the tensor Indicate tensor layout #GstTensorDataType of tensor data Model batch size #GstBuffer holding tensor data Indicate tensor elements layout in memory. number of tensor dimensions number of tensor dimensions Allocate a tensor with @num_dims dimensions. tensor allocated Number of dimension of the tensors Allocates a new #GstTensor of @dims_order ROW_MAJOR or COLUMN_MAJOR and with an interleaved layout A newly allocated #GstTensor semantically identify the contents of the tensor #GstTensorDataType of tensor data Model batch size #GstBuffer holding tensor data Indicate tensor dimension indexing order number of tensor dimensions tensor dimensions Create a copy of @tensor. a new #GstTensor a #GstTensor to be copied Free tensor pointer to tensor to free Gets the dimensions of the tensor. The dims array form the tensor a #GstTensor The number of dimensions Describe the type of data contain in the tensor. signed 4 bit integer tensor data signed 8 bit integer tensor data signed 16 bit integer tensor data signed 32 bit integer tensor data signed 64 bit integer tensor data unsigned 4 bit integer tensor data unsigned 8 bit integer tensor data unsigned 16 bit integer tensor data unsigned 32 bit integer tensor data unsigned 64 bit integer tensor data 16 bit floating point tensor data 32 bit floating point tensor data 64 bit floating point tensor data "brain" 16 bit floating point tensor data Hold properties of the tensor's dimension Size of the dimension Dimension order in memory. @see_also #GST_TENSOR_DIM_ORDER_INDEXED Indicate to read tensor from memory in row-major or column-major order. elements along a row are consecutive in memory elements along a column are consecutive in memory elements storage follow the order defined by #GstTensorDim.order_index This mean that when iterating the tensor the dimension with index 0 is the most nested in the loops and consecutive in memory, followed by other dimensions in the order defined by #GstTensorDim.order_index. Indicate tensor storage in memory. indicate the tensor is stored in a dense format in memory parent number of tensors Retrieves a tensor from the #GstTensorMeta, the index must be smaller than #GstTensorMeta.num_tensors a GstTensor A #GstTensorMeta The number of the tensor to get Finds the first tensor with the requsted ID in the meta The index of the tensor inthe meta, or -1 if its not found. a #GstTensorMeta The tensor id to look for Sets tensors into the #GstTensorMeta a #GstTensorMeta The number of tensors in the @tensors array An array of poiners to #GstTensor Store information on results of object tracking Instance identifier Instance of #GstAnalyticsRelationMeta where the analytics-metadata identified by @id is stored. Retrieve tracking information. Successfully retrieved info. Instance of tracking metadata Updated tracking id Updated timestamp of the tracking first observation. Updated timestamp of the tracking last observation. Has the tracking been lost Update successful Instance of GstAnalyticsTrackingMtd. Set tracking to lost GstAnalyticsTrackingMtd instance Timestamp of last time this object was tracked id representing the type of GstAnalyticsRelatableMtd Get the opaque id identifying the relatable type Attach a analysis-results-meta-relation meta (#GstAnalyticsRelationMeta)to @buffer. A #GstAnalyticsRelationMeta is a metadata describing relation between other analysis meta. It's more efficient to use #gst_buffer_add_analytics_relation_meta_full and providing the maximum number of analysis meta that will attached to a buffer. Newly attached #GstAnalyticsRelationMeta a #GstBuffer Attache a analysis-results relation-meta (#GstAnalyticsRelationMeta) to @buffer. A #GstAnalyticsRelationMeta is a metadata describing relation between other analysis meta. Newly attached #GstAnalyticsRelationMeta a #GstBuffer Initialization parameters Adds a #GstTensorMeta to a buffer or returns the existing one The new #GstTensorMeta A writable #GstBuffer Retrives the meta or %NULL if it doesn't exist The #GstAnalyticsRelationMeta if there is one a #GstBuffer Gets the #GstTensorMeta from a buffer The #GstTensorMeta if there is wone A #GstBuffer Get an id identifying #GstAnalyticsMtd type. opaque id of #GstAnalyticsMtd type This type of metadata holds classification, it is generally used in relationship with another metadata type to enhance its content. For example, it can enhance the classifcation of an object detection held by the #GstAnalyticsODMtd metadata type. The #GstAnalyticsRelationMeta is a #GstMeta that can contain a large number of results from the analysis of a meta. Each result can be accessed by using its id, or more conviently, by using a #GstAnalyticsMtd. A matrix of relationships between the various metadata is also defined and can be filled by the analysis processes. This type of metadata holds the position of detected object inside the image, along with the probabily of each detection. This type of metadata holds tracking information. In many cases, it is desired to track an object across many frames. This type of metadata holds information about the tracking, for example, it can be used alongside a #GstAnalyticsODMtd to track an object. Gets the string version of the name of this type of analytics data the name The type of analytics data Get an id that represent object-detection metadata type Opaque id of the #GstAnalyticsMtd type Get number of relatable meta attached to instance Number of analysis-meta attached to this instance. Instance of #GstAnalyticsRelationMeta GType of GstAnalyticsRelationMeta Get the meta info Get an instance of #GstAnalyticsMtdType that represent segmentation metadata type. A #GstAnalyticsMtdType type id representing the type of GstAnalyticsRelatableMtd Get the opaque id identifying the relatable type