design: write about the current state of tag-handling

Document the taghandling in gstreamer. List gaps and propose new mechanisms to deal with them.
2024-11-18 07:47:17 +00:00 · 2010-02-17 09:55:52 +02:00 · 2010-02-17 09:55:52 +02:00 · a12fc26461
commit a12fc26461
parent b45abd2710
1 changed files with 199 additions and 0 deletions
--- a/docs/design/draft-metadata.txt
+++ b/docs/design/draft-metadata.txt
@ -0,0 +1,199 @@
+Metadata
+--------
+
+This draft recaps the current metadata handling in GStreamer and proposes some
+additions.
+
+
+Supported Metadata standards
+----------------------------
+
+The paragraphs below list supported native metadata standards sorted by type and
+then in alphabetical order. Some standards have been extended to support
+additional metadata. GStreamer already supports all of those to some extend. 
+This is showns in the table below as either [--], [r-], [-w] or [rw] depending on
+read/write support (08.Feb.2010).
+
+Audio
+- mp3
+  ID3v2: [rw]
+    http://www.id3.org/Developer_Information
+  ID3v1: [rw] 
+    http://www.id3.org/ID3v1
+  XMP: [--] (inside ID3v2 PRIV tag of owner XMP)
+    http://www.adobe.com/devnet/xmp/
+- ogg/vorbis
+  vorbiscomment: [rw] 
+    http://www.xiph.org/vorbis/doc/v-comment.html
+    http://wiki.xiph.org/VorbisComment
+- wav
+  LIST/INFO chunk: [rw]
+    http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/RIFF.html#Info
+    http://www.kk.iij4u.or.jp/~kondo/wave/mpidata.txt
+  XMP: [--]
+    http://www.adobe.com/devnet/xmp/
+
+Video
+- 3gp
+  {moov,trak}.udta:  [rw]
+     http://www.3gpp.org/ftp/Specs/html-info/26244.htm 
+  ID3V2: [--]
+     http://www.3gpp.org/ftp/Specs/html-info/26244.htm 
+     http://www.mp4ra.org/specs.html#id3v2
+- avi
+  LIST/INFO chunk: [rw]
+    http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/RIFF.html#Info
+    http://www.kk.iij4u.or.jp/~kondo/wave/mpidata.txt
+  XMP: [--] (inside "_PMX" chunk)
+    http://www.adobe.com/devnet/xmp/
+- asf
+  ??: 
+  XMP: [--]
+    http://www.adobe.com/devnet/xmp/
+- flv [--]
+  XMP: (inside onXMPData script data tag)
+    http://www.adobe.com/devnet/xmp/
+- mkv
+  tags: [rw]
+    http://www.matroska.org/technical/specs/tagging/index.html
+- mov
+  XMP: [--] (inside moov.udta.XMP_ box)
+    http://www.adobe.com/devnet/xmp/
+- mp4
+  {moov,trak}.udta: [rw]
+    http://standards.iso.org/ittf/PubliclyAvailableStandards/c051533_ISO_IEC_14496-12_2008.zip
+  moov.udta.meta.ilst: [rw]
+    http://atomicparsley.sourceforge.net/
+    http://atomicparsley.sourceforge.net/mpeg-4files.html
+  ID3v2: [--]
+    http://www.mp4ra.org/specs.html#id3v2
+  XMP: [--] (inside UUID box)
+    http://www.adobe.com/devnet/xmp/
+- mxf
+  ??
+
+Images
+- gif
+  XMP: [--]
+    http://www.adobe.com/devnet/xmp/
+- jpg
+  jif: [rw] (only comments)
+  EXIF: [rw] (via metadata plugin)
+    http://www.exif.org/specifications.html
+  IPTC: [rw] (via metadata plugin)
+    http://www.iptc.org/IPTC4XMP/
+  XMP: [rw] (via metadata plugin)
+    http://www.adobe.com/devnet/xmp/
+- png
+  XMP: [--]
+    http://www.adobe.com/devnet/xmp/
+
+further Links:
+http://age.hobba.nl/audio/tag_frame_reference.html
+http://wiki.creativecommons.org/Tracker_CC_Indexing
+
+
+Current Metadata handling
+-------------------------
+
+When reading files, demuxers or parsers extract the metadata. It will be sent as
+a GST_MESSAGE_TAG to the GstBus and GST_EVENT_TAG to downstream elements.
+
+Elements receiving GST_EVENT_TAG events can mangle them, mux them into the 
+buffers they send or just pass them through. Usually is muxers that will format
+the tag data into the form required by the format they mux. Such elements would
+also implement the GstTagSetter interface to receive tags from the application.
+
+   GstMessage(tag) over GstBus to application
+       ^
+       |
+  +----------+
+  | demux    |
+ sink       src --> GstEvent(tag) over GstPad to downstream element
+  +----------+
+
+           method call over GstTagSetter interface from application
+                                                          |
+                                                          v
+                                                    +----------+
+                                                    | mux      |
+GstEvent(tag) over GstPad from upstream element --> sink       src
+                                                    +----------+
+
+The data used in all those interfaces is GstTagList. It is based on a 
+GstStructure which is like a hash table with differently typed entries. The key
+is always a string/GQuark. Many keys are predefined in GStreamer core. More keys
+are defined in gst-plugins-base/gst-libs/gst/tag/tag.h.
+If elements and applications use predefined types, it is possible to transcode a
+file from one format into another while preserving all known and mapped
+metadata.
+
+
+Issues
+------
+
+Unknown/Unmapped metadata
+- - - - - - - - - - - - -
+Right now GStreamer can lose metadata when transcoding, remuxing content. This
+can happend as we don't map all metadata fields to generic ones.
+
+We should probably also add the whole metadata blob to the GstTagList. We would
+need a GST_TAG_SYSTEM_xxx define (e.g. GST_TAG_SYSTEM_ID3V2) for each standard.
+The content is not printable and should be treated as binary if not known. The
+tag is not mergeable - call gst_tag_register() with GstTagMergeFunc=NULL. Also
+the tag data is only useful for upstream elements, not for the application. 
+
+A muxer would first scan a taglist for known system tags. Unknown tags are
+ignored as already. It would first populate its own metadata store with the
+entries from the system tag and the update the entries with the data in normal
+tags.
+
+Below is an initial list of tag systems:
+ID3V1           - GST_TAG_SYSTEM_ID3V1
+ID3V2           - GST_TAG_SYSTEM_ID3V2
+RIFF_INFO       - GST_TAG_SYSTEM_RIFF_INFO
+XMP             - GST_TAG_SYSTEM_XMP
+
+We would basically need this for each container format.
+
+See also https://bugzilla.gnome.org/show_bug.cgi?id=345352
+
+Lost metadata
+- - - - - - -
+A case slighly different from the previous is that when an application sets a
+GstTagList on a pipeline. Right elements consuming tags do not report which tags
+have been consumed.  Especially when using elements that make metadata 
+persistent, we have no means of knowing which of the tags made it into the 
+target stream and which were not serialized. Ideally the application would like
+to know which kind of metadata is accepted by a pipleine to reflect that in the
+UI.
+
+Although it is in practise so that elements implementing GstTagSetter are the
+ones that serialize, this does not have to be so. Otherwise we could add a
+means to that interface, where elements add the tags they have serialized. The
+application could build one list from all the tag messages and then query all
+the serialized tags from tag-setters. The delta tells what has not been
+serialized.
+
+A different approach would be to query the list of supported tags in advance.
+This could be a query (GST_QUERY_TAG_SUPPORT). The query result could be a list
+of elements and their tags. As a convenience we could flatten the list of tags
+for the top-level element (if the query was sent to a bin) and add that.
+
+Tags are per Element
+- - - - - - - - - -
+In many cases we want tags per stream. Even metadata standards like mp4/3gp
+metadata supports that. Right now GST_MESSAGE_SRC(tags) is the element. We tried
+changing that to the pad, but that broke applications.
+Also we miss the symmetric functionality in GstTagSetter. This interface is
+usually implemented by elements.
+
+Open bugs
+- - - - -
+
+https://bugzilla.gnome.org/buglist.cgi?query_format=advanced;short_desc=tag;bug_status=UNCONFIRMED;bug_status=NEW;bug_status=ASSIGNED;bug_status=REOPENED;bug_status=NEEDINFO;short_desc_type=allwordssubstr;product=GStreamer
+
+Add GST_TAG_MERGE_REMOVE
+https://bugzilla.gnome.org/show_bug.cgi?id=560302
+
+