diff --git a/docs/random/streamheader b/docs/random/streamheader new file mode 100644 index 0000000000..31bff21510 --- /dev/null +++ b/docs/random/streamheader @@ -0,0 +1,83 @@ +A lot of data streams need a few initial buffers to make sense of the +data stream. With these initial buffers, it can pick up at any point in +the rest of the data stream. + +Exampes: +- Vorbis and Theora have three initial Ogg packets +- MPEG4 has codec data +- GDP protocol serializes the initial new_segment event and the initial caps + +It is important that elements that get connected after the stream starts, +receive these initial buffers. Also, sink elements should know that these +initial buffers are necessary for every new "client" of the sink element; +for example, multifdsink needs to send these initial buffers before the +normal buffers. + +Currently, this is done by putting a 'streamheader' value on the caps. +'streamheader' is a GST_TYPE_ARRAY of GstBuffer, ie. an ordered list of +buffers. + +The buffers themselves still get pushed as normal buffers, but with the +IN_CAPS flag set. This allows elements that do not know about streamheader +to still function correctly (ie. filesink and others) + +It is of course important that these streamheader buffers are not sent twice +(once because they're in the caps, and once because they're received as +buffers). + +So, an element that is aware of streamheader in the caps should probably best +drop all incoming IN_CAPS buffers. + +Rules for sending streamheaders: +- all streamheader buffers should have IN_CAPS set +- from this list of streamheader buffers, a "streamheader" caps field should + be created as a GST_TYPE_ARRAY of GstBuffer. This should be done by + copying the buffers. + (the only important thing about the buffers in this list is the data; + other attributes of the buffers get ignored) +- each of these buffers should then have these caps set on them. +- when all streamheader buffers are collected in the element, pad caps should + be set, including this streamheader +- streamheader buffers should be sent consecutively, and before any of the + data (non-IN_CAPS) buffers they apply to. +- when new streamheader buffers need to be pushed out, this process is + repeated. Receiving a new IN_CAPS buffer after a non-IN_CAPS buffer + signifies resetting streamheader, as does the new set_caps with different + streamheader right before. + +Elements that can send streamheader caps: +- vorbisenc +- theoraenc +- gdppay + +Elements that can receive streamheader caps: +- multifdsink + +Elements that can receive and send streamheader caps: +- oggmux + +Elements that could/should use this: +- ffenc_mpeg4 (currently uses/sets codec_data) +- theoradec, vorbisdec (currently only look at incoming buffers) +- oggdemux + +FUTURE PLANS +------------ +The concept of streamheader is more generally applicable. We may want to +find a better way of implementing this than having two ways of telling +downstream elements about them. Currently an element that is +streamheader-aware needs to look both at caps and incoming buffers. + +One option would be to only rely on the buffer flow, and make all elements +aware of a HEADER flag, causing them to keep these buffers around and push +them for each new pad link. This needs to be done in every element; a +scenario like + videotestsrc ! theoraenc +where later on an oggmux gets connected, should still work without dropping +these header buffers. But similarly, + videotestsrc ! theoraenc ! identity +with a later connection of oggmux to identity should work. + +This could tie in with an idea to have pads store some initial data (the +first new_segment event, tag events, ...), which at the moment also get +dropped on dynamic in-stream links.