mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-11-03 16:09:39 +00:00
366 lines
15 KiB
Text
366 lines
15 KiB
Text
This document describes some things to know about the Ogg format, as well
|
|
as implementation details in GStreamer.
|
|
|
|
INTRODUCTION
|
|
============
|
|
|
|
ogg and the granulepos
|
|
----------------------
|
|
|
|
An ogg stream contains pages with a serial number and a granulepos.
|
|
The granulepos is a 64 bit signed integer. It is a value that in some way
|
|
represents a time since the start of the stream.
|
|
The interpretation as such is however both codec-specific and
|
|
stream-specific.
|
|
|
|
ogg has no notion of time: it only knows about bytes and granulepos values
|
|
on pages.
|
|
|
|
The granule position is just a number; the only guarantee for a valid ogg
|
|
stream is that within a logical stream, this number never decreases.
|
|
|
|
While logically a granulepos value can be constructed for every ogg packet,
|
|
the page is marked with only one granulepos value: the granulepos of the
|
|
last packet to end on that page.
|
|
|
|
theora and the granulepos
|
|
-------------------------
|
|
|
|
The granulepos in theora is an encoding of the frame number of the last
|
|
key frame ("i frame"), and the number of frames since the last key frame
|
|
("p frame"). The granulepos is constructed as the sum of the first number,
|
|
shifted to the left for granuleshift bits, and the second number:
|
|
granulepos = (pframe << granuleshift) + iframe
|
|
|
|
(This means that given a framenumber or a timestamp, one cannot generate
|
|
the one and only granulepos for that page; several granulepos possibilities
|
|
correspond to this frame number. You also need the last keyframe, as well
|
|
as the granuleshift.
|
|
However, given a granulepos, the theora codec can still map that to a
|
|
unique timestamp and frame number for that theora stream)
|
|
|
|
Note: currently theora stores the "presentation time" as the granulepos;
|
|
ie. a first data page with one packet contains one video frame and
|
|
will be marked with 0/0. Changing that to be 1/0 (so that it
|
|
represents the number of decodable frames up to that point, like
|
|
for Vorbis) is being discussed.
|
|
|
|
vorbis and granulepos
|
|
---------------------
|
|
|
|
In Vorbis, the granulepos represents the number of samples that can be
|
|
decoded from all packets up to that point.
|
|
|
|
In GStreamer, the vorbisenc elements produces a stream where:
|
|
- OFFSET is the time corresponding to the granulepos
|
|
number of bytes produced before
|
|
- OFFSET_END is the granulepos of the produced vorbis buffer
|
|
- TIMESTAMP is the timestamp matching the begin of the buffer
|
|
- DURATION is set to the length in time of the buffer
|
|
|
|
Ogg media mapping
|
|
-----------------
|
|
|
|
Ogg defines a mapping for each media type that it embeds.
|
|
|
|
For Vorbis:
|
|
|
|
- 3 header pages, with granulepos 0.
|
|
- 1 page with 1 packet header identification
|
|
- N pages with 2 packets comments and codebooks
|
|
- granulepos is samplenumber of next page
|
|
- one packet can contain a variable number of samples but one frame
|
|
that should be handed to the vorbis decoder.
|
|
|
|
For Theora
|
|
|
|
- 3 header pages, with granulepos 0.
|
|
- 1 page with 1 packet header identification
|
|
- N pages with 2 packets comments and codebooks
|
|
- granulepos is framenumber of last packet in page, where framenumber
|
|
is a combination of keyframe number and p frames since keyframe.
|
|
- one packet contains 1 frame
|
|
|
|
|
|
|
|
|
|
DEMUXING
|
|
========
|
|
|
|
ogg demuxer
|
|
-----------
|
|
|
|
This ogg demuxer has two modes of operation, which both share a significant
|
|
amount of code. The first mode is the streaming mode which is automatically
|
|
selected when the demuxer is connected to a non-getrange based element. When
|
|
connected to a getrange based element the ogg demuxer can do full seeking
|
|
with great efficiency.
|
|
|
|
1) the streaming mode.
|
|
|
|
In this mode, the ogg demuxer receives buffers in the _chain() function which
|
|
are then simply submitted to the ogg sync layer. Pages are then processed when
|
|
the sync layer detects them, pads are created for new chains and packets are
|
|
sent to the peer elements of the pads.
|
|
|
|
In this mode, no seeking is possible. This is the typical case when the
|
|
stream is read from a network source.
|
|
|
|
In this mode, no setup is done at startup, the pages are just read and decoded.
|
|
A new logical chain is detected when one of the pages has the BOS flag set. At
|
|
this point the existing pads are removed and new pads are created for all the
|
|
logical streams in this new chain.
|
|
|
|
|
|
2) the random access mode.
|
|
|
|
In this mode, the ogg file is first scanned to detect the position and length
|
|
of all chains. This scanning is performed using a recursive binary search
|
|
algorithm that is explained below.
|
|
|
|
find_chains(start, end)
|
|
{
|
|
ret1 = read_next_pages (start);
|
|
ret2 = read_prev_page (end);
|
|
|
|
if (WAS_HEADER (ret1)) {
|
|
}
|
|
else {
|
|
}
|
|
|
|
}
|
|
|
|
a) read first and last pages
|
|
|
|
start end
|
|
V V
|
|
+-----------------------+-------------+--------------------+
|
|
| 111 | 222 | 333 |
|
|
BOS BOS BOS EOS
|
|
|
|
|
|
after reading start, serial 111, BOS, chain[0] = 111
|
|
after reading end, serial 333, EOS
|
|
|
|
start serialno != end serialno, binary search start, (end-start)/2
|
|
|
|
start bisect end
|
|
V V V
|
|
+-----------------------+-------------+--------------------+
|
|
| 111 | 222 | 333 |
|
|
|
|
|
|
after reading start, serial 111, BOS, chain[0] = 111
|
|
after reading end, serial 222, EOS
|
|
|
|
while (
|
|
|
|
|
|
|
|
testcases
|
|
---------
|
|
|
|
a) stream without BOS
|
|
|
|
+----------------------------------------------------------+
|
|
111 |
|
|
EOS
|
|
|
|
b) chained stream, first chain without BOS
|
|
|
|
+-------------------+--------------------------------------+
|
|
111 | 222 |
|
|
BOS EOS
|
|
|
|
|
|
c) chained stream
|
|
|
|
+-------------------+--------------------------------------+
|
|
| 111 | 222 |
|
|
BOS BOS EOS
|
|
|
|
|
|
d) chained stream, second without BOS
|
|
|
|
+-------------------+--------------------------------------+
|
|
| 111 | 222 |
|
|
BOS EOS
|
|
|
|
What can an ogg demuxer do?
|
|
---------------------------
|
|
|
|
An ogg demuxer can read pages and get the granulepos from them.
|
|
It can ask the decoder elements to convert a granulepos to time.
|
|
|
|
An ogg demuxer can also get the granulepos of the first and the last page of a
|
|
stream to get the start and end timestamp of that stream.
|
|
It can also get the length in bytes of the stream
|
|
(when the peer is seekable, that is).
|
|
|
|
An ogg demuxer is therefore basically able to seek to any byte position and
|
|
timestamp.
|
|
|
|
When asked to seek to a given granulepos, the ogg demuxer should always convert
|
|
the value to a timestamp using the peer decoder element conversion function. It
|
|
can then binary search the file to eventually end up on the page with the given
|
|
granule pos or a granulepos with the same timestamp.
|
|
|
|
Seeking in ogg currently
|
|
------------------------
|
|
|
|
When seeking in an ogg, the decoders can choose to forward the seek event as a
|
|
granulepos or a timestamp to the ogg demuxer.
|
|
|
|
In the case of a granulepos, the ogg demuxer will seek back to the beginning of
|
|
the stream and skip pages until it finds one with the requested timestamp.
|
|
|
|
In the case of a timestamp, the ogg demuxer also seeks back to the beginning of
|
|
the stream. For each page it reads, it asks the decoder element to convert the
|
|
granulepos back to a timestamp. The ogg demuxer keeps on skipping pages until
|
|
the page has a timestamp bigger or equal to the requested one.
|
|
|
|
It is therefore important that the decoder elements in vorbis can convert a
|
|
granulepos into a timestamp or never seek on timestamp on the oggdemuxer.
|
|
|
|
The default format on the oggdemuxer source pads is currently defined as a the
|
|
granulepos of the packets, it is also the value of the OFFSET field in the
|
|
GstBuffer.
|
|
|
|
MUXING
|
|
======
|
|
|
|
Oggmux
|
|
------
|
|
|
|
The ogg muxer's job is to output complete Ogg pages such that the absolute
|
|
time represented by the valid (ie, not -1) granulepos values on those pages
|
|
never decreases. This has to be true for all logical streams in the group at
|
|
the same time.
|
|
|
|
To achieve this, encoders are required to pass along the exact time that the
|
|
granulepos represents for each ogg packet that it pushes to the ogg muxer.
|
|
This is ESSENTIAL: without this exact time representation of the granulepos,
|
|
the muxer can not produce valid streams.
|
|
|
|
The ogg muxer has a packet queue per sink pad. From this queue a page can
|
|
be flushed when:
|
|
- total byte size of queued packets exceeds a given value
|
|
- total time duration of queued packets exceeds a given value
|
|
- total byte size of queued packets exceeds maximum Ogg page size
|
|
- eos of the pad
|
|
- encoder sent a command to flush out an ogg page after this new packet
|
|
(in 0.8, through a flush event; in 0.10, with a GstOggBuffer)
|
|
- muxer wants a flush to happen (so it can output pages)
|
|
|
|
The ogg muxer also has a page queue per sink pad. This queue collects
|
|
Ogg pages from the corresponding packet queue. Each page is also marked
|
|
with the timestamp that the granulepos in the header represents.
|
|
|
|
A page can be flushed from this collection of page queues when:
|
|
- ideally, every page queue has at least one page with a valid granulepos
|
|
-> choose the page, from all queues, with the lowest timestamp value
|
|
- if not, muxer can wait if the following limits aren't reached:
|
|
- total byte size of any page queue exceeds a limit
|
|
- total time duration of any page queue exceeds a limit
|
|
- if this limit is reached, then:
|
|
- request a page flush from packet queue to page queue for each queue
|
|
that does not have pages
|
|
- now take the page from all queues with the lowest timestamp value
|
|
- make sure all later-coming data is marked as old, either to be still
|
|
output (but producing an invalid stream, though it can be fixed later)
|
|
or dropped (which means it's gone forever)
|
|
|
|
The oggmuxer uses the offset fields to fill in the granulepos in the pages.
|
|
|
|
GStreamer implementation details
|
|
--------------------------------
|
|
As said before, the basic rule is that the ogg muxer needs an exact time
|
|
representation for each granulepos. This needs to be provided by the encoder.
|
|
|
|
Potential problems are:
|
|
- initial offsets for a raw stream need to be preserved somehow. Example:
|
|
if the first audio sample has time 0.5, the granulepos in the vorbis encoder
|
|
needs to be adjusted to take this into account.
|
|
- initial offsets may need be on rate boundaries. Example:
|
|
if the framerate is 5 fps, and the first video frame has time 0.1 s, the
|
|
granulepos cannot correctly represent this timestamp.
|
|
This can be handled out-of-band (initial offset in another muxing format,
|
|
skeleton track with initial offsets, ...)
|
|
|
|
Given that the basic rule for muxing is that the muxer needs an exact timestamp
|
|
matching the granulepos, we need some way of communicating this time value
|
|
from encoders to the Ogg muxer. So we need a mechanism to communicate
|
|
a granulepos and its time representation for each GstBuffer.
|
|
|
|
(This is an instance of a more generic problem - having a way to attach
|
|
more fields to a GstBuffer)
|
|
|
|
Possible ways:
|
|
- setting TIMESTAMP to this value: bad - this value represents the end time
|
|
of the buffer, and thus conflicts with GStreamer's idea of what TIMESTAMP
|
|
is. This would cause problems muxing the encoded stream in other muxing
|
|
formats, or for streaming. Note that this is what was done in GStreamer 0.8
|
|
- setting DURATION to GP_TIME - TIMESTAMP: bad - this breaks the concept of
|
|
duration for this frame. Take the video example above; each buffer would
|
|
have a correct timestamp, but always a 0.1 s duration as opposed to the
|
|
correct 0.2 s duration
|
|
- subclassing GstBuffer: clean, but requires a common header used between
|
|
ogg muxer and all encoders that can be muxed into ogg. Also, what if
|
|
a format can be muxed into more than one container, and they each have
|
|
their own "extra" info to communicate ?
|
|
- adding key/value pairs to GstBuffer: clean, but requires changes to
|
|
core. Also, the overhead of allocating e.g. a GstStructure for *each* buffer
|
|
may be expensive.
|
|
- "cheating":
|
|
- abuse OFFSET to store the timestamp matching this granulepos
|
|
- abuse OFFSET_END to store the granulepos value
|
|
The drawback here is that before, it made sense to use OFFSET and OFFSET_END
|
|
to store a byte count. Given that this is not used for anything critical
|
|
(you can't store a raw theora or vorbis stream in a file anyway),
|
|
this is what's being done for now.
|
|
|
|
In practice
|
|
-----------
|
|
- all encoders of formats that can be muxed into Ogg produce a stream where:
|
|
- OFFSET is abused to be the timestamp corresponding exactly to the
|
|
granulepos
|
|
- OFFSET_END is abused to be the granulepos of the encoded theora buffer
|
|
- TIMESTAMP is the timestamp matching the begin of the buffer
|
|
- DURATION is the length in time of the buffer
|
|
|
|
- initial delays should be handled in the GStreamer encoders by mangling
|
|
the granulepos of the encoded packet to take the delay into account as
|
|
best as possible and store that in OFFSET;
|
|
this then brings TIMESTAMP + DURATION to within less
|
|
than a frame period of the granulepos's time representation
|
|
The ogg muxer will then create new ogg packets with this OFFSET as
|
|
the granulepos. So in effect, the granulepos produced by the encoders
|
|
does not get used directly.
|
|
|
|
TODO
|
|
----
|
|
- decide on a proper mechanism for communicating extra per-buffer fields
|
|
- the ogg muxer sets timestamp and duration on outgoing ogg pages based on
|
|
timestamp/duration of incoming ogg packets.
|
|
Note that:
|
|
- since the ogg muxer *has* to output pages sorted by gp time, representing
|
|
end time of the page, this means that the buffer's timestamps are not
|
|
necessarily monotonically increasing
|
|
- timestamp + duration of buffers don't match up; the duration represents
|
|
the length of the ogg page *for that stream*. Hence, for a normal
|
|
two-stream file, the sum of all durations is twice the length of the
|
|
muxed file.
|
|
|
|
TESTING
|
|
-------
|
|
Proper muxing can be tested by generating test files with command lines like:
|
|
- video and audio start from 0:
|
|
gst-launch -v videotestsrc ! theoraenc ! oggmux audiotestsrc ! audioconvert ! vorbisenc ! identity ! oggmux0. oggmux0. ! filesink location=test.ogg
|
|
|
|
- video starts after audio:
|
|
gst-launch -v videotestsrc timestamp-offset=500000000 ! theoraenc ! oggmux audiotestsrc ! audioconvert ! vorbisenc ! identity ! oggmux0. oggmux0. ! filesink location=test.ogg
|
|
|
|
- audio starts after video:
|
|
gst-launch -v videotestsrc ! theoraenc ! oggmux audiotestsrc timestamp-offset=500000000 ! audioconvert ! vorbisenc ! identity ! oggmux0. oggmux0. ! filesink location=test.ogg
|
|
|
|
The resulting files can be verified with oggz-validate for correctness.
|