diff --git a/ChangeLog b/ChangeLog index 6685239cc2..1f77d8aaf2 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,11 @@ +2005-05-24 Wim Taymans + + * docs/design/draft-ghostpads.txt: + * docs/design/draft-push-pull.txt: + * docs/design/draft-query.txt: + * docs/design/part-overview.txt: + Docs updates, added general overview doc. + 2005-05-21 David Schleef * docs/gst/tmpl/old/GstBin.sgml: diff --git a/docs/design/draft-ghostpads.txt b/docs/design/draft-ghostpads.txt new file mode 100644 index 0000000000..3794392c9c --- /dev/null +++ b/docs/design/draft-ghostpads.txt @@ -0,0 +1,108 @@ +Ghostpads +--------- + +Status: + + DRAFT. + + +Purpose: + + To create compound elements that look and behave like real elements + one needs to be able to expose pads on the element that are not + implemented by the element itself but by one of its components. + + ex1: + + Consider an rtp receiver element. It uses 2 UDP input ports to + capture RTP and RTCP messages using UDP. These messages are + then processed by an rtpsession element, which will reorder, make + statistics, and generate the RTP and RTCP messages. + + This element can be build from existing elements as shown in the + figure below. + + +------------------------------------+ + | rtpsrc | + | | + | +--------+ +---------+ | + | | udpsrc | | rtpsess | | + | | port1 --- ---------------- + | +--------+ | | | + | +--------+ | | +---------+ | + | | udpsrc | | | | udpsink | | + | | port2 --- --- port3 | | + | +--------+ +---------+ +---------+ | + +------------------------------------+ + + The element has one output pad that contains the raw RTP data. This + pad will be connected to the next element that will decode the RTP + data. Since this pad is actually from the rtpsession element, there + has to be a way to expose this pad on the rtpsrc element. + + +Current implementation: + + Version 0.8 creates a new GstGhostPad type that extends GstPad and has + a link to the real pad internally. + + Any operation on a pad potentially requires to check if this pad is a + GhostPad and if so, follow the pointer to the real pad to perform the + pad operation. + + +Current problem: + + - following the ghostpad to resolve the real pad adds code. + - operations on pads are not performed on the ghostpad but on the + real pad. This means: + - pads are not linked to a ghostpad but to the real pad. For + compound objects, the link is not really performed to the + pad of the compound element but with some internal pad. + - for state changes, it is hard to follow linked elements upstream + bacause pads enter bins to the real element. + + +Proposal: + + - The pad still has one parent, the element that owns (has sinked) the + pad. the GST_OBJECT_PARENT() is the real parent. + + - The pad receives a GList of ghostparents. Ghostparents are sorted by depth. + + - When a pad is ghostparented to another element, the element is added to + the ghostparent list of the pad. The pad is added to the padlist of the + element. + + - A pad cannot be ghostparented to an element that is not a parent of the + pad parent element. this ensures that all ghostpads do not skip an hierarchy. + + - When a pad is removed from the parent element, it is also removed from all + the ghostparents. This is easy to do as you can follow the ghostparent list of + that pad. + + - When an element is removed from another element, all the elements pads + ghostparented to parents of the element are removed as well. This is easy + to do with the sorted ghostparent list of the pad. + + +Consequences: + + - All pads are Real pads. + - All operations are performed on the real pads. This improves performance + and code simplicity + - more checks can be done. Linking nested elements can be done sanely. + compount elements that expose usable pads must ghostpad them before they + can be linked. + - All linked pads have a common grandparent. + + +Issues: + + - Name of ghostpad cannot be changed and is the name of the element. This could + be solved by temporarily renaming the pad when ghostpadding. This could then + again result in the element being confused about the padnames or the + element having two pads with the same name. + + + diff --git a/docs/design/draft-push-pull.txt b/docs/design/draft-push-pull.txt index e56ce5f6c1..739373c913 100644 --- a/docs/design/draft-push-pull.txt +++ b/docs/design/draft-push-pull.txt @@ -1,6 +1,10 @@ DRAFT push-pull scheduling -------------------------- +Status + + DRAFT + Observations: - The main scheduling mode is chain based scheduling where the source @@ -34,12 +38,13 @@ Types of pull based scheduling: - some sources can do random access (file source, ...) - some sources can read a random number of bytes but not at a random - offset. (audio cards, ...) + offset. (audio cards, ...) Audio cards using a ringbuffer can + however do random access in the ringbuffer. - some sources can do random access in a range of bytes but not in another range. (a caching network source). - - some sources can do a fixed size data but without an offset. + - some sources can do a fixed size data and without an offset. (video sources, ...) @@ -54,7 +59,8 @@ Current scheduling decision: Problems: - core makes a touch desicion without knowing anything about the - element. + element. Some elements are able to deal with a pull_range() + without offset while others need full random access. Requirements: diff --git a/docs/design/draft-query.txt b/docs/design/draft-query.txt index c7dcb48a8d..2e6b445de7 100644 --- a/docs/design/draft-query.txt +++ b/docs/design/draft-query.txt @@ -1,6 +1,10 @@ DRAFT Query ----------- +Status + + Implemented, move me to design... + Purpose Queries are used to get information about the stream. diff --git a/docs/design/part-overview.txt b/docs/design/part-overview.txt new file mode 100644 index 0000000000..5a25af78f2 --- /dev/null +++ b/docs/design/part-overview.txt @@ -0,0 +1,500 @@ +Overview +-------- + + This part gives an overview of the design of GStreamer with references to + the more detailed explanations of the different topics. + + This document is intented for people that want to have a global overview of + the inner workings of GStreamer. + + +Introduction +------------ + + GStreamer is a set of libraries and plugins that can be used to implement various + multimedia applications ranging from desktop players, audio/video recorders, + multimedia servers, transcoders, etc. + + Applications are built by constructing a pipeline composed of elements. An element + is an object that performs some action on a multimedia stream such as: + + - read a file + - decode or encoder between formats + - capture from a hardware device + - render to a hardware device + - mix or multiplex multiple streams + + Elements have input and output pads called sink and source pads in GStreamer. An + application links elements together on pads to construct a pipeline. Below is + an example of an ogg/vorbis playback pipeline. + + +-----------------------------------------------------------+ + | pipeline | + | +---------+ +----------+ +-----------+ +----------+ | + | | filesrc | | oggdemux | | vorbisdec | | alsasink | | + | | src-sink src-sink src-sink | | + | +---------+ +----------+ +-----------+ +----------+ | + +-----------------------------------------------------------+ + + The filesrc element reads data from a file on disk. The oggdemux element parses + the data and sends the compressed audio data to the vorbisdec element. The + vorbisdec element decodes the compressed data and sends it to the alsasink + element. The alsasink element sends the samples to the audio card for playback. + + The task of the application is to construct a pipeline as above using existing + elements. This is further explained in the pipeline building topic. + + The application does not have to manage any of the complexities of the + actual dataflow/decoding/conversions/synchronsiation etc. but only calls high + level functions on the pipeline object such as PLAY/PAUSE/STOP. + + The application also receives messages and notifications from the pipeline such + as metadata, warning or error messages. + + If the application needs more control over the graph it is possible to directly + access the elements and pads in the pipeline. + + +Design overview +--------------- + + GStreamer design goals include: + + - Process large amounts of data quickly + - Allow fully multithreaded processing + - Ability to deal with multiple formats + - Synchronize different dataflows + - Ability to deal with multiple devices + + The capabilities presented to the application depends on the number of elements + installed on the system and their functionality. + + The GStreamer core is designed to be media agnostic but provides many features + to elements to describe media formats. + + +Elements +-------- + + The smallest building blocks in a pipeline are elements. An element provides a + number of pads which can be source or sinkpads. Sourcepads provide data and + sinkpads consume data. Below is an example of an ogg demuxer element that has + one pad that takes (sinks) data and two source pads that produce data. + + +-----------+ + | oggdemux | + | src0 + sink src1 + +-----------+ + + An element can be in four different states: NULL, READY, PAUSED, PLAYING. In the + NULL and READY state, the element is not processing any data. In the PLAYING state + it is processing data. The intermediate PAUSED state is used to preroll data in + the pipeline. A state change can be performed with gst_element_set_state(). + + An element always goes through all the intermediate state changes. This means that + when en element is in the READY state and is put to PLAYING, it will first go + through the intermediate PAUSED state. + + An element state change to PAUSED will activate the pads of the element. First the + source pads are activated, then the sinkpads. When the pads are activated, the + pad activate function is called. Some pads will start a thread or some other + mechanism to start producing or consuming data. + + The PAUSED state is special as it is used to preroll data in the pipeline. The purpose + is to fill all connected elements in the pipeline with data so that the subsequent + PLAYING state change happens very quickly. Some elements will therefore not complete + the state change to PAUSED before they have received enough data. Sink elements are + required to only complete the state change to PAUSED after receiving the first data. + + Normally the state changes of elements are coordinated by the pipeline as explained + in [part-states.txt]. + + Different categories of elements exist: + + - source elements, these are elements that do not consume data but only provide data + for the pipeline. + - sink elements, these are elements that do not produce data but renders data to + an output device. + - transform elements, these elements transform an input stream in a certain format + into a stream of another format. Encoder/decoder/converters are examples. + - demuxer elements, these elements parse a stream and produce several output streams. + - mixer/muxer elements, combine several input streams into one output stream. + + Other categories of elements can be constructed. + + +Bins +---- + + A bin is an element subclass and acts as a container for other elements so that multiple + elements can be combined into one element. + + A bin coordinates its children's state changes as explained later. It also distributes + events and various other functionality to elements. + + A bin can have its own source and sinkpads by ghostpadding one or more of its children's + pads to itself. + + Below is a picture of a bin with two elements. The sinkpad of one element is ghostpadded + to the bin. + + +---------------------------+ + | bin | + | +--------+ +-------+ | + | | | | | | + | /sink src-sink | | + sink +--------+ +-------+ | + +---------------------------+ + + +Pipeline +-------- + + A pipeline is a special bin subclass that provides the following features to its + children: + + - Select and manage a clock + - Provide means for elements to comunicate with the application by the Bus. + - Manage the global state of the elements such as Errors and end-of-stream. + + Normally the application creates one pipeline that will manage all the elements + in the application. + + +Dataflow and buffers +-------------------- + + GStreamer supports two possible types of dataflow, the push and pull model. In the + push model, an upstream element sends data to a downstream element by calling a + method on a sinkpad. In the pull model, a downstream element requests data from + an upstream element by calling a method on a source pad. + + The most common dataflow is the push model. The pull model can be used in specific + circumstances by demuxer elements. The pull model can also be used by low latency + audio applications. + + The data passed between pads is encapsulated in Buffers. The buffer contains a + pointer to the actual data and also metadata describing the data. This metadata + includes: + + - timestamp of the data, this is the time instance at which the data was captured + or the time at which the data should be played back. + - offset of the data: a media specific offset, this could be samples for audio or + frames for video. + - the duration of the data in time. + - the media type of the data described with caps, these are key/value pairs that + describe the media type in a unique way. + + When an element whishes to send a buffer to another element is does this using one + of the pads that is linked to a pad of the other element. In the push model, a + buffer is pushed to the peer pad with gst_pad_push(). In the pull model, a buffer + is pulled from the peer with the gst_pad_pull_region() function. + + Before an element pushes out a buffer, it should make sure that the peer element + can understand the buffer contents. It does this by querying the peer element + for the supported formats and by selecting a suitable common format. The selected + format is then attached to the buffer with gst_buffer_set_caps() before pushing + out the buffer. + + When an element pad receives a buffer, if has to check if it understands the media + type of the buffer before starting processing it. + + Both gst_pad_push() and gst_pad_pull_range() have a return value indicating wether + the operation succeeded. An error code means that no more data should be send + to that pad. A source element that initiates the data flow in a thread typically + pauses the producing thread when this happens. + + A buffer can be created with gst_buffer_new() or by requesting a usable buffer + from the peer pad using gst_pad_alloc_buffer(). Using the second method, it is + possible for the peer element to suggest the element to produce data in another + format by attaching another media type caps to the buffer. + + The process of selecting a media type and attaching it to the buffers is called + caps negotiation. + + +Caps +---- + + A media type (Caps) is described using a generic list of key/value pairs. The key is + a string and the value can be a single/list/range of int/float/string. + + Caps that have no ranges/list or other variable parts are said to be fixed and + can be used to put on a buffer. + + Caps with variables in them are used to describe possible media types that can be + handled by a pad. + + +Dataflow and events +------------------- + + Parallel to the dataflow is a flow of events. Unlike the buffers, events can pass + upstream and downstream. Some events only travel upstream others only downstream. + + the events are used to denote special conditions in the dataflow such as EOS or + to inform plugins of special events such as flushing or seeking. + + +Pipeline construction +--------------------- + + The application starts by creating a Pipeline element using gst_pipeline_new (). + Elements are added to and removed from the pipeline with gst_bin_add() and + gst_bin_remove(). + + After adding the elements, the pads of an element can be retrieved with + gst_element_get_pad(). Pads can then be linked together with gst_pad_link(). + + Some elements create new pads when actual dataflow is happening in the pipeline. + With g_signal_connect() one can receive a notification when an element has created + a pad. These new pads can then be linked to other unlinked pads. + + Some elements cannot be linked together because they operate on different + incompatible data types. The possible datatypes a pad can provide or consume can + be retrieved with gst_pad_get_caps(). + + Below is a simple mp3 playback pipeline that we constructed. We will use this + pipeline in further examples. + + +-------------------------------------------+ + | pipeline | + | +---------+ +----------+ +----------+ | + | | filesrc | | mp3dec | | alsasink | | + | | src-sink src-sink | | + | +---------+ +----------+ +----------+ | + +-------------------------------------------+ + + +Pipeline clock +-------------- + + One of the important functions of the pipeline is to select a global clock + for all the elements in the pipeline. + + Before the pipeline is set to PAUSED, the pipeline asks each element if they can + provide a clock. The clock is selected in the following order: + + - If the application selected a clock, us that one. + - If a source element provides a clock, use that clock. + - Select a clock from any other element that provides a clock, start with the + sinks. + - If no element provides a clock a default system clock is used for the pipeline. + + In a typical playback pipeline this will select the clock provided by a sink element + such as an audio sink. + + +Pipeline states +--------------- + + When all the pads are linked or signals have been connected, the pipeline can + be put in the PAUSED state to start dataflow. + + When a bin (and hence a pipeline) performs a state change, it will change the state + of all its children. The pipeline will change the state of its children from the + sink elements to the source elements, this to make sure that no upstream element + produces data to an element that is not yet ready to accept it. + + In the mp3 playback pipeline, the state of the elements is changed in the order + alsasink, mp3dec, filesrc. + + All intermediate states are traversed for each element resulting in the following + chain of state changes: + + alsasink to READY: the audio device is opened + mp3dec to READY: the decoding library is initialized + filesrc to READY: the file is opened + alsasink to PAUSED: alsasink is a sink and returns ASYNC because it did not receive + data yet. + mp3dec to PAUSED: nothing happens + filesrc to PAUSED: a thread is started to push data to mp3dec + + At this point data flows from filesrc to mp3dec and alsasink. Since mp3dec is PAUSED, + it accepts the data from filesrc on the sinkpad and starts decoding the compressed + data to raw audio samples. + + The mp3 decoder figures out the samplerate, the number of channels and other audio + properties of the raw audio samples, puts the decoded samples into a Buffer, + attaches the media type caps to the buffer and pushes this buffer to the next + element. + + Alsasink then receives the buffer, inspects the caps and reconfigures itself to process + the buffer. Since it received the first buffer of samples, it completes the state change + to the PAUSED state. At this point the pipeline is prerolled and all elements have + samples. + + Since alsasink is now in the PAUSED state it blocks while receiving the first buffer. This + effectively blocks both mp3dec and filesrc in their gst_pad_push(). + + Since all elements now return SUCCESS from the gst_element_get_state() function, + the pipeline can be put in the PLAYING state. + + Before going to PLAYING, the pipeline samples the current time of the clock. This is + the base time. It then distributes this time to all elements. Elements can then + synchronize against the clock using the buffer timestamp+base time. + + The following chain of state changes then takes place: + + alsasink to PLAYING: the samples are played to the audio device + mp3dec to PLAYING: nothing happens + filesrc to PLAYING: nothing happens + + +Pipeline status +--------------- + + The pipeline informs the application of any special events that occur in the + pipeline with the bus. The bus is an object that the pipeline provides and that + can be retrieved with gst_pipeline_get_bus(). + + The bus can be polled or added to the glib mainloop. + + The bus is distributed to all elements added to the pipeline. The elements use the bus + to post messages on. Various message types exist such as ERRORS, WARNINGS, EOS, + STATE_CHANGED, etc.. + + The pipeline handles EOS messages received from elements in a special way. It will + only forward the message to the application when all sink elements have posted an + EOS message. + + Other methods for obtaining the pipeline status include the Query functionality that + can be performed with gst_element_query() on the pipeline. This type of query + is usefull for obtaining information about the current position and total time of + the pipeline. It can also be used to query for the supported seeking formats and + ranges. + + +Pipeline EOS +------------ + + When the source filter encounters the end of the stream, it sends an EOS event to + the peer element. This event will then travel downstream to all of the connected + elements to inform them of the EOS. The element is not supposed to accept any more + data after receiving an EOS event on a sinkpad. + + The source element stops sending data after sending the EOS message. + + The EOS even will eventually arrive in the sink element. The sink will then post + an EOS message on the bus to inform the pipeline that a particular stream has + finished. When all sinks have reported EOS, the pipeline forwards the EOS message + to the application. + + When in EOS, the pipeline remains in the playing state, if is the application + responsability to PAUSE or READY the pipeline. + + +Pipeline READY +-------------- + + When a running pipeline is set from the RUNNING to READY the following actions + occur in the pipeline: + + alsasink to PAUSED: alsasink blocks and completes the state change on the + next sample. If the element was EOS, it does not wait for + a sample to complete the state change. + mp3dec to PAUSED: nothing + filesrc to PAUSED: nothing + + Going to the intermediate PAUSED state will block all elements in the _push() + functions. + + Some elements might be performing blocking operations in the PLAYING state that + must be unblocked when they go into the PAUSED state. This makes sure that the + state change happens very fast. + + In the next PAUSED to READY state change the pipeline has to shut down and all + streaming threads must stop sending data. This happens in the following sequence: + + alsasink to READY: alsasink unblocks from the _chain() function and returns a + WRONG_STATE return value to the peer element. The sinkpad is + deactivated and becomes unusable for sending more data. + mp3dec to READY: the pads are deactivated and the state change completes when + mp3dec leaves its _chain() function. + filesrc to PAUSED: the pads are deactivated and the thread is paused. + + The upstream elements finish their chain() function because the downstream element + returned an error code from the _push() functions. These error codes are eventually + returned to the element that started the streaming thread (filesrc), which pauses + the thread and completes the state change. + + This sequence of events ensure that all elements are unblocked and all streaming + threads stopped. + + +Pipeline seeking +---------------- + + Seeking in the pipeline requires a very specific order of operations to make + sure that the elements remain synchronized and that the seek is performed with + a minimal amount of latency. + + An application issues a seek event on the pipeline using gst_element_send_event() + on the pipeline element. The event can be a seek event in any of the formats + supported by the elements. + + The pipeline first pauses the pipeline to speed up the seek operations. + + The pipeline then issues the seek event to all sink elements. The sink then forwards + the seek event upstream until some element can perform the seek operation, which is + typically the source or demuxer element. All intermediate elements can transform the + requested seek offset to another format, this way a decoder element can transform a + seek to a frame number to a timestamp, for example. + + When the seek event reaches an element that will perform the seek operation, that + element performs the following steps. + + 1) send a FLUSH event to all downstream and upstream peer elements. + 2) make sure the streaming thread is not running. + 3) perform the seek operation + 4) send a FLUSH done event to all downstream and upstream peer elements. + 5) send DISCONT event to inform all elements of the new position and to complete + the seek. + + In step 1) all dowstream elements have to return from any blocking operations + and have to refuse any further buffers or events different from a FLUSH done. + + The first step ensures that the streaming thread eventually unblocks and that + step 2) can be performed. At this point, dataflow is completely stopped in the + pipeline. + + In step 3) the element performs the seek to the requested position. + + In step 4) all peer elements are allowed to accept data again and streaming + can continue from the new position. A FLUSH done event is sent to all the peer + elements so that they accept new data again and restart their streaming threads. + + Step 5) informs all elements of the new position in the stream. After that the + event function returns back to the application. and the streaming threads start + to produce new data. + + Since the pipeline is still PAUSED, this will preroll the next media sample in the + sinks. + + The last step in the seek operation is then to adjust the media time of the pipeline + to the new position and to set the pipeline back to PLAYING. + + The sequence of events in out mp3 playback example. + + | a) seek on pipeline + | b) PAUSE pipeline + +----------------------------------V--------+ + | pipeline | c) seek on sink + | +---------+ +----------+ +---V------+ | + | | filesrc | | mp3dec | | alsasink | | + | | src-sink src-sink | | + | +---------+ +----------+ +----|-----+ | + +-----------------------------------|-------+ + <------------------------+ + d) seek travels upstream + + --------------------------> 1) FLUSH event + | 2) stop streaming + | 3) perform seek + --------------------------> 4) FLUSH done event + --------------------------> 5) DISCONT event + + | e) update stream time + | f) PLAY pipeline +