diff --git a/ChangeLog b/ChangeLog index 8ae0249386d..b9ca274a6cc 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,18 @@ +2005-07-20 Ronald S. Bultje + + * docs/pwg/advanced-clock.xml: + * docs/pwg/appendix-porting.xml: + * docs/pwg/intro-preface.xml: + * docs/pwg/other-base.xml: + * docs/pwg/other-manager.xml: + * docs/pwg/other-nton.xml: + * docs/pwg/other-ntoone.xml: + * docs/pwg/other-oneton.xml: + * docs/pwg/pwg.xml: + Document base classes, update sections of n-to-1 and 1-to-n (muxer, + demuxer), remove n-to-n (was never written), fix some code examples + and links and update the porting section to include all this. + 2005-07-19 Wim Taymans * gst/gstqueue.c: (gst_queue_init), (gst_queue_handle_sink_event), diff --git a/docs/pwg/advanced-clock.xml b/docs/pwg/advanced-clock.xml index 78f19098ce7..7b0ef56eae2 100644 --- a/docs/pwg/advanced-clock.xml +++ b/docs/pwg/advanced-clock.xml @@ -135,7 +135,6 @@ under discussion, and might change in a future release. - See an example in diff --git a/docs/pwg/appendix-porting.xml b/docs/pwg/appendix-porting.xml index b96d13acac7..f0160620e96 100644 --- a/docs/pwg/appendix-porting.xml +++ b/docs/pwg/appendix-porting.xml @@ -8,7 +8,12 @@ With this list, it should be possible to port most plugins to &GStreamer;-0.9 in less than a day. Exceptions are elements that will require a base class in 0.9 (sources, sinks), in which case it may take - a lot longer, depending on the coder's skills. + a lot longer, depending on the coder's skills (however, when using the + GstBaseSink and GstBaseSrc + base-classes, it shouldn't be all too bad), and elements requiring + the deprecated bytestream interface, which should take 1-2 days with + random access. The scheduling parts of muxers will also need a rewrite, + which will take about the same amount of time. @@ -54,7 +59,36 @@ - base classes, async state changes. + For as far as possible, elements should try to use existing base + classes in 0.9. Sink and source elements, for example, could derive + from GstBaseSrc and + GstBaseSink. Audio sinks or sources could even + derive from audio-specific base classes. All existing base classes + have been discussed in and the + next few chapters. + + + + + In 0.9, event handling and buffers are separated once again. This + means that in order to receive events, one no longer has to set the + GST_FLAG_EVENT_AWARE flag, but can simply + set an event handling function on its sinkpad(s), using the function + gst_pad_set_event_function (). The + _chain ()-function will only receive buffers. + + + + + Although core will wrap most threading-related locking for you (e.g. + it takes the stream lock before calling your data handling + functions), you are still responsible for locking around certain + functions, e.g. object properties. Be sure to lock properly here, + since applications will change those properties in a different thread + than the thread which does the actual data passing! You can use the + GST_LOCK () and GST_UNLOCK + () helpers in most cases, fortunately, which grabs the + default property lock of the element. diff --git a/docs/pwg/intro-preface.xml b/docs/pwg/intro-preface.xml index 9fd999a32d1..975fe4544d6 100644 --- a/docs/pwg/intro-preface.xml +++ b/docs/pwg/intro-preface.xml @@ -197,9 +197,10 @@ the concepts apply equally to other plugin types, including sources, sinks, and autopluggers. This part of the guide presents the issues that arise when working on these more specialized plugin types. The - part includes chapters on , - , , ), and + later also goes into writing special types of elements in + , and . diff --git a/docs/pwg/other-base.xml b/docs/pwg/other-base.xml new file mode 100644 index 00000000000..ec3fc3e9fab --- /dev/null +++ b/docs/pwg/other-base.xml @@ -0,0 +1,327 @@ + + Pre-made base classes + + So far, we've been looking at low-level concepts of creating any type of + &GStreamer; element. Now, let's assume that all you want is to create an + simple audiosink that works exactly the same as, say, + esdsink, or a filter that simply normalizes audio volume. + Such elements are very general in concept and since they do nothing + special, they should be easier to code than to provide your own scheduler + activation functions and doing complex caps negotiation. For this purpose, + &GStreamer; provides base classes that simplify some types of elements. + Those base classes will be discussed in this chapter. + + + + Writing a sink + + Sinks are special elements in &GStreamer;. This is because sink elements + have to take care of preroll, which is the process + that takes care that elements going into the + GST_STATE_PAUSED state will have buffers ready + after the state change. The result of this is that such elements can + start processing data immediately after going into the + GST_STATE_PLAYING state, without requiring to + take some time to initialize outputs or set up decoders; all that is done + already before the state-change to GST_STATE_PAUSED + successfully completes. + + + Preroll, however, is a complex process that would require the same + code in many elements. Therefore, sink elements can derive from the + GstBaseSink base-class, which does preroll and + a few other utility functions automatically. The derived class only + needs to implement a bunch of virtual functions and will work + automatically. + + + The GstBaseSink base-class specifies some + limitations on elements, though: + + + + + It requires that the sink only has one sinkpad. Sink elements that + need more than one sinkpad, cannot use this base-class. + + + + + The base-class owns the pad, and specifies caps negotiation, data + handling, pad allocation and such functions. If you need more than + the ones provided as virtual functions, then you cannot use this + base-class. + + + + + By implementing the pad_allocate () function, + it is possible for upstream elements to use special memory, such + as memory on the X server side that only the sink can allocate, or + even hardware memory mmap ()'ed from the kernel. + Note that in almost all cases, you will want to subclass the + GstBuffer object, so that your own set of + functions will be called when the buffer loses its last reference. + + + + + Sink elements can derive from GstBaseSink using + the usual GObject type creation voodoo, or by + using the convenience macro GST_BOILERPLATE (): + + +GST_BOILERPLATE_FULL (GstMySink, gst_my_sink, GstBaseSink, GST_TYPE_BASE_SINK); + +[..] + +static void +gst_my_sink_class_init (GstMySinkClass * klass) +{ + klass->set_caps = [..]; + klass->render = [..]; +[..] +} + + + The advantages of deriving from GstBaseSink are + numerous: + + + + + Derived implementations barely need to be aware of preroll, and do + not need to know anything about the technical implementation + requirements of preroll. The base-class does all the hard work. + + + Less code to write in the derived class, shared code (and thus + shared bugfixes). + + + + + There are also specialized base classes for audio and video, let's look + at those a bit. + + + + Writing an audio sink + + Essentially, audio sink implementations are just a special case of a + general sink. There are two audio base classes that you can choose to + derive from, depending on your needs: + GstBaseAudiosink and + GstAudioSink. The baseaudiosink provides full + control over how synchronization and scheduling is handled, by using + a ringbuffer that the derived class controls and provides. The + audiosink base-class is a derived class of the baseaudiosink, + implementing a standard ringbuffer implementing default + synchronization and providing a standard audio-sample clock. Derived + classes of this base class merely need to provide a _open + (), _close () and a _write + () function implementation, and some optional functions. + This should suffice for many sound-server output elements and even + most interfaces. More demanding audio systems, such as Jack, would + want to implement the GstBaseAudioSink + base-class. + + + The GstBaseAusioSink has little to no + limitations and should fit virtually every implementation, but is + hard to implement. The GstAudioSink, on the + other hand, only fits those systems with a simple open + () / close () / write + () API (which practically means pretty much all of them), + but has the advantage that it is a lot easier to implement. The + benefits of this second base class are large: + + + + + Automatic synchronization, without any code in the derived class. + + + + + Also automatically provides a clock, so that other sinks (e.g. in + case of audio/video playback) are synchronized. + + + + + Features can be added to all audiosinks by making a change in the + base class, which makes maintainance easy. + + + + + Derived classes require only three small functions, plus some + GObject boilerplate code. + + + + + In addition to implementing the audio base-class virtual functions, + derived classes can (should) also implement the + GstBaseSink set_caps () and + get_caps () virtual functions for negotiation. + + + + + Writing a video sink + + Writing a videosink can be done using the + GstVideoSink base-class, which derives from + GstBaseSink internally. Currently, it does + nothing yet but add another compile dependency, so derived classes + will need to implement all base-sink virtual functions. When they do + this correctly, this will have some positive effects on the end user + experience with the videosink: + + + + + Because of preroll (and the preroll () virtual + function), it is possible to display a video frame already when + going into the GST_STATE_PAUSED state. + + + + + By adding new features to GstVideoSink, it + will be possible to add extensions to videosinks that affect all of + them, but only need to be coded once, which is a huge maintainance + benefit. + + + + + + + + Writing a source + + In the previous part, particularly , we have learned that some types + of elements can provide random access. This applies most definitely to + source elements reading from a randomly seekable location, such as file + sources. However, other source elements may be better described as a + live source element, such as a camera source, an audio card source and + such; those are not seekable and do not provide byte-exact access. For + all such use cases, &GStreamer; provides two base classes: + GstBaseSrc for the basic source functionality, and + GstPushSrc, which is a non-byte exact source + base-class. The pushsource base class itself derives from basesource as + well, and thus all statements about the basesource apply to the + pushsource, too. + + + The basesrc class does several things automatically for derived classes, + so they no longer have to worry about it: + + + + + Fixes to GstBaseSrc apply to all derived + classes automatically. + + + + + Automatic pad activation handling, and task-wrapping in case we get + assigned to start a task ourselves. + + + + + The GstBaseSrc may not be suitable for all cases, + though; it has limitations: + + + + + There is one and only one sourcepad. Source elements requiring + multiple sourcepads cannot use this base-class. + + + + + Since the base-class owns the pad and derived classes can only + control it as far as the virtual functions allow, you are limited + to the functionality provided by the virtual functions. If you need + more, you cannot use this base-class. + + + + + It is possible to use special memory, such as X server memory pointers + or mmap ()'ed memory areas, as data pointers in + buffers returned from the create() virtual function. + In almost all cases, you will want to subclass + GstBuffer so that your own set of functions can + be called when the buffer is destroyed. + + + + Writing an audio source + + An audio source is nothing more but a special case of a pushsource. + Audio sources would be anything that reads audio, such as a source + reading from a soundserver, a kernel interface (such as ALSA) or a + test sound / signal generator. &GStreamer; provides two base classes, + similar to the two audiosinks described in ; one is ringbuffer-based, and + requires the derived class to take care of its own scheduling, + synchronization and such. The other is based on this + GstBaseAudioSrc and is called + GstAudioSrc, and provides a simple + open (), close () and + read () interface, which is rather simple to + implement and will suffice for most soundserver sources and audio + interfaces (e.g. ALSA or OSS) out there. + + + The GstAudioSrc base-class has several benefits + for derived classes, on top of the benefits of the + GstPushSrc base-class that it is based on: + + + + + Does syncronization and provides a clock. + + + + + New features can be added to it and will apply to all derived + classes automatically. + + + + + + + + Writing a transformation element + + A third base-class that &GStreamer; provides is the + GstBaseTransform. This is a base class for + elements with one sourcepad and one sinkpad which act as a filter + of some sort, such as volume changing, audio resampling, audio format + conversion, and so on and so on. There is quite a lot of bookkeeping + that such elements need to do in order for things such as buffer + allocation forwarding, passthrough, in-place processing and such to all + work correctly. This base class does all that for you, so that you just + need to do the actual processing. + + + Since the GstBaseTransform is based on the 1-to-1 + model for filters, it may not apply well to elements such as decoders, + which may have to parse properties from the stream. Also, it will not + work for elements requiring more than one sourcepad or sinkpad. + + + diff --git a/docs/pwg/other-manager.xml b/docs/pwg/other-manager.xml index a4f1381589b..ebd1ace2ba3 100644 --- a/docs/pwg/other-manager.xml +++ b/docs/pwg/other-manager.xml @@ -1,6 +1,3 @@ - - - Writing a Manager @@ -30,25 +27,18 @@ function). + + + To embed an element, or a series of elements, into something that + looks and works like a simple element to the outside world. + + - This chapter will explain the setup of managers. As a specific example, - we will try to add EOS event support to source elements. This can be - used to finish capturing an audio stream to a file. Source elements - normally don't do any EOS handling at all, so a manager is perfect to - extend those element's functionalities. + Making a manager is about as simple as it gets. You can derive from a + GstBin, and in most cases, you can embed the + required elements in the _init () already, including + setup of ghostpads. If you need any custom data handlers, you can connect + signals or embed a second element which you control. - - Specifically, this element will contain two child elements: the actual - source element and a helper element that implement an - event handler on its source pad. This event handler will respond to - EOS events by storing them internally and returning the event (rather - than data) on the next call to the _get () - function. After that, it will go into EOS and set the parent (and - thereby the contained source element) to EOS as well. Other events will - be forwarded to the source element, which will handle them as usual. - - -.. - diff --git a/docs/pwg/other-nton.xml b/docs/pwg/other-nton.xml deleted file mode 100644 index 17d2b8d803e..00000000000 --- a/docs/pwg/other-nton.xml +++ /dev/null @@ -1,8 +0,0 @@ - - - - Writing a N-to-N element - - FIXME: write. - - diff --git a/docs/pwg/other-ntoone.xml b/docs/pwg/other-ntoone.xml index 9e9015213a9..5cb1f352b7e 100644 --- a/docs/pwg/other-ntoone.xml +++ b/docs/pwg/other-ntoone.xml @@ -1,82 +1,31 @@ - - - - + Writing a N-to-1 Element or Muxer N-to-1 elements have been previously mentioned and discussed in both and in . The main noteworthy thing - about N-to-1 elements is that they should always, - without any single exception, be _loop ()-based. - Apart from that, there is not much general that you need to know. We - will discuss one special type of N-to-1 elements here, these being - muxers. The first two of these sections apply to N-to-1 elements in - general, though. + about N-to-1 elements is that each pad is push-based in its own thread, + and the N-to-1 element synchronizes those streams by + expected-timestamp-based logic. This means it lets all streams wait + except for the one that provides the earliest next-expected timestamp. + When that stream has passwed one buffer, the next + earliest-expected-timestamp is calculated, and we start back where we + were, until all streams have reached EOS. There is a helper base class, + called GstCollectPads, that will help you to do + this. + + + Note, however, that this helper class will only help you with grabbing + a buffer from each input and giving you the one with earliest timestamp. + If you need anything more difficult, such as "don't-grab-a-new-buffer + until a given timestamp" or something like that, you'll need to do this + yourself. - - The Data Loop Function - - As previously mentioned in , - N-to-1 elements generally try to have one buffer from each sink pad - and then handle the one with the earliest timestamp. There's some - exceptions to this rule, we will come to those later. This only works - if all streams actually continuously provide input. There might be - cases where this is not true, for example subtitles (there might be - no subtitle for a while), overlay images and so forth. For this - purpose, there is a _select () function in - &GStreamer;. It checks whether input is available on a (list of) - pad(s). In this way, you can skip over the pads that are 'non- - continuous'. - - -/* Pad selection is currently broken, FIXME some day */ - - - - - Events in the Loop Function - - N-to-1 elements using a cache will sometimes receive events, and it - is often unclear how to handle those. For example, how do you seek - to a frame in an output file (and what's the - point of it anyway)? So, do discontinuity or seek events make sense, - and should you use them? - - - Discontinuities and flushes - - Don't do anything. They specify a discontinuity in the output, and - you should continue to playback as you would otherwise. You - generally do not need to put a discontinuity in the output stream - in muxers; you would have to manually start adapting timestamps of - output frames (if appliccable) to match the previous timescale, - though. Note that the output data stream should be continuous. For - other types of N-to-1-elements, it is generally fine to forward - the discontinuity once it has been received from all pads. This - depends on the specific element. - - - - Seeks - - Depends on the element. Muxers would generally not implement this, - because the concept of seeking in an output - stream at frame level is not very useful. Seeking at byte level - can be useful, but that is more generally done - by muxers on sink - elements. - - - - End-of-Stream - - Speaks for itself. - - - - + diff --git a/docs/pwg/other-oneton.xml b/docs/pwg/other-oneton.xml index 5d178f86cba..bc61bac4ff4 100644 --- a/docs/pwg/other-oneton.xml +++ b/docs/pwg/other-oneton.xml @@ -1,18 +1,7 @@ - - - - - Writing a 1-to-N Element, Demuxer or Parser + + Writing a Demuxer or Parser - 1-to-N elements don't have much special needs or requirements that - haven't been discussed already. The most important thing to take care - of in 1-to-N elements (things like tee-elements - or so) is to use proper buffer refcounting and caps negotiation. If - those two are taken care of (see the tee element - if you need example code), there's little that can go wrong. - - - Demuxers are the 1-to-N elements that need very special care, though. + Demuxers are the 1-to-N elements that need very special care. They are responsible for timestamping raw, unparsed data into elementary video or audio streams, and there are many things that you can optimize or do wrong. Here, several culprits will be mentioned @@ -20,132 +9,37 @@ one source pad. Also, they only cut the stream into buffers, they don't touch the data otherwise. - - - Demuxer Caps Negotiation - - Demuxers will usually contain several elementary streams, and each - of those streams' properties will be defined in a stream header at - the start of the file (or, rather, stream) that you're parsing. - Since those are fixed and there is no possibility to negotiate - stream properties with elements earlier in the pipeline, you should - always use explicit caps on demuxer source pads. This prevents a - whole lot of caps negotiation or re-negotiation errors. - - - - - Data processing and downstream events - - Data parsing, pulling this into subbuffers and sending that to the - source pads of the elementary streams is the one single most - important task of demuxers and parsers. Usually, an element will - have a _loop () function using the - bytestream object to read data. Try to have - a single point of data reading from the bytestream object. In this - single point, do proper event handling (in - case there is any) and proper error handling - in case that's needed. Make your element as fault-tolerant as - possible, but do not go further than possible. - - - - - Parsing versus interpreting - - One particular convention that &GStreamer; demuxers follow is that - of separation of parsing and interpreting. The reason for this is - maintainability, clarity and code reuse. An easy example of this - is something like RIFF, which has a chunk header of 4 bytes, then - a length indicator of 4 bytes and then the actual data. We write - special functions to read one chunk, to peek a chunk ID and all - those; that's the parsing part of the demuxer. - Then, somewhere else, we like to write the main data processing - function, which calls this parse function, reads one chunk and - then does with the data whatever it needs to do. - - - Some example code for RIFF-reading to illustrate the above two points: - - -static gboolean -gst_my_demuxer_peek (GstMyDemuxer *demux, - guint32 *id, - guint32 *size) -{ - guint8 *data; - - while (gst_bytestream_peek_bytes (demux->bs, &data, 4) != 4) { - guint32 remaining; - GstEvent *event; - - gst_bytestream_get_status (demux->bs, &remaining, &event); - if (event) { - GstEventType type = GST_EVENT_TYPE (event); - - /* or maybe custom event handling, up to you - we lose reference! */ - gst_pad_event_default (demux->sinkpad, event); - - if (type == GST_EVENT_EOS) - return FALSE; - } else { - GST_ELEMENT_ERROR (demux, STREAM, READ, (NULL), (NULL)); - return FALSE; - } - } - - *id = GUINT32_FROM_LE (((guint32 *) data)[0]); - *size = GUINT32_FROM_LE (((guint32 *) data)[0]); - - return TRUE; -} - -static void -gst_my_demuxer_loop (GstElement *element) -{ - GstMyDemuxer *demux = GST_MY_DEMUXER (element); - guint32 id, size; - - if (!gst_my_demuxer_peek (demux, &id, &size)) - return; - - switch (id) { - [.. normal chunk handling ..] - } -} - - - Reason for this is that event handling is now centralized in one - place and the _loop () function is a lot - cleaner and more readable. Those are common code practices, but - since the mistake of not using such common - code practices has been made too often, we explicitely mention - this here. - - - - - Simple seeking and indexes - - Sources will generally receive a seek event in the exact supported - format by the element. Demuxers, however, can not seek in - themselves directly, but need to convert from one unit (e.g. - time) to the other (e.g. bytes) and send a new event to its sink - pad. Given this, the _convert ()-function (or, - more general: unit conversion) is the most important function in a - demuxer. Some demuxers (AVI, Matroska) and parsers will keep an - index of all chunks in a stream, firstly to improve seeking - precision and secondly so they won't lose sync. Some other demuxers - will seek the stream directly without index (e.g. MPEG, Ogg) - - usually based on something like a cumulative bitrate - and then - find the closest next chunk from their new position. The best - choice depends on the format. - - - Note that it is recommended for demuxers to implement event, - conversion and query handling functions (using time units or so), - in addition to the ones (usually in byte units) provided by the - pipeline source element. - - + + As mentioned previously in , + demuxers should use fixed caps, since their data type will not change. + + + As discussed in , demuxer elements + can be written in multiple ways: + + + + + They can be the driving force of the pipeline, by running their own + task. This works particularly well for elements that need random + access, for example an AVI demuxer. + + + + + They can also run in push-based mode, which means that an upstream + element drives the pipeline. This works particularly well for streams + that may come from network, such as Ogg. + + + + + In addition, audio parsers with one output can, in theory, also be written + in random access mode. Although simple playback will mostly work if your + element only accepts one mode, it may be required to implement multiple + modes to work in combination with all sorts of applications, such as + editing. Also, performance may become better if you implement multiple + modes. See to see how an element + can accept multiple scheduling modes. + diff --git a/docs/pwg/pwg.xml b/docs/pwg/pwg.xml index 06d5a06f5a7..8674170f3cb 100644 --- a/docs/pwg/pwg.xml +++ b/docs/pwg/pwg.xml @@ -31,11 +31,9 @@ - - + - @@ -152,26 +150,26 @@ - - Other Element Types + + Creating special element types By now, we have looked at pretty much any feature that can be embedded - into a &GStreamer; element. However, we have limited ourselves to the - simple model of a filter element. In this chapter, we will look at the - specific difficulties and things to keep in mind when writing specific - types of elements. We will discuss output elements (sinks), input - elements (sources), 1-to-N elements, N-to-1 elements, N-to-N elements, - autopluggers and managers. Some of these represent elements that don't - actually exist. Rather, they represent a general concept. + into a &GStreamer; element. Most of this has been fairly low-level and + given deep insights in how &GStreamer; works internally. Fortunately, + &GStreamer; contains some easier-to-use interfaces to create such + elements. In order to do that, we will look closer at the element + types for which &GStreamer; provides base classes (sources, sinks and + transformation elements). We will also look closer at some types of + elements that require no specific coding such as scheduling-interaction + or data passing, but rather require specific pipeline control (e.g. + N-to-1 elements and managers). - &OTHER_SOURCE; - &OTHER_SINK; + &OTHER_BASE; &OTHER_ONETON; &OTHER_NTOONE; - &OTHER_NTON; &OTHER_MANAGER;