mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-11-15 22:01:27 +00:00
269 lines
7.9 KiB
Text
269 lines
7.9 KiB
Text
Phonon backend
|
|
--------------
|
|
|
|
The phonon design is based around forming graphs using 3 basic components:
|
|
|
|
- a source component that generates raw audio/video/subtitle data, aka
|
|
MediaObject.
|
|
- an effect component that applies effects to raw audio/video known as
|
|
AudioPath/VideoPath respectively. Subtitles are routed to a VideoPath
|
|
- output components that render audio or video called AudioOutput and
|
|
VideoOutput.
|
|
|
|
there is also a special input object that allows for feeding raw data in the
|
|
pipeline and specialized sinks to retrieve audio samples and video frames from
|
|
the pipeline.
|
|
|
|
A typical graph or a source that produces an audio and a video stream that
|
|
need to be played. The VideoPath and AudioPath typically contain no filters
|
|
in this case:
|
|
|
|
+----+ +---
|
|
| FX | | ...
|
|
+----+ +---
|
|
V V
|
|
+-----------+ +-------------+
|
|
----->| VideoPath |----->| VideoOutput |
|
|
| +-----------+ +-------------+
|
|
+-------------+
|
|
| MediaObject |
|
|
+-------------+
|
|
| +-----------+ +-------------+
|
|
+---->| AudioPath |----->| AudioOutput |
|
|
+-----------+ +-------------+
|
|
^ ^
|
|
+----+ +---
|
|
| FX | | ...
|
|
+-+--+ +---
|
|
|
|
- This is very similar to a regular gstreamer playback pipeline.
|
|
|
|
A typical graph of playing and crosfading two sources:
|
|
|
|
+--------+
|
|
| volume |
|
|
+--------+
|
|
V
|
|
+-------------+ +-----------+
|
|
| MediaObject |--->| AudioPath |\
|
|
+-------------+ +-----------+ \ +-------------+
|
|
---->| AudioOutput |
|
|
+-------------+ +-----------+ / +-------------+
|
|
| MediaObject |--->| AudioPath |/
|
|
+-------------+ +-----------+
|
|
^
|
|
+--------+
|
|
| volume |
|
|
+--------+
|
|
|
|
- As soon as two audio paths are connected to one sink, the input signals are
|
|
mixed before sending them to the sink. The mixing is typically done in the
|
|
audio sink by an element such as adder.
|
|
|
|
Other types of graphs are possible too:
|
|
|
|
+-----------+
|
|
/| AudioPath |\
|
|
+-------------+ / +-----------+ \ +-------------+
|
|
| MediaObject |-- ---->| AudioOutput |
|
|
+-------------+ \ +-----------+ / +-------------+
|
|
\| AudioPath |/
|
|
+-----------+
|
|
|
|
- This graph sends the same out data to 2 effect filter graphs and then mixes
|
|
it to an audio output. The splitting of the graph typically happens with a
|
|
tee element after the media object.
|
|
|
|
|
|
Questions
|
|
---------
|
|
|
|
1) do the following chains run
|
|
|
|
- synchronized with a shared clock?
|
|
|
|
+-------------+ +-----------+ +-------------+
|
|
| MediaObject |--->| AudioPath |--->| AudioOutput |
|
|
+-------------+ +-----------+ +-------------+
|
|
|
|
+-------------+ +-----------+ +-------------+
|
|
| MediaObject |--->| VideoPath |--->| VideoOutput |
|
|
+-------------+ +-----------+ +-------------+
|
|
|
|
- no API to set both MediaObjects atomically to play so it is assumed that
|
|
the playback starts and follows the rate of the global clock as soon as
|
|
the MediaObject is set to play. This makes unconnected chains run as if
|
|
they were in different GstPipelines.
|
|
|
|
2) Threading:
|
|
|
|
- Can signals be emitted from any thread?
|
|
- what operations are permitted from a signal handler?
|
|
|
|
3) Error reporting
|
|
|
|
- How does error reporting work?
|
|
* an audio/video device/port is busy.
|
|
* a fatal decoding error occurred.
|
|
* a media type is not supported
|
|
|
|
|
|
General
|
|
-------
|
|
|
|
- Setting up KDE and Phonon build environment
|
|
- Testing, identifying test applications, building test cases
|
|
- Asking questions to Phonon maintainers/designers
|
|
|
|
Essential classes
|
|
-----------------
|
|
|
|
These classes are essential to implement a backend and should be implemented
|
|
first.
|
|
|
|
Phonon::BackendCapabilities
|
|
|
|
Mostly exposes features in the registry like available decoders and effects.
|
|
|
|
Phonon::Factory
|
|
|
|
Entry point for the GStreamer backend. Provides methods to create instances of
|
|
object from our backed.
|
|
|
|
|
|
Simple playback
|
|
---------------
|
|
|
|
The following classes need to be implemented in order to have simple playback
|
|
capabilities from the backend.
|
|
|
|
Phonon::AudioOutput
|
|
|
|
- Wrapper around audiosinks. Also needs provision for rate and format
|
|
conversions.
|
|
- Mixing capabilities in the case when 2 audio paths are routed to it.
|
|
|
|
Notes:
|
|
|
|
* is the volume related to the device or to the connection to the device.
|
|
|
|
Phonon::VideoWidget
|
|
|
|
- Wrapper around videosinks. Also needs provision for colorspace and size
|
|
conversions. Extends QWidget and probably needs to hook into the XOverlay
|
|
stuff to draw in the QT widget. Supports fullscreen mode with a switch.
|
|
|
|
- Needs mixing capabilities in the case when 2 video paths are routed to it.
|
|
|
|
Phonon::AbstractMediaProducer
|
|
|
|
- contains stream selection
|
|
- play/pause/stop
|
|
- seeking
|
|
- periodically performs tick callbacks.
|
|
|
|
Phonon::MediaObject:
|
|
|
|
- The object that decodes the media into raw audio/video/subtitle.
|
|
This object will use the GStreamer decodebin element to perform the
|
|
typefinding and decoding.
|
|
|
|
Phonon::AudioPath/Phonon::VideoPath
|
|
|
|
- Simple container for audio/video effect plugins.
|
|
- Handles adding/removing of effects, making sure that the streaming is not
|
|
interrupted and the formats are all compatible.
|
|
|
|
|
|
Effect support
|
|
--------------
|
|
|
|
Phonon::Visualization
|
|
|
|
Connects an AudioPath to a VideoWidget and allows for selection of a
|
|
visualisation plugin.
|
|
|
|
Phonon::AudioEffect/Phonon::VideoEffect
|
|
|
|
Base classes
|
|
|
|
Phonon::VolumeFaderEffect
|
|
|
|
Allows fade-in and fade-out with a configurable curve and time. Needs
|
|
GstController.
|
|
|
|
Phonon::BrightnessControl
|
|
|
|
Controls the brightness of video.
|
|
|
|
|
|
Playlist support
|
|
----------------
|
|
|
|
Phonon::MediaQueue:
|
|
|
|
?? don't know yet where this fits in.
|
|
|
|
|
|
Capture
|
|
-------
|
|
|
|
Phonon::AvCapture
|
|
|
|
Synchronized audio and video capture.
|
|
|
|
Phonon::AudioWriter
|
|
|
|
Compress audio.
|
|
|
|
|
|
Advanced features
|
|
-----------------
|
|
|
|
Phonon::ByteStream
|
|
|
|
Feed raw data into the pipeline. Used for streaming network access.
|
|
|
|
Implementation:
|
|
|
|
Possibly a specialized source element connected to a decodebin.
|
|
|
|
Notes:
|
|
|
|
* Phonon::ByteStream::writeData
|
|
- can it block?
|
|
|
|
* Phonon::ByteStream::setStreamSeekable
|
|
- If called before starting the ByteStream, decodebin might operate in pull
|
|
based mode when supported. Else the source is activated in push mode.
|
|
- If called after starting ByteStream, the Phonon::ByteStream::seekStream
|
|
signal can be called for push-based seekable streams.
|
|
|
|
* Can the signals be emitted from a streaming thread?
|
|
|
|
|
|
Phonon::AudioDataOutput/Phonon::VideoDataOutput/
|
|
|
|
Receive raw audio/video data from the pipeline. Used to allow applications to
|
|
deal with the raw data themselves.
|
|
|
|
Implementation:
|
|
|
|
Possibly a specialized sink element.
|
|
|
|
Notes :
|
|
|
|
* Phonon::AudioDataOutput::dataReady
|
|
- can this be emitted from the streaming threads?
|
|
|
|
* Phonon::AudioDataOutput::endOfMedia
|
|
- can this be emitted from the streaming threads?
|
|
- We need to grab this EOS message synchronously from the bus.
|
|
- should be emitted _before_ sending the last dataReady. This means we need
|
|
to cache at least one dataReady.
|
|
|
|
* Phonon::AudioDataOutput::setDataSize
|
|
- can this be a _suggested_ data size or does every callback need to be of
|
|
this size?
|
|
|
|
|