diff --git a/gst-plugin-tutorial/README.md b/gst-plugin-tutorial/README.md new file mode 100644 index 00000000..62d70a6a --- /dev/null +++ b/gst-plugin-tutorial/README.md @@ -0,0 +1,8 @@ +# Tutorial on how to write GStreamer plugins in Rust + +This tutorial is for the `gst-plugin-tutorial` plugin. This plugin provides 4 features. There are seperate turoials for `rgb2gray` and `sinesrc` feature for now and there will be more in future. + +1. [Part 1: `rgb2gray` - A Video Filter for converting RGB to grayscale](tutorial-1.md) +2. [Part 2: `sinesrc` - A raw audio sine wave source](tutorial-2.md) +3. Part 3: `identity` +4. Part 4: `progressBin` - Prints progress information diff --git a/gst-plugin-tutorial/img/bbb.jpg b/gst-plugin-tutorial/img/bbb.jpg new file mode 100644 index 00000000..e69de29b diff --git a/gst-plugin-tutorial/tutorial-1.md b/gst-plugin-tutorial/tutorial-1.md new file mode 100644 index 00000000..2bb24ba4 --- /dev/null +++ b/gst-plugin-tutorial/tutorial-1.md @@ -0,0 +1,869 @@ +# How to write GStreamer Elements in Rust Part 1: A Video Filter for converting RGB to grayscale + +In this first part we’re going to write a plugin that contains a video filter element. The video filter can convert from RGB to grayscale, either output as 8-bit per pixel grayscale or 32-bit per pixel RGB. In addition there’s a property to invert all grayscale values, or to shift them by up to 255 values. In the end this will allow you to watch [Big Bucky Bunny](https://peach.blender.org/), or anything else really that can somehow go into a GStreamer pipeline, in grayscale. Or encode the output to a new video file, send it over the network via [WebRTC](https://gstconf.ubicast.tv/videos/gstreamer-webrtc/) or something else, or basically do anything you want with it. + +![alt text](img/bbb.jpg "Big Bucky Bunny – Grayscale") + +This will show the basics of how to write a GStreamer plugin and element in Rust: the basic setup for registering a type and implementing it in Rust, and how to use the various GStreamer API and APIs from the Rust standard library to do the processing. + +The final code for this plugin can be found [here](https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/tree/master/gst-plugin-tutorial), and it is based on latest git version of the [gstreamer](https://gitlab.freedesktop.org/gstreamer/gstreamer-rs). At least Rust 1.32 is required for all this. You also need to have GStreamer (at least version 1.8) installed for your platform, see e.g. the GStreamer bindings [installation instructions](https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/blob/master/README.md). + +# Table of contents + +1. [Project Structure](#project-structure) +1. [Plugin Initialization](#plugin-initialization) +1. [Type Registration](#type-registration) +1. [Type Class and Instance Initialization](#type-class-and-instance-initialization) +1. [Caps and Pad Templates](#caps-and-pad-templates) +1. [Caps Handling Part 1](#caps-handling-part-1) +1. [Caps Handling Part 2](#caps-handling-part-2) +1. [Conversion of BGRx Video Frames to Grayscale](#conversion-of-bgrx-video-frames-to-grayscale) +1. [Testing the new element](#testing-the-new-element) +1. [Properties](#properties) +1. [What next](#what-next) + + +## Project Structure + +We’ll create a new `cargo` project with `cargo init --lib --name gst-plugin-tutorial`. This will create a basically empty `Cargo.toml` and a corresponding `src/lib.rs`. We will use this structure: `lib.rs` will contain all the plugin related code, separate modules will contain any GStreamer plugins that are added. + +The empty `Cargo.toml` has to be updated to list all the dependencies that we need, and to define that the crate should result in a `cdylib`, i.e. a C library that does not contain any Rust-specific metadata. The final `Cargo.toml` looks as follows. + + +```toml +[package] +name = "gst-plugin-tutorial" +version = "0.1.0" +authors = ["Sebastian Dröge "] +repository = "https://gitlab.freedesktop.org/gstreamer/gst-plugin-rs" +license = "MIT/Apache-2.0" +edition = "2018" + +[dependencies] +glib = { git = "https://github.com/gtk-rs/glib", features = ["subclassing"] } +gstreamer = { git = "https://gitlab.freedesktop.org/gstreamer/gstreamer-rs", features = ["subclassing"] } +gstreamer-base = { git = "https://gitlab.freedesktop.org/gstreamer/gstreamer-rs", features = ["subclassing"] } +gstreamer-video = { git = "https://gitlab.freedesktop.org/gstreamer/gstreamer-rs" } + +[lib] +name = "gstrstutorial" +crate-type = ["cdylib"] +path = "src/lib.rs" +``` + +We depend on the `gstreamer`, `gstreamer-base` and `gstreamer-video` crates for various GStreamer APIs that will be used later, and the `glib` crate to be able to use some GLib API that we’ll need. GStreamer is building upon GLib, and this leaks through in various places. + +With the basic project structure being set-up, we should be able to compile the project with `cargo build` now, which will download and build all dependencies and then creates a file called `target/debug/libgstrstutorial.so` (or .dll on Windows, .dylib on macOS). This is going to be our GStreamer plugin. + +To allow GStreamer to find our new plugin and make it available in every GStreamer-based application, we could install it into the system or user-wide GStreamer plugin path or simply point the `GST_PLUGIN_PATH` environment variable to the directory containing it: + +```bash +export GST_PLUGIN_PATH=`pwd`/target/debug +``` + +If you now run the `gst-inspect-1.0` tool on the `libgstrstutorial.so`, it will not yet print all information it can extract from the plugin but for now just complains that this is not a valid GStreamer plugin. Which is true, we didn’t write any code for it yet. + +## Plugin Initialization + +Let’s start editing `src/lib.rs` to make this an actual GStreamer plugin. First of all, we need to add various extern crate directives to be able to use our dependencies and also mark some of them `#[macro_use]` because we’re going to use `macros` defined in some of them. This looks like the following + +``` +#[macro_use] +extern crate glib; +#[macro_use] +extern crate gstreamer as gst; +extern crate gstreamer_base as gst_base; +extern crate gstreamer_video as gst_video; +``` + +Next we make use of the `gst_plugin_define!` `macro` from the `gstreamer` crate to set-up the static metadata of the plugin (and make the shared library recognizeable by GStreamer to be a valid plugin), and to define the name of our entry point function (`plugin_init`) where we will register all the elements that this plugin provides. + +```rust +gst_plugin_define!( + rstutorial, + "Rust Tutorial Plugin", + plugin_init, + "1.0", + "MIT/X11", + "rstutorial", + "rstutorial", + "https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs", + "2017-12-30" +); +``` +GStreamer requires this information to be statically available in the shared library, not returned by a function. + +The static plugin metadata that we provide here is +1. name of the plugin +1. short description for the plugin +1. name of the plugin entry point function +1. version number of the plugin +1. license of the plugin (only a fixed set of licenses is allowed here, [see](https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstPlugin.html#GstPluginDesc)) +1. source package name +1. binary package name (only really makes sense for e.g. Linux distributions) +1. origin of the plugin +1. release date of this version + +In addition we’re defining an empty plugin entry point function that just returns `Ok(())` + +```rust +fn plugin_init(plugin: &gst::Plugin) -> Result<(), glib::BoolError> { + Ok(()) +} +``` + +With all that given, `gst-inspect-1.0` should print exactly this information when running on the `libgstrstutorial.so` file (or .dll on Windows, or .dylib on macOS) + +```sh +gst-inspect-1.0 target/debug/libgstrstutorial.so +``` + +## Type Registration + +As a next step, we’re going to add another module `rgb2gray` to our project, and call a function called `register` from our `plugin_init` function. + +```rust +mod rgb2gray; + +fn plugin_init(plugin: &gst::Plugin) -> Result<(), glib::BoolError> { + rgb2gray::register(plugin)?; + Ok(()) +} +``` + +With that our `src/lib.rs` is complete, and all following code is only in `src/rgb2gray.rs`. At the top of the new file we first need to add various `use-directives` to import various types and functions we’re going to use into the current module’s scope. + +```rust +use glib; +use glib::subclass; +use glib::subclass::prelude::*; + +use gst; +use gst::prelude::*; +use gst::subclass::prelude::*; +use gst_base; +use gst_base::subclass::prelude::*; +use gst_video; + +use std::i32; +use std::sync::Mutex; +``` + +GStreamer is based on the GLib object system ([GObject](https://developer.gnome.org/gobject/stable/)). C (just like Rust) does not have built-in support for object orientated programming, inheritance, virtual methods and related concepts, and GObject makes these features available in C as a library. Without language support this is a quite verbose endeavour in C, and the `glib` crate tries to expose all this in a (as much as possible) Rust-style API while hiding all the details that do not really matter. + +So, as a next step we need to register a new type for our RGB to Grayscale converter GStreamer element with the GObject type system, and then register that type with GStreamer to be able to create new instances of it. We do this with the following code + +```rust +struct Rgb2Gray{} + +impl Rgb2Gray{} + +impl ObjectSubclass for Rgb2Gray { + const NAME: &'static str = "RsRgb2Gray"; + type ParentType = gst_base::BaseTransform; + type Instance = gst::subclass::ElementInstanceStruct; + type Class = subclass::simple::ClassStruct; + + // This macro provides some boilerplate + glib_object_subclass!(); + + fn new() -> Self { + Self {} + } + + fn class_init(klass: &mut subclass::simple::ClassStruct) { + + } +} + +pub fn register(plugin: &gst::Plugin) -> Result<(), glib::BoolError> { + gst::Element::register(Some(plugin), "rsrgb2gray", 0, Rgb2Gray::get_type()) +} + +``` + +This defines a struct `Rgb2Gray` which is empty for now and an empty implementation of the struct which will later be used. The `ObjectSubclass` trait is implemented on the struct `Rgb2Gray` for providing static information about the type to the type system. By implementing `ObjectSubclass` we allow registering our struct with the GObject object system. + +`ObjectSubclass` has an associated constant which contains the name of the type, some associated types, and functions for initializing/returning a new instance of our element (`new`) and for initializing the class metadata (`class_init`, more on that later). We simply let those functions proxy to associated functions on the `Rgb2Gray` struct that we’re going to define at a later time. + +In addition, we also define a `register` function (the one that is already called from our `plugin_init` function). When `register` function is called it registers the element factory with GStreamer based on the type ID, to be able to create new instances of it with the name “rsrgb2gray” (e.g. when using [`gst::ElementFactory::make`](https://slomo.pages.freedesktop.org/rustdocs/gstreamer/gstreamer/struct.ElementFactory.html#method.make)). The `get_type` function will register the type with the GObject type system on the first call and the next time it's called (or on all the following calls) it will return the type ID. + +## Type Class & Instance Initialization + +As a next step we implement the `new` funtion and `class_init` functions. In the first version, this struct is almost empty but we will later use it to store all state of our element. + +```rust +struct Rgb2Gray { + cat: gst::DebugCategory, +} + +impl Rgb2Gray{} + +impl ObjectSubclass for Rgb2Gray { + [...] + + fn new() -> Self { + Self { + cat: gst::DebugCategory::new( + "rsrgb2gray", + gst::DebugColorFlags::empty(), + Some("Rust RGB-GRAY converter"), + ), + } + } + + fn class_init(klass: &mut subclass::simple::ClassStruct) { + klass.set_metadata( + "RGB-GRAY Converter", + "Filter/Effect/Converter/Video", + "Converts RGB to GRAY or grayscale RGB", + "Sebastian Dröge ", + ); + + klass.configure( + gst_base::subclass::BaseTransformMode::NeverInPlace, + false, + false, + ); + } +} +``` + + +In the `new` function we return our struct, containing a newly created GStreamer debug category of name “rsrgb2gray”. We’re going to use this debug category later for making use of GStreamer’s debug logging system for logging the state and changes of our element. + +In the `class_init` function we, again, set up some metadata for our new element. In this case these are a description, a classification of our element, a longer description and the author. The metadata can later be retrieved and made use of via the [`Registry`](https://slomo.pages.freedesktop.org/rustdocs/gstreamer/gstreamer/struct.Registry.html) and [`PluginFeature`](https://slomo.pages.freedesktop.org/rustdocs/gstreamer/gstreamer/struct.PluginFeature.html)/[`ElementFactory`](https://slomo.pages.freedesktop.org/rustdocs/gstreamer/gstreamer/struct.ElementFactory.html) API. We also configure the `BaseTransform` class and define that we will never operate in-place (producing our output in the input buffer), and that we don’t want to work in passthrough mode if the input/output formats are the same. + +Additionally we need to implement various traits on the Rgb2Gray struct, which will later be used to override virtual methods of the various parent classes of our element. For now we can keep the trait implementations empty, except for `ObjectImpl` trait which should simply call the `glib_object_impl!()` macro for some boilerplate code. There is one trait implementation required per parent class. + +```rust +impl ObjectImpl for Rgb2Gray { + glib_object_impl!(); +} +impl ElementImpl for Rgb2Gray {} +impl BaseTransformImpl for Rgb2Gray {} +``` + +With all this defined, `gst-inspect-1.0` should be able to show some more information about our element already but will still complain that it’s not complete yet. + +**Side note:** This is the basic code that should be in `rgb2gray.rs` to successfully build the plugin. You can fill up the code while going through the later part of the tutorial. + +```rust +//all imports... + +struct Rgb2Gray {} + +impl Rgb2Gray {} + +impl ObjectSubclass for Rgb2Gray { + const NAME: &'static str = "RsRgb2Gray"; + type ParentType = gst_base::BaseTransform; + type Instance = gst::subclass::ElementInstanceStruct; + type Class = subclass::simple::ClassStruct; + + glib_object_subclass!(); +} + +impl ObjectImpl for Rgb2Gray { + glib_object_impl!(); +} + +impl ElementImpl for Rgb2Gray {} + +impl BaseTransformImpl for Rgb2Gray {} + +pub fn register(plugin: &gst::Plugin) -> Result<(), glib::BoolError> { + gst::Element::register(Some(plugin), "rsrgb2gray", 0, Rgb2Gray::get_type()) +} +``` + +## Caps & Pad Templates + +Data flow of GStreamer elements is happening via pads, which are the input(s) and output(s) (or sinks and sources in GStreamer terminology) of an element. Via the pads, buffers containing actual media data, events or queries are transferred. An element can have any number of sink and source pads, but our new element will only have one of each. + +To be able to declare what kinds of pads an element can create (they are not necessarily all static but could be created at runtime by the element or the application), it is necessary to install so-called pad templates during the class initialization (in the `class_init` funtion). These pad templates contain the name (or rather “name template”, it could be something like `src_%u` for e.g. pad templates that declare multiple possible pads), the direction of the pad (sink or source), the availability of the pad (is it always there, sometimes added/removed by the element or to be requested by the application) and all the possible media types (called caps) that the pad can consume (sink pads) or produce (src pads). + +In our case we only have always pads, one sink pad called “sink”, on which we can only accept RGB (BGRx to be exact) data with any width/height/framerate and one source pad called “src”, on which we will produce either RGB (BGRx) data or GRAY8 (8-bit grayscale) data. We do this by adding the following code to the class_init function. + +```rust + let caps = gst::Caps::new_simple( + "video/x-raw", + &[ + ( + "format", + &gst::List::new(&[ + &gst_video::VideoFormat::Bgrx.to_string(), + &gst_video::VideoFormat::Gray8.to_string(), + ]), + ), + ("width", &gst::IntRange::::new(0, i32::MAX)), + ("height", &gst::IntRange::::new(0, i32::MAX)), + ( + "framerate", + &gst::FractionRange::new( + gst::Fraction::new(0, 1), + gst::Fraction::new(i32::MAX, 1), + ), + ), + ], + ); + let src_pad_template = gst::PadTemplate::new( + "src", + gst::PadDirection::Src, + gst::PadPresence::Always, + &caps, + ) + .unwrap(); + klass.add_pad_template(src_pad_template); + + + let caps = gst::Caps::new_simple( + "video/x-raw", + &[ + ("format", &gst_video::VideoFormat::Bgrx.to_string()), + ("width", &gst::IntRange::::new(0, i32::MAX)), + ("height", &gst::IntRange::::new(0, i32::MAX)), + ( + "framerate", + &gst::FractionRange::new( + gst::Fraction::new(0, 1), + gst::Fraction::new(i32::MAX, 1), + ), + ), + ], + ); + let sink_pad_template = gst::PadTemplate::new( + "sink", + gst::PadDirection::Sink, + gst::PadPresence::Always, + &caps, + ) + .unwrap(); + klass.add_pad_template(sink_pad_template); +``` + +The names “src” and “sink” are pre-defined by the `BaseTransform` class and this base-class will also create the actual pads with those names from the templates for us whenever a new element instance is created. Otherwise we would have to do that in our `new` function but here this is not needed. + +If you now run gst-inspect-1.0 on the rsrgb2gray element, these pad templates with their caps should also show up. + +## Caps Handling Part 1 + +As a next step we will add caps handling to our new element. This involves overriding 4 virtual methods from the BaseTransformImpl trait, and actually storing the configured input and output caps inside our element struct. Let’s start with the latter + +```rust +struct State { + in_info: gst_video::VideoInfo, + out_info: gst_video::VideoInfo, +} + +struct Rgb2Gray { + cat: gst::DebugCategory, + state: Mutex> +} + +impl Rgb2Gray{} + +impl ObjectSubclass for Rgb2Gray { + [...] + + fn new() -> Self { + Self { + cat: gst::DebugCategory::new( + "rsrgb2gray", + gst::DebugColorFlags::empty(), + "Rust RGB-GRAY converter", + ), + state: Mutex::new(None), + } + } +} +``` + +We define a new struct State that contains the input and output caps, stored in a [`VideoInfo`](https://slomo.pages.freedesktop.org/rustdocs/gstreamer/gstreamer_video/struct.VideoInfo.html). `VideoInfo` is a struct that contains various fields like width/height, framerate and the video format and allows to conveniently with the properties of (raw) video formats. We have to store it inside a [`Mutex`](https://doc.rust-lang.org/std/sync/struct.Mutex.html) in our `Rgb2Gray` struct as this can (in theory) be accessed from multiple threads at the same time. + +Whenever input/output caps are configured on our element, the `set_caps` virtual method of `BaseTransform` is called with both caps (i.e. in the very beginning before the data flow and whenever it changes), and all following video frames that pass through our element should be according to those caps. Once the element is shut down, the `stop` virtual method is called and it would make sense to release the `State` as it only contains stream-specific information. We’re doing this by adding the following to the `BaseTransformImpl` trait implementation + +```rust +impl BaseTransformImpl for Rgb2Gray { + fn set_caps( + &self, + element: &gst_base::BaseTransform, + incaps: &gst::Caps, + outcaps: &gst::Caps, + ) -> bool { + let in_info = match gst_video::VideoInfo::from_caps(incaps) { + None => return false, + Some(info) => info, + }; + let out_info = match gst_video::VideoInfo::from_caps(outcaps) { + None => return false, + Some(info) => info, + }; + + gst_debug!( + self.cat, + obj: element, + "Configured for caps {} to {}", + incaps, + outcaps + ); + + *self.state.lock().unwrap() = Some(State { in_info, out_info }); + + true + } + + fn stop(&self, element: &gst_base::BaseTransform) -> Result<(), gst::ErrorMessage> { + // Drop state + let _ = self.state.lock().unwrap().take(); + + gst_info!(self.cat, obj: element, "Stopped"); + + Ok(()) + } +} +``` + +This code should be relatively self-explanatory. In `set_caps` we’re parsing the two caps into a `VideoInfo` and then store this in our `State`, in `stop` we drop the `State` and replace it with `None`. In addition we make use of our debug category here and use the `gst_info!` and `gst_debug!` macros to output the current caps configuration to the GStreamer debug logging system. This information can later be useful for debugging any problems once the element is running. + +Next we have to provide information to the `BaseTransform` base class about the size in bytes of a video frame with specific caps. This is needed so that the base class can allocate an appropriately sized output buffer for us, that we can then fill later. This is done with the `get_unit_size` virtual method, which is required to return the size of one processing unit in specific caps. In our case, one processing unit is one video frame. In the case of raw audio it would be the size of one sample multiplied by the number of channels. + +```rust +impl BaseTransformImpl for Rgb2Gray { + fn get_unit_size(&self, _element: &gst_base::BaseTransform, caps: &gst::Caps) -> Option { + gst_video::VideoInfo::from_caps(caps).map(|info| info.size()) + } +} +``` + +We simply make use of the `VideoInfo` API here again, which conveniently gives us the size of one video frame already. + +Instead of `get_unit_size` it would also be possible to implement the `transform_size` virtual method, which is getting passed one size and the corresponding caps, another caps and is supposed to return the size converted to the second caps. Depending on how your element works, one or the other can be easier to implement. + +## Caps Handling Part 2 + +We’re not done yet with caps handling though. As a very last step it is required that we implement a function that is converting caps into the corresponding caps in the other direction. That is, we should convert BGRx to BGRx or GRAY8. Similarly, if the element downstream of ours can accept GRAY8 with a specific width/height from our source pad, we have to convert this to BGRx with that very same width/height. For example, if we receive BGRx caps with some width/height on the sinkpad, we should convert this into new caps with the same width/height but BGRx or GRAY8. + +This has to be implemented in the `transform_caps` virtual method, and looks as follows + +```rust +impl BaseTransformImpl for Rgb2Gray { + fn transform_caps( + &self, + element: &gst_base::BaseTransform, + direction: gst::PadDirection, + caps: &gst::Caps, + filter: Option<&gst::Caps>, + ) -> Option { + let other_caps = if direction == gst::PadDirection::Src { + let mut caps = caps.clone(); + + for s in caps.make_mut().iter_mut() { + s.set("format", &gst_video::VideoFormat::Bgrx.to_string()); + } + + caps + } else { + let mut gray_caps = gst::Caps::new_empty(); + + { + let gray_caps = gray_caps.get_mut().unwrap(); + + for s in caps.iter() { + let mut s_gray = s.to_owned(); + s_gray.set("format", &gst_video::VideoFormat::Gray8.to_string()); + gray_caps.append_structure(s_gray); + } + gray_caps.append(caps.clone()); + } + + gray_caps + }; + + gst_debug!( + self.cat, + obj: element, + "Transformed caps from {} to {} in direction {:?}", + caps, + other_caps, + direction + ); + + if let Some(filter) = filter { + Some(filter.intersect_with_mode(&other_caps, gst::CapsIntersectMode::First)) + } else { + Some(other_caps) + } + } +} +``` + +This caps conversion happens in 3 steps. First we check if we got caps for the source pad. In that case, the caps on the other pad (the sink pad) are going to be exactly the same caps but no matter if the caps contained BGRx or GRAY8 they must become BGRx as that’s the only format that our sink pad can accept. We do this by creating a clone of the input caps, then making sure that those caps are actually writable (i.e. we’re having the only reference to them, or a copy is going to be created) and then iterate over all the structures inside the caps and then set the “format” field to BGRx. After this, all structures in the new caps will be with the format field set to BGRx. + +Similarly, if we get caps for the sink pad and are supposed to convert it to caps for the source pad, we create new caps and in there append a copy of each structure of the input caps (which are BGRx) with the format field set to GRAY8. In the end we append the original caps, giving us first all caps as GRAY8 and then the same caps as BGRx. With this ordering we signal to GStreamer that we would prefer to output GRAY8 over BGRx. + +In the end the caps we created for the other pad are filtered against optional filter caps to reduce the potential size of the caps. This is done by intersecting the caps with that filter, while keeping the order (and thus preferences) of the filter caps (`gst::CapsIntersectMode::First`). + +## Conversion of BGRx Video Frames to Grayscale + +Now that all the caps handling is implemented, we can finally get to the implementation of the actual video frame conversion. For this we start with defining a helper function `bgrx_to_gray` that converts one BGRx pixel to a grayscale value. The BGRx pixel is passed as a `&[u8]` slice with 4 elements and the function returns another `u8` for the grayscale value. + +```rust +impl Rgb2Gray { + #[inline] + fn bgrx_to_gray(in_p: &[u8]) -> u8 { + // See https://en.wikipedia.org/wiki/YUV#SDTV_with_BT.601 + const R_Y: u32 = 19595; // 0.299 * 65536 + const G_Y: u32 = 38470; // 0.587 * 65536 + const B_Y: u32 = 7471; // 0.114 * 65536 + + assert_eq!(in_p.len(), 4); + + let b = u32::from(in_p[0]); + let g = u32::from(in_p[1]); + let r = u32::from(in_p[2]); + + let gray = ((r * R_Y) + (g * G_Y) + (b * B_Y)) / 65536; + + (gray as u8) + } +} +``` + +This function works by extracting the blue, green and red components from each pixel (remember: we work on BGRx, so the first value will be blue, the second green, the third red and the fourth unused), extending it from 8 to 32 bits for a wider value-range and then converts it to the Y component of the YUV colorspace (basically what your grandparents’ black & white TV would’ve displayed). The coefficients come from the Wikipedia page about YUV and are normalized to unsigned 16 bit integers so we can keep some accuracy, don’t have to work with floating point arithmetic and stay inside the range of 32 bit integers for all our calculations. As you can see, the green component is weighted more than the others, which comes from our eyes being more sensitive to green than to other colors. + +Note: This is only doing the actual conversion from linear RGB to grayscale (and in [BT.601](https://en.wikipedia.org/wiki/YUV#SDTV_with_BT.601) colorspace). To do this conversion correctly you need to know your colorspaces and use the correct coefficients for conversion, and also do [gamma correction](https://en.wikipedia.org/wiki/Gamma_correction). See [this](https://web.archive.org/web/20161024090830/http://www.4p8.com/eric.brasseur/gamma.html) about why it is important. + +Afterwards we have to actually call this function on every pixel. For this the transform virtual method is implemented, which gets a input and output buffer passed and we’re supposed to read the input buffer and fill the output buffer. The implementation looks as follows, and is going to be our biggest function for this element + +```rust +impl BaseTransformImpl for Rgb2Gray { + fn transform( + &self, + element: &gst_base::BaseTransform, + inbuf: &gst::Buffer, + outbuf: &mut gst::BufferRef, + ) -> Result { + + let mut state_guard = self.state.lock().unwrap(); + let state = state_guard.as_mut().ok_or_else(|| { + gst_element_error!(element, gst::CoreError::Negotiation, ["Have no state yet"]); + gst::FlowError::NotNegotiated + })?; + + let in_frame = + gst_video::VideoFrameRef::from_buffer_ref_readable(inbuf.as_ref(), &state.in_info) + .ok_or_else(|| { + gst_element_error!( + element, + gst::CoreError::Failed, + ["Failed to map input buffer readable"] + ); + gst::FlowError::Error + })?; + + let mut out_frame = + gst_video::VideoFrameRef::from_buffer_ref_writable(outbuf, &state.out_info) + .ok_or_else(|| { + gst_element_error!( + element, + gst::CoreError::Failed, + ["Failed to map output buffer writable"] + ); + gst::FlowError::Error + })?; + + let width = in_frame.width() as usize; + let in_stride = in_frame.plane_stride()[0] as usize; + let in_data = in_frame.plane_data(0).unwrap(); + let out_stride = out_frame.plane_stride()[0] as usize; + let out_format = out_frame.format(); + let out_data = out_frame.plane_data_mut(0).unwrap(); + + if out_format == gst_video::VideoFormat::Bgrx { + assert_eq!(in_data.len() % 4, 0); + assert_eq!(out_data.len() % 4, 0); + assert_eq!(out_data.len() / out_stride, in_data.len() / in_stride); + + let in_line_bytes = width * 4; + let out_line_bytes = width * 4; + + assert!(in_line_bytes <= in_stride); + assert!(out_line_bytes <= out_stride); + + for (in_line, out_line) in in_data + .chunks_exact(in_stride) + .zip(out_data.chunks_exact_mut(out_stride)) + { + for (in_p, out_p) in in_line[..in_line_bytes] + .chunks_exact(4) + .zip(out_line[..out_line_bytes].chunks_exact_mut(4)) + { + assert_eq!(out_p.len(), 4); + + let gray = Rgb2Gray::bgrx_to_gray(in_p); + out_p[0] = gray; + out_p[1] = gray; + out_p[2] = gray; + } + } + } else if out_format == gst_video::VideoFormat::Gray8 { + assert_eq!(in_data.len() % 4, 0); + assert_eq!(out_data.len() / out_stride, in_data.len() / in_stride); + + let in_line_bytes = width * 4; + let out_line_bytes = width; + + assert!(in_line_bytes <= in_stride); + assert!(out_line_bytes <= out_stride); + + for (in_line, out_line) in in_data + .chunks_exact(in_stride) + .zip(out_data.chunks_exact_mut(out_stride)) + { + for (in_p, out_p) in in_line[..in_line_bytes] + .chunks_exact(4) + .zip(out_line[..out_line_bytes].iter_mut()) + { + let gray = Rgb2Gray::bgrx_to_gray(in_p); + *out_p = gray; + } + } + } else { + unimplemented!(); + } + + Ok(gst::FlowSuccess::Ok) + } +} +``` + +What happens here is that we first of all lock our state (the input/output `VideoInfo`) and error out if we don’t have any yet (which can’t really happen unless other elements have a bug, but better safe than sorry). After that we map the input buffer readable and the output buffer writable with the [`VideoFrameRef`](https://slomo.pages.freedesktop.org/rustdocs/gstreamer/gstreamer_video/video_frame/struct.VideoFrameRef.html) API. By mapping the buffers we get access to the underlying bytes of them, and the mapping operation could for example make GPU memory available or just do nothing and give us access to a normally allocated memory area. We have access to the bytes of the buffer until the `VideoFrameRef` goes out of scope. + +Instead of `VideoFrameRef` we could’ve also used the [`gst::Buffer::map_readable()`](https://slomo.pages.freedesktop.org/rustdocs/gstreamer/gstreamer/buffer/struct.BufferRef.html#method.map_readable) and [`gst::Buffer::map_writable()`](https://slomo.pages.freedesktop.org/rustdocs/gstreamer/gstreamer/buffer/struct.BufferRef.html#method.map_writable) API, but different to those the `VideoFrameRef` API also extracts various metadata from the raw video buffers and makes them available. For example we can directly access the different planes as slices without having to calculate the offsets ourselves, or we get directly access to the width and height of the video frame. + +After mapping the buffers, we store various information we’re going to need later in local variables to save some typing later. This is the width (same for input and output as we never changed the width in `transform_caps`), the input and out (row-) stride (the number of bytes per row/line, which possibly includes some padding at the end of each line for alignment reasons), the output format (which can be BGRx or GRAY8 because of how we implemented `transform_caps`) and the pointers to the first plane of the input and output (which in this case also is the only plane, BGRx and GRAY8 both have only a single plane containing all the RGB/gray components). + +Then based on whether the output suppose to be BGRx or GRAY8, we iterate over all pixels. The code is basically the same in both cases, so we're only going to go through the case where BGRx is output. + +We start by iterating over each line of the input and output, and do so by using the [`chunks_exact`](https://doc.rust-lang.org/std/primitive.slice.html#method.chunks_exact) iterator and [`chunks_exact_mut`](https://doc.rust-lang.org/std/primitive.slice.html#method.chunks_exact_mut) to give us chunks of as many bytes as the (row-) stride of the video frame is, do the same for the other frame and then zip both iterators together. This means that on each iteration we get exactly one line as a slice from each of the frames and can then start accessing the actual pixels in each line. The only difference of `chunks_exact_mut` from `chunks_exact` is that it gives the mutable sub-slices so that their content can be changed. + +To access the individual pixels in each line, we again use the chunks iterator the same way, but this time to always give us chunks of 4 bytes from each line. As BGRx uses 4 bytes for each pixel, this gives us exactly one pixel. Instead of iterating over the whole line, we only take the actual sub-slice that contains the pixels, not the whole line with stride number of bytes containing potential padding at the end. Now for each of these pixels we call our previously defined `bgrx_to_gray` function and then fill the B, G and R components of the output buffer with that value to get grayscale output. And that’s all. + +Using Rust high-level abstractions like the chunks iterators and bounds-checking slice accesses might seem like it’s going to cause quite some performance penalty, but if you look at the generated assembly most of the bounds checks are completely optimized away and the resulting assembly code is close to what one would’ve written manually. Here you’re getting safe and high-level looking code with low-level performance! + +You might’ve also noticed the various assertions in the processing function. These are there to document the assumptions in the code and to give further hints to the compiler about properties of the code, and thus potentially being able to optimize the code better and moving e.g. bounds checks out of the inner loop and just having the assertion outside the loop check for the same. In Rust adding assertions can often improve performance by allowing further optimizations to be applied, but in the end always check the resulting assembly to see if what you did made any difference. + +## Testing the new element + +Now we implemented almost all functionality of our new element and could run it on actual video data. This can be done now with the `gst-launch-1.0` tool, or any application using GStreamer and allowing us to insert our new element somewhere in the video part of the pipeline. With `gst-launch-1.0` you could run for example the following pipelines + +```bash +# Run on a test pattern +gst-launch-1.0 videotestsrc ! rsrgb2gray ! videoconvert ! autovideosink + +# Run on some video file, also playing the audio +gst-launch-1.0 playbin uri=file:///path/to/some/file video-filter=rsrgb2gray +``` + +Note that you will likely want to compile with `cargo build --release` and add the `target/release` directory to `GST_PLUGIN_PATH` instead. The debug build might be too slow, and generally the release builds are multiple orders of magnitude (!) faster. + +## Properties + +The only feature missing now are the properties mentioned in the opening paragraph: one boolean property to invert the grayscale value and one integer property to shift the value by up to 255. Implementing this on top of the previous code is not a lot of work. Let’s start with defining a struct for holding the property values and defining the property metadata. + +```rust +const DEFAULT_INVERT: bool = false; +const DEFAULT_SHIFT: u32 = 0; + +#[derive(Debug, Clone, Copy)] +struct Settings { + invert: bool, + shift: u32, +} + +impl Default for Settings { + fn default() -> Self { + Settings { + invert: DEFAULT_INVERT, + shift: DEFAULT_SHIFT, + } + } +} + +static PROPERTIES: [subclass::Property; 2] = [ + subclass::Property("invert", |name| { + glib::ParamSpec::boolean( + name, + "Invert", + "Invert grayscale output", + DEFAULT_INVERT, + glib::ParamFlags::READWRITE, + ) + }), + subclass::Property("shift", |name| { + glib::ParamSpec::uint( + name, + "Shift", + "Shift grayscale output (wrapping around)", + 0, + 255, + DEFAULT_SHIFT, + glib::ParamFlags::READWRITE, + ) + }), +]; + +struct Rgb2Gray { + cat: gst::DebugCategory, + settings: Mutex, + state: Mutex>, +} + +impl Rgb2Gray{...} + +impl ObjectSubclass for Rgb2Gray { + [...] + + fn new() -> Self { + Self { + cat: gst::DebugCategory::new( + "rsrgb2gray", + gst::DebugColorFlags::empty(), + "Rust RGB-GRAY converter", + ), + settings: Mutex::new(Default::default()), + state: Mutex::new(None), + } + } +} +``` + +This should all be rather straightforward: we define a `Settings` struct that stores the two values, implement the [`Default`](https://doc.rust-lang.org/nightly/std/default/trait.Default.html) trait for it, then define a two-element array with property metadata (names, description, ranges, default value, writability), and then store the default value of our `Settings` struct inside another `Mutex` inside the element struct. + +In the next step we have to make use of these: we need to tell the GObject type system about the properties, and we need to implement functions that are called whenever a property value is set or get. + +```rust +impl ObjectSubclass for Rgb2Gray { + fn class_init(klass: &mut subclass::simple::ClassStruct) { + [...] + klass.install_properties(&PROPERTIES); + [...] + } +} + +impl ObjectImpl for Rgb2Gray { + [...] + + fn set_property(&self, obj: &glib::Object, id: usize, value: &glib::Value) { + let prop = &PROPERTIES[id]; + let element = obj.downcast_ref::().unwrap(); + + match *prop { + subclass::Property("invert", ..) => { + let mut settings = self.settings.lock().unwrap(); + let invert = value.get().unwrap(); + gst_info!( + self.cat, + obj: element, + "Changing invert from {} to {}", + settings.invert, + invert + ); + settings.invert = invert; + } + subclass::Property("shift", ..) => { + let mut settings = self.settings.lock().unwrap(); + let shift = value.get().unwrap(); + gst_info!( + self.cat, + obj: element, + "Changing shift from {} to {}", + settings.shift, + shift + ); + settings.shift = shift; + } + _ => unimplemented!(), + } + } + + fn get_property(&self, _obj: &glib::Object, id: usize) -> Result { + let prop = &PROPERTIES[id]; + + match *prop { + subclass::Property("invert", ..) => { + let settings = self.settings.lock().unwrap(); + Ok(settings.invert.to_value()) + } + subclass::Property("shift", ..) => { + let settings = self.settings.lock().unwrap(); + Ok(settings.shift.to_value()) + } + _ => unimplemented!(), + } + } +} +``` + +`Property` values can be changed from any thread at any time, that’s why the `Mutex` is needed here to protect our struct. And we’re using a new mutex to be able to have it locked only for the shorted possible amount of time: we don’t want to keep it locked for the whole time of the `transform` function, otherwise applications trying to set/get values would block for up to the processing time of one frame. + +In the property setter/getter functions we are working with a `glib::Value`. This is a dynamically typed value type that can contain values of any type, together with the type information of the contained value. Here we’re using it to handle an unsigned integer (`u32`) and a boolean for our two properties. To know which property is currently set/get, we get an identifier passed which is the index into our `PROPERTIES` array. We then simply match on the name of that to decide which property was meant + +With this implemented, we can already compile everything, see the properties and their metadata in `gst-inspect-1.0` and can also set them on `gst-launch-1.0` like this + +```bash +# Set invert to true and shift to 128 +gst-launch-1.0 videotestsrc ! rsrgb2gray invert=true shift=128 ! videoconvert ! autovideosink +``` + +If we set `GST_DEBUG=rsrgb2gray:6` in the environment before running that, we can also see the corresponding debug output when the values are changing. The only thing missing now is to actually make use of the property values for the processing. For this we add the following changes to `bgrx_to_gray` and the transform function + +```rust +impl Rgb2Gray { + #[inline] + fn bgrx_to_gray(in_p: &[u8], shift: u8, invert: bool) -> u8 { + [...] + + let gray = ((r * R_Y) + (g * G_Y) + (b * B_Y)) / 65536; + let gray = (gray as u8).wrapping_add(shift); + + if invert { + 255 - gray + } else { + gray + } + } +} + +impl BaseTransformImpl for Rgb2Gray { + fn transform( + &self, + element: &gst_base::BaseTransform, + inbuf: &gst::Buffer, + outbuf: &mut gst::BufferRef, + ) -> Result { + let settings = *self.settings.lock().unwrap(); + [...] + let gray = Rgb2Gray::bgrx_to_gray(in_p, settings.shift as u8, settings.invert); + [...] + } +} +``` + + +And that’s all. If you run the element in `gst-launch-1.0` and change the values of the properties you should also see the corresponding changes in the video output. + +Note that we always take a copy of the `Settings` struct at the beginning of the transform function. This ensures that we take the mutex only the shortest possible amount of time and then have a local snapshot of the settings for each frame. + +Also keep in mind that the usage of the property values in the `bgrx_to_gray` function is far from optimal. It means the addition of another condition to the calculation of each pixel, thus potentially slowing it down a lot. Ideally this condition would be moved outside the inner loops and the `bgrx_to_gray` function would made generic over that. See for example [this blog post](https://bluejekyll.github.io/blog/rust/2018/01/10/branchless-rust.html) about “branchless Rust” for ideas how to do that, the actual implementation is left as an exercise for the reader. + +## What next? + +We hope the code walkthrough above was useful to understand how to implement GStreamer plugins and elements in Rust. If you have any questions, feel free to ask them in the IRC channel (#gstreamer on freenode) and on our [mailing list](https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel). + +The same approach also works for audio filters or anything that can be handled in some way with the API of the `BaseTransform` base class. You can find another filter, an audio echo filter, using the same approach [here](https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/blob/master/gst-plugin-audiofx/src/audioecho.rs). + +In the [next tutorial](tutorial-2.md) in this series we’ll discuss how to use another base class to implement another kind of element, but for the time being you can also check the [git repository](https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs) for various other element implementations. + + diff --git a/gst-plugin-tutorial/tutorial-2.md b/gst-plugin-tutorial/tutorial-2.md new file mode 100644 index 00000000..de76e73f --- /dev/null +++ b/gst-plugin-tutorial/tutorial-2.md @@ -0,0 +1,1051 @@ +# How to write GStreamer Elements in Rust Part 2: A raw audio sine wave source + +In this part, a raw audio sine wave source element is going to be written. The final code can be found [here](https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/blob/master/gst-plugin-tutorial/src/sinesrc.rs). + +### Table of Contents + +1. [Boilerplate](#boilerplate) +2. [Caps Negotiation](#caps-negotiation) +3. [Query Handling](#query-handling) +4. [Buffer Creation](#buffer-creation) +5. [(Pseudo) Live Mode](#pseudo-live-mode) +6. [Unlocking](#unlocking) +7. [Seeking](#seeking) + +### Boilerplate + +The first part here will be all the boilerplate required to set up the element. You can safely [skip](#caps-negotiation) this if you remember all this from the [previous tutorial](tutorial-1.md). + +Our sine wave element is going to produce raw audio, with a number of channels and any possible sample rate with both 32 bit and 64 bit floating point samples. It will produce a simple sine wave with a configurable frequency, volume/mute and number of samples per audio buffer. In addition it will be possible to configure the element in (pseudo) live mode, meaning that it will only produce data in real-time according to the pipeline clock. And it will be possible to seek to any time/sample position on our source element. It will basically be a more simply version of the [`audiotestsrc`](https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-audiotestsrc.html) element from gst-plugins-base. + +So let's get started with all the boilerplate. This time our element will be based on the [`BaseSrc`](https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-libs/html/GstBaseSrc.html) base class instead of [`BaseTransform`](https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-libs/html/GstBaseTransform.html). + +```rust +use glib; +use gst; +use gst::prelude::*; +use gst_base::prelude::*; +use gst_audio; + +use byte_slice_cast::*; + +use gst_plugin::properties::*; +use gst_plugin::object::*; +use gst_plugin::element::*; +use gst_plugin::base_src::*; + +use std::{i32, u32}; +use std::sync::Mutex; +use std::ops::Rem; + +use num_traits::float::Float; +use num_traits::cast::NumCast; + +// Default values of properties +const DEFAULT_SAMPLES_PER_BUFFER: u32 = 1024; +const DEFAULT_FREQ: u32 = 440; +const DEFAULT_VOLUME: f64 = 0.8; +const DEFAULT_MUTE: bool = false; +const DEFAULT_IS_LIVE: bool = false; + +// Property value storage +#[derive(Debug, Clone, Copy)] +struct Settings { + samples_per_buffer: u32, + freq: u32, + volume: f64, + mute: bool, + is_live: bool, +} + +impl Default for Settings { + fn default() -> Self { + Settings { + samples_per_buffer: DEFAULT_SAMPLES_PER_BUFFER, + freq: DEFAULT_FREQ, + volume: DEFAULT_VOLUME, + mute: DEFAULT_MUTE, + is_live: DEFAULT_IS_LIVE, + } + } +} + +// Metadata for the properties +static PROPERTIES: [Property; 5] = [ + Property::UInt( + "samples-per-buffer", + "Samples Per Buffer", + "Number of samples per output buffer", + (1, u32::MAX), + DEFAULT_SAMPLES_PER_BUFFER, + PropertyMutability::ReadWrite, + ), + Property::UInt( + "freq", + "Frequency", + "Frequency", + (1, u32::MAX), + DEFAULT_FREQ, + PropertyMutability::ReadWrite, + ), + Property::Double( + "volume", + "Volume", + "Output volume", + (0.0, 10.0), + DEFAULT_VOLUME, + PropertyMutability::ReadWrite, + ), + Property::Boolean( + "mute", + "Mute", + "Mute", + DEFAULT_MUTE, + PropertyMutability::ReadWrite, + ), + Property::Boolean( + "is-live", + "Is Live", + "(Pseudo) live output", + DEFAULT_IS_LIVE, + PropertyMutability::ReadWrite, + ), +]; + +// Stream-specific state, i.e. audio format configuration +// and sample offset +struct State { + info: Option, + sample_offset: u64, + sample_stop: Option, + accumulator: f64, +} + +impl Default for State { + fn default() -> State { + State { + info: None, + sample_offset: 0, + sample_stop: None, + accumulator: 0.0, + } + } +} + +// Struct containing all the element data +struct SineSrc { + cat: gst::DebugCategory, + settings: Mutex, + state: Mutex, +} + +impl SineSrc { + // Called when a new instance is to be created + fn new(element: &BaseSrc) -> Box> { + // Initialize live-ness and notify the base class that + // we'd like to operate in Time format + element.set_live(DEFAULT_IS_LIVE); + element.set_format(gst::Format::Time); + + Box::new(Self { + cat: gst::DebugCategory::new( + "rssinesrc", + gst::DebugColorFlags::empty(), + Some("Rust Sine Wave Source"), + ), + settings: Mutex::new(Default::default()), + state: Mutex::new(Default::default()), + }) + } + + // Called exactly once when registering the type. Used for + // setting up metadata for all instances, e.g. the name and + // classification and the pad templates with their caps. + // + // Actual instances can create pads based on those pad templates + // with a subset of the caps given here. In case of basesrc, + // a "src" and "sink" pad template are required here and the base class + // will automatically instantiate pads for them. + // + // Our element here can output f32 and f64 + fn class_init(klass: &mut BaseSrcClass) { + klass.set_metadata( + "Sine Wave Source", + "Source/Audio", + "Creates a sine wave", + "Sebastian Dröge ", + ); + + // On the src pad, we can produce F32/F64 with any sample rate + // and any number of channels + let caps = gst::Caps::new_simple( + "audio/x-raw", + &[ + ( + "format", + &gst::List::new(&[ + &gst_audio::AUDIO_FORMAT_F32.to_string(), + &gst_audio::AUDIO_FORMAT_F64.to_string(), + ]), + ), + ("layout", &"interleaved"), + ("rate", &gst::IntRange::::new(1, i32::MAX)), + ("channels", &gst::IntRange::::new(1, i32::MAX)), + ], + ); + // The src pad template must be named "src" for basesrc + // and specific a pad that is always there + let src_pad_template = gst::PadTemplate::new( + "src", + gst::PadDirection::Src, + gst::PadPresence::Always, + &caps, + ); + klass.add_pad_template(src_pad_template); + + // Install all our properties + klass.install_properties(&PROPERTIES); + } +} + +impl ObjectImpl for SineSrc { + // Called whenever a value of a property is changed. It can be called + // at any time from any thread. + fn set_property(&self, obj: &glib::Object, id: u32, value: &glib::Value) { + let prop = &PROPERTIES[id as usize]; + let element = obj.clone().downcast::().unwrap(); + + match *prop { + Property::UInt("samples-per-buffer", ..) => { + let mut settings = self.settings.lock().unwrap(); + let samples_per_buffer = value.get().unwrap(); + gst_info!( + self.cat, + obj: &element, + "Changing samples-per-buffer from {} to {}", + settings.samples_per_buffer, + samples_per_buffer + ); + settings.samples_per_buffer = samples_per_buffer; + drop(settings); + + let _ = + element.post_message(&gst::Message::new_latency().src(Some(&element)).build()); + } + Property::UInt("freq", ..) => { + let mut settings = self.settings.lock().unwrap(); + let freq = value.get().unwrap(); + gst_info!( + self.cat, + obj: &element, + "Changing freq from {} to {}", + settings.freq, + freq + ); + settings.freq = freq; + } + Property::Double("volume", ..) => { + let mut settings = self.settings.lock().unwrap(); + let volume = value.get().unwrap(); + gst_info!( + self.cat, + obj: &element, + "Changing volume from {} to {}", + settings.volume, + volume + ); + settings.volume = volume; + } + Property::Boolean("mute", ..) => { + let mut settings = self.settings.lock().unwrap(); + let mute = value.get().unwrap(); + gst_info!( + self.cat, + obj: &element, + "Changing mute from {} to {}", + settings.mute, + mute + ); + settings.mute = mute; + } + Property::Boolean("is-live", ..) => { + let mut settings = self.settings.lock().unwrap(); + let is_live = value.get().unwrap(); + gst_info!( + self.cat, + obj: &element, + "Changing is-live from {} to {}", + settings.is_live, + is_live + ); + settings.is_live = is_live; + } + _ => unimplemented!(), + } + } + + // Called whenever a value of a property is read. It can be called + // at any time from any thread. + fn get_property(&self, _obj: &glib::Object, id: u32) -> Result { + let prop = &PROPERTIES[id as usize]; + + match *prop { + Property::UInt("samples-per-buffer", ..) => { + let settings = self.settings.lock().unwrap(); + Ok(settings.samples_per_buffer.to_value()) + } + Property::UInt("freq", ..) => { + let settings = self.settings.lock().unwrap(); + Ok(settings.freq.to_value()) + } + Property::Double("volume", ..) => { + let settings = self.settings.lock().unwrap(); + Ok(settings.volume.to_value()) + } + Property::Boolean("mute", ..) => { + let settings = self.settings.lock().unwrap(); + Ok(settings.mute.to_value()) + } + Property::Boolean("is-live", ..) => { + let settings = self.settings.lock().unwrap(); + Ok(settings.is_live.to_value()) + } + _ => unimplemented!(), + } + } +} + +// Virtual methods of gst::Element. We override none +impl ElementImpl for SineSrc { } + +impl BaseSrcImpl for SineSrc { + // Called when starting, so we can initialize all stream-related state to its defaults + fn start(&self, element: &BaseSrc) -> bool { + // Reset state + *self.state.lock().unwrap() = Default::default(); + + gst_info!(self.cat, obj: element, "Started"); + + true + } + + // Called when shutting down the element so we can release all stream-related state + fn stop(&self, element: &BaseSrc) -> bool { + // Reset state + *self.state.lock().unwrap() = Default::default(); + + gst_info!(self.cat, obj: element, "Stopped"); + + true + } +} + +struct SineSrcStatic; + +// The basic trait for registering the type: This returns a name for the type and registers the +// instance and class initializations functions with the type system, thus hooking everything +// together. +impl ImplTypeStatic for SineSrcStatic { + fn get_name(&self) -> &str { + "SineSrc" + } + + fn new(&self, element: &BaseSrc) -> Box> { + SineSrc::new(element) + } + + fn class_init(&self, klass: &mut BaseSrcClass) { + SineSrc::class_init(klass); + } +} + +// Registers the type for our element, and then registers in GStreamer under +// the name "sinesrc" for being able to instantiate it via e.g. +// gst::ElementFactory::make(). +pub fn register(plugin: &gst::Plugin) { + let type_ = register_type(SineSrcStatic); + gst::Element::register(plugin, "rssinesrc", 0, type_); +} +``` + +If any of this needs explanation, please see the [previous](tutorial-1.md) and the comments in the code. The explanation for all the structs fields and what they're good for will follow in the next sections. + +With all of the above and a small addition to `src/lib.rs` this should compile now. + +```rust +mod sinesrc; +[...] + +fn plugin_init(plugin: &gst::Plugin) -> bool { + [...] + sinesrc::register(plugin); + true +} +``` + +Also a couple of new crates have to be added to `Cargo.toml` and `src/lib.rs`, but you best check the code in the [repository](https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/tree/master/gst-plugin-tutorial) for details. + +### Caps Negotiation + +The first part that we have to implement, just like last time, is caps negotiation. We already notified the base class about any caps that we can potentially handle via the caps in the pad template in `class_init` but there are still two more steps of behaviour left that we have to implement. + +First of all, we need to get notified whenever the caps that our source is configured for are changing. This will happen once in the very beginning and then whenever the pipeline topology or state changes and new caps would be more optimal for the new situation. This notification happens via the `BaseTransform::set_caps` virtual method. + +```rust + fn set_caps(&self, element: &BaseSrc, caps: &gst::Caps) -> bool { + use std::f64::consts::PI; + + let info = match gst_audio::AudioInfo::from_caps(caps) { + None => return false, + Some(info) => info, + }; + + gst_debug!(self.cat, obj: element, "Configuring for caps {}", caps); + + element.set_blocksize(info.bpf() * (*self.settings.lock().unwrap()).samples_per_buffer); + + let settings = *self.settings.lock().unwrap(); + let mut state = self.state.lock().unwrap(); + + // If we have no caps yet, any old sample_offset and sample_stop will be + // in nanoseconds + let old_rate = match state.info { + Some(ref info) => info.rate() as u64, + None => gst::SECOND_VAL, + }; + + // Update sample offset and accumulator based on the previous values and the + // sample rate change, if any + let old_sample_offset = state.sample_offset; + let sample_offset = old_sample_offset + .mul_div_floor(info.rate() as u64, old_rate) + .unwrap(); + + let old_sample_stop = state.sample_stop; + let sample_stop = + old_sample_stop.map(|v| v.mul_div_floor(info.rate() as u64, old_rate).unwrap()); + + let accumulator = + (sample_offset as f64).rem(2.0 * PI * (settings.freq as f64) / (info.rate() as f64)); + + *state = State { + info: Some(info), + sample_offset: sample_offset, + sample_stop: sample_stop, + accumulator: accumulator, + }; + + drop(state); + + let _ = element.post_message(&gst::Message::new_latency().src(Some(element)).build()); + + true + } +``` + +In here we parse the caps into a [`AudioInfo`](https://slomo.pages.freedesktop.org/rustdocs/gstreamer/gstreamer_audio/struct.AudioInfo.html) and then store that in our internal state, while updating various fields. We tell the base class about the number of bytes each buffer is usually going to hold, and update our current sample position, the stop sample position (when a seek with stop position happens, we need to know when to stop) and our accumulator. This happens by scaling both positions by the old and new sample rate. If we don't have an old sample rate, we assume nanoseconds (this will make more sense once seeking is implemented). The scaling is done with the help of the [`muldiv`](https://crates.io/crates/muldiv) crate, which implements scaling of integer types by a fraction with protection against overflows by doing up to 128 bit integer arithmetic for intermediate values. + +The accumulator is the updated based on the current phase of the sine wave at the current sample position. + +As a last step we post a new `LATENCY` message on the bus whenever the sample rate has changed. Our latency (in live mode) is going to be the duration of a single buffer, but more about that later. + +`BaseSrc` is by default already selecting possible caps for us, if there are multiple options. However these defaults might not be (and often are not) ideal and we should override the default behaviour slightly. This is done in the `BaseSrc::fixate` virtual method. + +```rust + fn fixate(&self, element: &BaseSrc, caps: gst::Caps) -> gst::Caps { + // Fixate the caps. BaseSrc will do some fixation for us, but + // as we allow any rate between 1 and MAX it would fixate to 1. 1Hz + // is generally not a useful sample rate. + // + // We fixate to the closest integer value to 48kHz that is possible + // here, and for good measure also decide that the closest value to 1 + // channel is good. + let mut caps = gst::Caps::truncate(caps); + { + let caps = caps.make_mut(); + let s = caps.get_mut_structure(0).unwrap(); + s.fixate_field_nearest_int("rate", 48_000); + s.fixate_field_nearest_int("channels", 1); + } + + // Let BaseSrc fixate anything else for us. We could've alternatively have + // called Caps::fixate() here + element.parent_fixate(caps) + } +``` + +Here we take the caps that are passed in, truncate them (i.e. remove all but the very first [`Structure`](https://slomo.pages.freedesktop.org/rustdocs/gstreamer/gstreamer/structure/struct.Structure.html)) and then manually fixate the sample rate to the closest value to 48kHz. By default, caps fixation would result in the lowest possible sample rate but this is usually not desired. + +For good measure, we also fixate the number of channels to the closest value to 1, but this would already be the default behaviour anyway. And then chain up to the parent class' implementation of `fixate`, which for now basically does the same as `Caps::fixate()`. After this, the caps are fixated, i.e. there is only a single `Structure` left and all fields have concrete values (no ranges or sets). + +### Query Handling + +As our source element will work by generating a new audio buffer from a specific offset, and especially works in `Time` format, we want to notify downstream elements that we don't want to run in `Pull` mode, only in `Push` mode. In addition would prefer sequential reading. However we still allow seeking later. For a source that does not know about `Time`, e.g. a file source, the format would be configured as `Bytes`. Other values than `Time` and `Bytes` generally don't make any sense. + +The main difference here is that otherwise the base class would ask us to produce data for arbitrary `Byte` offsets, and we would have to produce data for that. While possible in our case, it's a bit annoying and for other audio sources it's not easily possible at all. + +Downstream elements will try to query this very information from us, so we now have to override the default query handling of `BaseSrc` and handle the `SCHEDULING` query differently. Later we will also handle other queries differently. + +```rust + fn query(&self, element: &BaseSrc, query: &mut gst::QueryRef) -> bool { + use gst::QueryView; + + match query.view_mut() { + // We only work in Push mode. In Pull mode, create() could be called with + // arbitrary offsets and we would have to produce for that specific offset + QueryView::Scheduling(ref mut q) => { + q.set(gst::SchedulingFlags::SEQUENTIAL, 1, -1, 0); + q.add_scheduling_modes(&[gst::PadMode::Push]); + true + } + _ => BaseSrcImplExt::parent_query(self, element, query), + } + } +``` + +To handle the `SCHEDULING` query specifically, we first have to match on a view (mutable because we want to modify the view) of the query check the type of the query. If it indeed is a scheduling query, we can set the `SEQUENTIAL` flag and specify that we handle only `Push` mode, then return `true` directly as we handled the query already. + +In all other cases we fall back to the parent class' implementation of the `query` virtual method. + +### Buffer Creation + +Now we have everything in place for a working element, apart from the virtual method to actually generate the raw audio buffers with the sine wave. From a high-level `BaseSrc` works by calling the `create` virtual method over and over again to let the subclass produce a buffer until it returns an error or signals the end of the stream. + +Let's first talk about how to generate the sine wave samples themselves. As we want to operate on 32 bit and 64 bit floating point numbers, we implement a generic function for generating samples and storing them in a mutable byte slice. This is done with the help of the [`num_traits`](https://crates.io/crates/num-traits) crate, which provides all kinds of useful traits for abstracting over numeric types. In our case we only need the [`Float`](https://docs.rs/num-traits/0.2.0/num_traits/float/trait.Float.html) and [`NumCast`](https://docs.rs/num-traits/0.2.0/num_traits/cast/trait.NumCast.html) traits. + +Instead of writing a generic implementation with those traits, it would also be possible to do the same with a simple macro that generates a function for both types. Which approach is nicer is a matter of taste in the end, the compiler output should be equivalent for both cases. + +```rust + fn process( + data: &mut [u8], + accumulator_ref: &mut f64, + freq: u32, + rate: u32, + channels: u32, + vol: f64, + ) { + use std::f64::consts::PI; + + // Reinterpret our byte-slice as a slice containing elements of the type + // we're interested in. GStreamer requires for raw audio that the alignment + // of memory is correct, so this will never ever fail unless there is an + // actual bug elsewhere. + let data = data.as_mut_slice_of::().unwrap(); + + // Convert all our parameters to the target type for calculations + let vol: F = NumCast::from(vol).unwrap(); + let freq = freq as f64; + let rate = rate as f64; + let two_pi = 2.0 * PI; + + // We're carrying a accumulator with up to 2pi around instead of working + // on the sample offset. High sample offsets cause too much inaccuracy when + // converted to floating point numbers and then iterated over in 1-steps + let mut accumulator = *accumulator_ref; + let step = two_pi * freq / rate; + + for chunk in data.chunks_mut(channels as usize) { + let value = vol * F::sin(NumCast::from(accumulator).unwrap()); + for sample in chunk { + *sample = value; + } + + accumulator += step; + if accumulator >= two_pi { + accumulator -= two_pi; + } + } + + *accumulator_ref = accumulator; + } +``` + +This function takes the mutable byte slice from our buffer as argument, as well as the current value of the accumulator and the relevant settings for generating the sine wave. + +As a first step, we "cast" the byte slice to one of the target type (f32 or f64) with the help of the [`byte_slice_cast`](https://crates.io/crates/byte-slice-cast) crate. This ensures that alignment and sizes are all matching and returns a mutable slice of our target type if successful. In case of GStreamer, the buffer alignment is guaranteed to be big enough for our types here and we allocate the buffer of a correct size later. + +Now we convert all the parameters to the types we will use later, and store them together with the current accumulator value in local variables. Then we iterate over the whole floating point number slice in chunks with all channels, and fill each channel with the current value of our sine wave. + +The sine wave itself is calculated by `val = volume * sin(2 * PI * frequency * (i + accumulator) / rate)`, but we actually calculate it by simply increasing the accumulator by `2 * PI * frequency / rate` for every sample instead of doing the multiplication for each sample. We also make sure that the accumulator always stays between `0` and `2 * PI` to prevent any inaccuracies from floating point numbers to affect our produced samples. + +Now that this is done, we need to implement the `BaseSrc::create` virtual method for actually allocating the buffer, setting timestamps and other metadata and it and calling our above function. + +```rust + fn create( + &self, + element: &BaseSrc, + _offset: u64, + _length: u32, + ) -> Result { + // Keep a local copy of the values of all our properties at this very moment. This + // ensures that the mutex is never locked for long and the application wouldn't + // have to block until this function returns when getting/setting property values + let settings = *self.settings.lock().unwrap(); + + // Get a locked reference to our state, i.e. the input and output AudioInfo + let mut state = self.state.lock().unwrap(); + let info = match state.info { + None => { + gst_element_error!(element, gst::CoreError::Negotiation, ["Have no caps yet"]); + return Err(gst::FlowReturn::NotNegotiated); + } + Some(ref info) => info.clone(), + }; + + // If a stop position is set (from a seek), only produce samples up to that + // point but at most samples_per_buffer samples per buffer + let n_samples = if let Some(sample_stop) = state.sample_stop { + if sample_stop <= state.sample_offset { + gst_log!(self.cat, obj: element, "At EOS"); + return Err(gst::FlowReturn::Eos); + } + + sample_stop - state.sample_offset + } else { + settings.samples_per_buffer as u64 + }; + + // Allocate a new buffer of the required size, update the metadata with the + // current timestamp and duration and then fill it according to the current + // caps + let mut buffer = + gst::Buffer::with_size((n_samples as usize) * (info.bpf() as usize)).unwrap(); + { + let buffer = buffer.get_mut().unwrap(); + + // Calculate the current timestamp (PTS) and the next one, + // and calculate the duration from the difference instead of + // simply the number of samples to prevent rounding errors + let pts = state + .sample_offset + .mul_div_floor(gst::SECOND_VAL, info.rate() as u64) + .unwrap() + .into(); + let next_pts: gst::ClockTime = (state.sample_offset + n_samples) + .mul_div_floor(gst::SECOND_VAL, info.rate() as u64) + .unwrap() + .into(); + buffer.set_pts(pts); + buffer.set_duration(next_pts - pts); + + // Map the buffer writable and create the actual samples + let mut map = buffer.map_writable().unwrap(); + let data = map.as_mut_slice(); + + if info.format() == gst_audio::AUDIO_FORMAT_F32 { + Self::process::( + data, + &mut state.accumulator, + settings.freq, + info.rate(), + info.channels(), + settings.volume, + ); + } else { + Self::process::( + data, + &mut state.accumulator, + settings.freq, + info.rate(), + info.channels(), + settings.volume, + ); + } + } + state.sample_offset += n_samples; + drop(state); + + gst_debug!(self.cat, obj: element, "Produced buffer {:?}", buffer); + + Ok(buffer) + } +``` + +Just like last time, we start with creating a copy of our properties (settings) and keeping a mutex guard of the internal state around. If the internal state has no `AudioInfo` yet, we error out. This would mean that no caps were negotiated yet, which is something we can't handle and is not really possible in our case. + +Next we calculate how many samples we have to generate. If a sample stop position was set by a seek event, we have to generate samples up to at most that point. Otherwise we create at most the number of samples per buffer that were set via the property. Then we allocate a buffer of the corresponding size, with the help of the `bpf` field of the `AudioInfo`, and then set its metadata and fill the samples. + +The metadata that is set is the timestamp (PTS), and the duration. The duration is calculated from the difference of the following buffer's timestamp and the current buffer's. By this we ensure that rounding errors are not causing the next buffer's timestamp to have a different timestamp than the sum of the current's and its duration. While this would not be much of a problem in GStreamer (inaccurate and jitterish timestamps are handled just fine), we can prevent it here and do so. + +Afterwards we call our previously defined function on the writably mapped buffer and fill it with the sample values. + +With all this, the element should already work just fine in any GStreamer-based application, for example `gst-launch-1.0`. Don't forget to set the `GST_PLUGIN_PATH` environment variable correctly like last time. Before running this, make sure to turn down the volume of your speakers/headphones a bit. + +```bash +export GST_PLUGIN_PATH=`pwd`/target/debug +gst-launch-1.0 rssinesrc freq=440 volume=0.9 ! audioconvert ! autoaudiosink +``` + +You should hear a 440Hz sine wave now. + +### (Pseudo) Live Mode + +Many audio (and video) sources can actually only produce data in real-time and data is produced according to some clock. So far our source element can produce data as fast as downstream is consuming data, but we optionally can change that. We simulate a live source here now by waiting on the pipeline clock, but with a real live source you would only ever be able to have the data in real-time without any need to wait on a clock. And usually that data is produced according to a different clock than the pipeline clock, in which case translation between the two clocks is needed but we ignore this aspect for now. For details check the [GStreamer documentation](https://gstreamer.freedesktop.org/documentation/application-development/advanced/clocks.html). + +For working in live mode, we have to add a few different parts in various places. First of all, we implement waiting on the clock in the `create` function. + +```rust + fn create(...) + [...] + state.sample_offset += n_samples; + drop(state); + + // If we're live, we are waiting until the time of the last sample in our buffer has + // arrived. This is the very reason why we have to report that much latency. + // A real live-source would of course only allow us to have the data available after + // that latency, e.g. when capturing from a microphone, and no waiting from our side + // would be necessary. + // + // Waiting happens based on the pipeline clock, which means that a real live source + // with its own clock would require various translations between the two clocks. + // This is out of scope for the tutorial though. + if element.is_live() { + let clock = match element.get_clock() { + None => return Ok(buffer), + Some(clock) => clock, + }; + + let segment = element + .get_segment() + .downcast::() + .unwrap(); + let base_time = element.get_base_time(); + let running_time = segment.to_running_time(buffer.get_pts() + buffer.get_duration()); + + // The last sample's clock time is the base time of the element plus the + // running time of the last sample + let wait_until = running_time + base_time; + if wait_until.is_none() { + return Ok(buffer); + } + + let id = clock.new_single_shot_id(wait_until).unwrap(); + + gst_log!( + self.cat, + obj: element, + "Waiting until {}, now {}", + wait_until, + clock.get_time() + ); + let (res, jitter) = id.wait(); + gst_log!( + self.cat, + obj: element, + "Waited res {:?} jitter {}", + res, + jitter + ); + } + + gst_debug!(self.cat, obj: element, "Produced buffer {:?}", buffer); + + Ok(buffer) + } +``` + +To be able to wait on the clock, we first of all need to calculate the clock time until when we want to wait. In our case that will be the clock time right after the end of the last sample in the buffer we just produced. Simply because you can't capture a sample before it was produced. + +We calculate the running time from the PTS and duration of the buffer with the help of the currently configured segment and then add the base time of the element on this to get the clock time as result. Please check the [GStreamer documentation](https://gstreamer.freedesktop.org/documentation/application-development/advanced/clocks.html) for details, but in short the running time of a pipeline is the time since the start of the pipeline (or the last reset of the running time) and the running time of a buffer can be calculated from its PTS and the segment, which provides the information to translate between the two. The base time is the clock time when the pipeline went to the `Playing` state, so just an offset. + +Next we wait and then return the buffer as before. + +Now we also have to tell the base class that we're running in live mode now. This is done by calling `set_live(true)` on the base class before changing the element state from `Ready` to `Paused`. For this we override the `Element::change_state` virtual method. + +```rust +impl ElementImpl for SineSrc { + fn change_state( + &self, + element: &BaseSrc, + transition: gst::StateChange, + ) -> gst::StateChangeReturn { + // Configure live'ness once here just before starting the source + match transition { + gst::StateChange::ReadyToPaused => { + element.set_live(self.settings.lock().unwrap().is_live); + } + _ => (), + } + + element.parent_change_state(transition) + } +} +``` + +And as a last step, we also need to notify downstream elements about our [latency](https://gstreamer.freedesktop.org/documentation/application-development/advanced/clocks.html#latency). Live elements always have to report their latency so that synchronization can work correctly. As the clock time of each buffer is equal to the time when it was created, all buffers would otherwise arrive late in the sinks (they would appear as if they should've been played already at the time when they were created). So all the sinks will have to compensate for the latency that it took from capturing to the sink, and they have to do that in a coordinated way (otherwise audio and video would be out of sync if both have different latencies). For this the pipeline is querying each sink for the latency on its own branch, and then configures a global latency on all sinks according to that. + +This querying is done with the `LATENCY` query, which we will now also have to handle. + +```rust + fn query(&self, element: &BaseSrc, query: &mut gst::QueryRef) -> bool { + use gst::QueryView; + + match query.view_mut() { + // We only work in Push mode. In Pull mode, create() could be called with + // arbitrary offsets and we would have to produce for that specific offset + QueryView::Scheduling(ref mut q) => { + [...] + } + // In Live mode we will have a latency equal to the number of samples in each buffer. + // We can't output samples before they were produced, and the last sample of a buffer + // is produced that much after the beginning, leading to this latency calculation + QueryView::Latency(ref mut q) => { + let settings = *self.settings.lock().unwrap(); + let state = self.state.lock().unwrap(); + + if let Some(ref info) = state.info { + let latency = gst::SECOND + .mul_div_floor(settings.samples_per_buffer as u64, info.rate() as u64) + .unwrap(); + gst_debug!(self.cat, obj: element, "Returning latency {}", latency); + q.set(settings.is_live, latency, gst::CLOCK_TIME_NONE); + true + } else { + false + } + } + _ => BaseSrcImplExt::parent_query(self, element, query), + } + + } +``` + +The latency that we report is the duration of a single audio buffer, because we're simulating a real live source here. A real live source won't be able to output the buffer before the last sample of it is captured, and the difference between when the first and last sample were captured is exactly the latency that we add here. Other elements further downstream that introduce further latency would then add their own latency on top of this. + +Inside the latency query we also signal that we are indeed a live source, and additionally how much buffering we can do (in our case, infinite) until data would be lost. The last part is important if e.g. the video branch has a higher latency, causing the audio sink to have to wait some additional time (so that audio and video stay in sync), which would then require the whole audio branch to buffer some data. As we have an artificial live source, we can always generate data for the next time but a real live source would only have a limited buffer and if no data is read and forwarded once that runs full, data would get lost. + +You can test this again with e.g. `gst-launch-1.0` by setting the `is-live`property to true. It should write in the output now that the pipeline is live. + +`audiotestsrc` element also does it via `get_times` virtual method. But as this is only really useful for pseudo live sources like this one, we decided to explain how waiting on the clock can be achieved correctly and even more important how that relates to the next section. + +### Unlocking + +With the addition of the live mode, the `create` function is now blocking and waiting on the clock for some time. This is suboptimal as for example a (flushing) seek would have to wait now until the clock waiting is done, or when shutting down the application would have to wait. + +To prevent this, all waiting/blocking in GStreamer streaming threads should be interruptible/cancellable when requested. And for example the `ClockID` that we got from the clock for waiting can be cancelled by calling `unschedule()` on it. We only have to do it from the right place and keep it accessible. The right place is the `BaseSrc::unlock` virtual method. + +```rust +struct ClockWait { + clock_id: Option, + flushing: bool, +} + +struct SineSrc { + cat: gst::DebugCategory, + settings: Mutex, + state: Mutex, + clock_wait: Mutex, +} + +[...] + + fn unlock(&self, element: &BaseSrc) -> bool { + // This should unblock the create() function ASAP, so we + // just unschedule the clock it here, if any. + gst_debug!(self.cat, obj: element, "Unlocking"); + let mut clock_wait = self.clock_wait.lock().unwrap(); + if let Some(clock_id) = clock_wait.clock_id.take() { + clock_id.unschedule(); + } + clock_wait.flushing = true; + + true + } +``` + +We store the clock ID in our struct, together with a boolean to signal whether we're supposed to flush already or not. And then inside `unlock`unschedule the clock ID and set this boolean flag to true. + +Once everything is unlocked, we need to reset things again so that data flow can happen in the future. This is done in the `unlock_stop` virtual method. + +```rust + fn unlock_stop(&self, element: &BaseSrc) -> bool { + // This signals that unlocking is done, so we can reset + // all values again. + gst_debug!(self.cat, obj: element, "Unlock stop"); + let mut clock_wait = self.clock_wait.lock().unwrap(); + clock_wait.flushing = false; + + true + } +``` + +To make sure that this struct is always initialized correctly, we also call `unlock` from `stop`, and `unlock_stop` from `start`. + +Now as a last step, we need to actually make use of the new struct we added around the code where we wait for the clock. + +```rust + // Store the clock ID in our struct unless we're flushing anyway. + // This allows to asynchronously cancel the waiting from unlock() + // so that we immediately stop waiting on e.g. shutdown. + let mut clock_wait = self.clock_wait.lock().unwrap(); + if clock_wait.flushing { + gst_debug!(self.cat, obj: element, "Flushing"); + return Err(gst::FlowReturn::Flushing); + } + + let id = clock.new_single_shot_id(wait_until).unwrap(); + clock_wait.clock_id = Some(id.clone()); + drop(clock_wait); + + gst_log!( + self.cat, + obj: element, + "Waiting until {}, now {}", + wait_until, + clock.get_time() + ); + let (res, jitter) = id.wait(); + gst_log!( + self.cat, + obj: element, + "Waited res {:?} jitter {}", + res, + jitter + ); + self.clock_wait.lock().unwrap().clock_id.take(); + + // If the clock ID was unscheduled, unlock() was called + // and we should return Flushing immediately. + if res == gst::ClockReturn::Unscheduled { + gst_debug!(self.cat, obj: element, "Flushing"); + return Err(gst::FlowReturn::Flushing); + } +``` + +The important part in this code is that we first have to check if we are already supposed to unlock, before even starting to wait. Otherwise we would start waiting without anybody ever being able to unlock. Then we need to store the clock id in the struct and make sure to drop the mutex guard so that the `unlock` function can take it again for unscheduling the clock ID. And once waiting is done, we need to remove the clock id from the struct again and in case of `ClockReturn::Unscheduled` we directly return `FlowReturn::Flushing` instead of the error. + +Similarly when using other blocking APIs it is important that they are woken up in a similar way when `unlock` is called. Otherwise the application developer's and thus user experience will be far from ideal. + +### Seeking + +As a last feature we implement seeking on our source element. In our case that only means that we have to update the `sample_offset` and `sample_stop` fields accordingly, other sources might have to do more work than that. + +Seeking is implemented in the `BaseSrc::do_seek` virtual method, and signalling whether we can actually seek in the `is_seekable` virtual method. + +```rust + fn is_seekable(&self, _element: &BaseSrc) -> bool { + true + } + + fn do_seek(&self, element: &BaseSrc, segment: &mut gst::Segment) -> bool { + // Handle seeking here. For Time and Default (sample offset) seeks we can + // do something and have to update our sample offset and accumulator accordingly. + // + // Also we should remember the stop time (so we can stop at that point), and if + // reverse playback is requested. These values will all be used during buffer creation + // and for calculating the timestamps, etc. + + if segment.get_rate() < 0.0 { + gst_error!(self.cat, obj: element, "Reverse playback not supported"); + return false; + } + + let settings = *self.settings.lock().unwrap(); + let mut state = self.state.lock().unwrap(); + + // We store sample_offset and sample_stop in nanoseconds if we + // don't know any sample rate yet. It will be converted correctly + // once a sample rate is known. + let rate = match state.info { + None => gst::SECOND_VAL, + Some(ref info) => info.rate() as u64, + }; + + if let Some(segment) = segment.downcast_ref::() { + use std::f64::consts::PI; + + let sample_offset = segment + .get_start() + .unwrap() + .mul_div_floor(rate, gst::SECOND_VAL) + .unwrap(); + + let sample_stop = segment + .get_stop() + .map(|v| v.mul_div_floor(rate, gst::SECOND_VAL).unwrap()); + + let accumulator = + (sample_offset as f64).rem(2.0 * PI * (settings.freq as f64) / (rate as f64)); + + gst_debug!( + self.cat, + obj: element, + "Seeked to {}-{:?} (accum: {}) for segment {:?}", + sample_offset, + sample_stop, + accumulator, + segment + ); + + *state = State { + info: state.info.clone(), + sample_offset: sample_offset, + sample_stop: sample_stop, + accumulator: accumulator, + }; + + true + } else if let Some(segment) = segment.downcast_ref::() { + use std::f64::consts::PI; + + if state.info.is_none() { + gst_error!( + self.cat, + obj: element, + "Can only seek in Default format if sample rate is known" + ); + return false; + } + + let sample_offset = segment.get_start().unwrap(); + let sample_stop = segment.get_stop().0; + + let accumulator = + (sample_offset as f64).rem(2.0 * PI * (settings.freq as f64) / (rate as f64)); + + gst_debug!( + self.cat, + obj: element, + "Seeked to {}-{:?} (accum: {}) for segment {:?}", + sample_offset, + sample_stop, + accumulator, + segment + ); + + *state = State { + info: state.info.clone(), + sample_offset: sample_offset, + sample_stop: sample_stop, + accumulator: accumulator, + }; + + true + } else { + gst_error!( + self.cat, + obj: element, + "Can't seek in format {:?}", + segment.get_format() + ); + + false + } + } +``` + +Currently no support for reverse playback is implemented here, that is left as an exercise for the reader. So as a first step we check if the segment has a negative rate, in which case we just fail and return false. + +Afterwards we again take a copy of the settings, keep a mutable mutex guard of our state and then start handling the actual seek. + +If no caps are known yet, i.e. the `AudioInfo` is `None`, we assume a rate of 1 billion. That is, we just store the time in nanoseconds for now and let the `set_caps` function take care of that (which we already implemented accordingly) once the sample rate is known. + +Then, if a `Time` seek is performed, we convert the segment start and stop position from time to sample offsets and save them. And then update the accumulator in a similar way as in the `set_caps` function. If a seek is in `Default` format (i.e. sample offsets for raw audio), we just have to store the values and update the accumulator but only do so if the sample rate is known already. A sample offset seek does not make any sense until the sample rate is known, so we just fail here to prevent unexpected surprises later. + +Try the following pipeline for testing seeking. You should be able to seek the current time drawn over the video, and with the left/right cursor key you can seek. Also this shows that we create a quite nice sine wave. + +```bash +gst-launch-1.0 rssinesrc ! audioconvert ! monoscope ! timeoverlay ! navseek ! glimagesink +``` + +And with that all features are implemented in our sine wave raw audio source. \ No newline at end of file