From a63d9c13fff6db00937d005e2d5da8765de987a8 Mon Sep 17 00:00:00 2001 From: Matthew Clark Date: Wed, 28 Mar 2018 23:12:16 +0100 Subject: [PATCH] More mixing examples --- mixing.md | 120 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-) diff --git a/mixing.md b/mixing.md index c05f681..b3cd169 100644 --- a/mixing.md +++ b/mixing.md @@ -1,6 +1,10 @@ # Mixing (GStreamer command-line cheat sheet) -This page talks about mixing video (i.e. replacing or overlaying), and also mixing audio (i.e. replacing or merging audio tracks). We'll do one at a time. +Page contents: + +* Mixing video (i.e. replacing or overlaying) +* Mixing audio (i.e. replacing or merging audio tracks) +* Mixing video & audio together ## Mixing video @@ -125,3 +129,117 @@ gst-launch-1.0 \ audiotestsrc is-live=true freq=400 ! audioconvert ! mix. \ filesrc location=$AUDIO_SRC ! mpegaudioparse ! decodebin ! audioconvert ! mix. ``` + +## Mixing video & audio together + +### Mix two fake video sources and two fake audio Sources + +We use `compositor` to mix the video and `audiomixer` to mix the audio. + +This example combines two test video inputs and also two test audio inputs: + +``` +gst-launch-1.0 \ + compositor name=videomix ! autovideosink \ + audiomixer name=audiomix ! audioconvert ! autoaudiosink \ + videotestsrc pattern=ball ! videomix. \ + videotestsrc pattern=pinwheel ! videoscale ! video/x-raw,width=100 ! videomix. \ + audiotestsrc freq=400 ! audiomix. \ + audiotestsrc freq=600 ! audiomix. +``` + +The output looks like: + +![Pinwheel and ball](images/pinwheel_and_ball.png "Pinwheel and ball") + +The above example is simple because we didn't have to split or combine the audio and video. +This example muxes the mixed audio & video together, and then outputs via TCP. + +``` +# View this in VLC with tcp://localhost:7001 +gst-launch-1.0 \ + mpegtsmux name=mux ! \ + tcpserversink port=7001 host=0.0.0.0 recover-policy=keyframe sync-method=latest-keyframe sync=false \ + compositor name=videomix ! x264enc ! queue2 ! mux. \ + audiomixer name=audiomix ! audioconvert ! audioconvert ! audioresample ! avenc_ac3 ! queue2 ! mux. \ + videotestsrc pattern=ball ! videomix. \ + videotestsrc pattern=pinwheel ! videoscale ! video/x-raw,width=100 ! videomix. \ + audiotestsrc freq=400 ! audiomix. \ + audiotestsrc freq=600 ! audiomix. +``` + +### Mix a AV file with fake video and audio + +This one puts a bouncing ball in the corner of a file: + +``` +gst-launch-1.0 \ + compositor name=videomix ! autovideosink \ + audiomixer name=audiomix ! audioconvert ! autoaudiosink \ + filesrc location=$SRC ! qtdemux name=demux \ + demux.video_0 ! queue2 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=360 ! videomix. \ + demux.audio_0 ! queue2 ! decodebin ! audioconvert ! audioresample ! audiomix. \ + videotestsrc pattern=ball ! videoscale ! video/x-raw,width=100,height=100 ! videomix. \ + audiotestsrc freq=400 volume=0.1 ! audiomix. +``` + +and this one also muxes the video and audio together to be sent over TCP: + +``` +# View this in VLC with tcp://localhost:7001 +gst-launch-1.0 \ + mpegtsmux name=mux ! \ + tcpserversink port=7001 host=0.0.0.0 recover-policy=keyframe sync-method=latest-keyframe sync=false \ + compositor name=videomix ! x264enc ! queue2 ! mux. \ + audiomixer name=audiomix ! audioconvert ! audioconvert ! audioresample ! avenc_ac3 ! queue2 ! mux. \ + filesrc location=$SRC ! qtdemux name=demux \ + demux.video_0 ! queue2 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=360 ! videomix. \ + demux.audio_0 ! queue2 ! decodebin ! audioconvert ! audioresample ! audiomix. \ + videotestsrc pattern=ball ! videoscale ! video/x-raw,width=100,height=100 ! videomix. \ + audiotestsrc freq=400 volume=0.1 ! audiomix. +``` + +This one uses `uridecodebin` which allows a wider range of inputs to be added: + +``` +gst-launch-1.0 \ + compositor name=videomix ! autovideosink \ + audiomixer name=audiomix ! audioconvert ! autoaudiosink \ + uridecodebin uri=file://$SRC name=demux ! \ + queue2 ! audioconvert ! audioresample ! audiomix. \ + demux. ! queue2 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=360 ! videomix. \ + videotestsrc pattern=ball ! videoscale ! video/x-raw,width=100,height=100 ! videomix. \ + audiotestsrc freq=400 volume=0.1 ! audiomix. +``` + +And again, here's a version that muxes again to send to TCP: + +``` +# View this in VLC with tcp://localhost:7001 +gst-launch-1.0 \ + mpegtsmux name=mux ! \ + tcpserversink port=7001 host=0.0.0.0 recover-policy=keyframe sync-method=latest-keyframe sync=false \ + compositor name=videomix ! x264enc ! queue2 ! mux. \ + audiomixer name=audiomix ! audioconvert ! audioresample ! avenc_ac3 ! queue2 ! mux. \ + uridecodebin uri=file://$SRC name=demux ! \ + queue2 ! audioconvert ! audioresample ! audiomix. \ + demux. ! queue2 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=360 ! videomix. \ + videotestsrc pattern=ball ! videoscale ! video/x-raw,width=100,height=100 ! videomix. \ + audiotestsrc freq=400 volume=0.2 ! audiomix. +``` + +### Mix two AV files + +This does picture-in-picture, with the audio from both files included. + +``` +gst-launch-1.0 \ + compositor name=videomix ! autovideosink \ + audiomixer name=audiomix ! audioconvert ! autoaudiosink \ + uridecodebin uri=file://$SRC name=demux1 ! \ + queue2 ! audioconvert ! audioresample ! audiomix. \ + demux1. ! queue2 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=360 ! videomix. \ + uridecodebin uri=file://$SRC2 name=demux2 ! \ + queue2 ! audioconvert ! audioresample ! audiomix. \ + demux2. ! queue2 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=320,height=180 ! videomix. +```