DEV Community

Igor Montagner
Igor Montagner

Posted on β€’ Edited on

1

One Pull Request per month: January

This year I'll be doing a challenge: submit at least one Pull Request to a FLOSS project. January's PR is a fix to No audio is recorded with videos #148 , a bug in Elementary OS 6's "rewrite" of the Camera app. The backend was changed to pure GStreamer and this introduced some regressions. However, it also allows for better device detection and possibly more control over video and audio settings. Hopefully this new version will solve many of the "Camera does not work on XXX" issues.

Having worked on Camera in the past (see my contributions in my previous post), I decided to tackle a new issue: no sound was being recorded on the app. After some investigation I realized that this was expected: the app never requested sound to be recorded!

GStreamer works with data processing pipelines with three basic elements:

  1. sources, such as webcams, microphones or files, forward data from an external source to the pipeline
  2. sinks, such as files or user interface elements, receive data from the pipeline and either save it or display it.
  3. elements receive data from the pipeline, transform it and forward this result to other elements or a sink.

Thus, data flow from one or more sources, passes thru many elements that transform it and finally reaches a sink where it is stored or shown to the user. The following pipeline was being used:

"v4l2src device=%s name=v4l2src !"
"video/x-raw, width=640, height=480, framerate=30/1 ! " +
"videoflip method=horizontal-flip name=hflip ! " +
"videobalance name=balance ! " +
"tee name=tee ! " +
"queue leaky=downstream max-size-buffers=10 ! " +
"videoconvert ! " +
"videoscale ! " +
"gtksink name=gtksink"
Enter fullscreen mode Exit fullscreen mode

We only have a single source, v4l2src, which corresponds to a Video for Linux 2 device (usually webcams or other external USB cameras). Data is passed thru many elements using a "!" character until reaching a gtksink that shows the video preview. To record videos a new branch was connected to the tee element that included a video encoder and a filesink that finally stored the video to a file. Thus, the current pipeline only has a single source that produces video frames.

To record audio we need a source that produces audio. This can be done by adding an alsasrc element that produces audio using the ALSA sound system. Gstreamer offers a very neat way of testing pipelines: the gst-launch-1.0 app.
The following sequence of elements obtains data from the microphone using ALSA, encodes it using the Vorbis codec and packages it into a Ogg container.

gst-launch-1.0 -e alsasrc ! audioconvert ! vorbisenc ! oggmux !  filesink location="aa.ogg"
Enter fullscreen mode Exit fullscreen mode

Notice that we still can't produce a file with both audio and video! Both pipelines produce only one or the other. That's where the oggmux element comes into play: its role is to combine audio and video streams into a stream. Thus, to record audio as well we only need to add the elements above to the existing pipeline and link them to the existing oggmux element. The code below does the job and is the basis of my January PR: Add audio recording elements to record_bin. #153 .


var alsasrc = Gst.ElementFactory.make ("alsasrc", null);
if (alsasrc == null) {
    missing_messages += Gst.PbUtils.missing_element_installer_detail_new ("alsasrc");
}

var audio_queue = Gst.ElementFactory.make ("queue", null);

var audio_convert = Gst.ElementFactory.make ("audioconvert", null);
if (audio_convert == null) {
    missing_messages += Gst.PbUtils.missing_element_installer_detail_new ("audioconvert");
}

var audio_vorbis = Gst.ElementFactory.make ("vorbisenc", null);
if (audio_vorbis == null) {
    missing_messages += Gst.PbUtils.missing_element_installer_detail_new ("vorbisenc");
}

// ...

record_bin.add_many (alsasrc, audio_queue, audio_convert, audio_vorbis);
alsasrc.link_many (audio_queue, audio_convert, audio_vorbis, muxer);

Enter fullscreen mode Exit fullscreen mode

The code is rather self explanatory and is basically a Vala version of the command line pipeline I showed earlier. And... that's it. I'm now waiting for the maintainers to review it and begin the improvements so it eventually gets merged. Now I only have 11 PRs left for 2021. Let's keep them coming!

Do your career a big favor. Join DEV. (The website you're on right now)

It takes one minute, it's free, and is worth it for your career.

Get started

Community matters

Top comments (0)

Sentry image

See why 4M developers consider Sentry, β€œnot bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more

πŸ‘‹ Kindness is contagious

Explore a sea of insights with this enlightening post, highly esteemed within the nurturing DEV Community. Coders of all stripes are invited to participate and contribute to our shared knowledge.

Expressing gratitude with a simple "thank you" can make a big impact. Leave your thanks in the comments!

On DEV, exchanging ideas smooths our way and strengthens our community bonds. Found this useful? A quick note of thanks to the author can mean a lot.

Okay