Client side Audio Editing

fiveko — Wed, 22 Jun 2022 06:17:39 +0000

Audio editing as an exciting topic that involve different knowledge and skills. Whether we are a professional musician or a regular user, sometimes we have to edit an audio file.
There are various audio tools, but my personal focus is on online technology. In general, the online audio tool includes the following areas:

Digital signal processing

Digital audio processing is a sub-field of Digital signal processing. This an interesting topic that appears in life in various forms. The purpose of this process is to extract useful information from the signal. Some common applications are:

Audio editors
Voice recognition
Text to speech
Sound matching

Like digital image processing, sound processing involves mathematical transformations to analyze or modify the signal.

Web technologies

Unlike native apps, online software depends on available web technologies. Fortunately, modern web standards provide quite useful tools for manipulating and reproducing sound.

WebAssembly

I put WebAssembly (Wasm) first here because I'm a big fan of this technology. It allows all kind of software developers to use the programming language that they love.
I prefer to work with the emscripten compiler tool chain and find it quite convenient for me.

Web Audio API

The Web Audio API is an essential addition to the web standard. It gives ability to manage audio sources, apply sound filters, effects, decode media files and more.

WebWorker

Sometimes our web app has to do some hard work in parallel and keep the UI thread unlocked. In such cases we resort to a working thread, which we start from WebWorker.
One of the frequently mentioned shortcomings of the worker is the lack of DOM access. However, it has access to a large number of common objects such as WebSockets, IndexedDB, Network IO and so on.

AudioWorklet

The AudioWorklet is relatively new addition to the Web Audio API. This tech gives ability to attach a custom JavaScript for audio processing. Like WebWorker, it runs in a dedicated thread to provide low latency during sound manipulation.
Note: This feature works in secure context (HTTPS) on most / all browsers!

WebCodecs API

The WebCodecs API is a recent extension of the web standard. As far as I know, it is currently supported by Google Chrome and Microsoft Edge, but not by Mozilla Firefox.
My experience with this technology is that it works well for video encoding in VP8 and VP9 and OPUS for audio. Unfortunately, there is currently no build-in media muxer. This means that we have to use / write a separate component when we want to export WebM, MKV, MP4 or similar container.

You may read the Video Processing with WebCodecs article for deeper details.

Audio editor for the Web

There are different web applications for audio processing. However, I want to mention the Sound CMD. This free online sound editor is still under development, but there are features that are of interest to me.

Multi track support - can open multiple files in separate tabs
Spectrogram viewer, player and editor - Can zoom into sound spectrum to listen (or edit) isolated frequency bands.
Sound generator - generate signal with different amplitude, phase or shape

Useful resources

Image edge detection for Web Apps

fiveko — Wed, 10 Feb 2021 08:11:03 +0000

Image edge detection is a fundamental part of many computer vision and image processing algorithms. The edges describe the shape of the objects and therefore they carry valuable information about the picture.

Popular edge detection techniques

One of the most popular algorithms for edge detection are the gradient operators.

First-order gradient operators

Some of the most popular first-order gradient detection operators are:

Prewitt operator
Sobel operator
Schaar operator

Second-order operators

The edge detectors from this group use the second-order derivative of image intensity. This means that they detect the rate of change in intensity, since the maximum of the gradient coincides with zero-crossing in the second derivative.
Popular algorithms are:

Laplacian of Gauss
Difference of Gauss (DoG)
Marr–Hildreth algorithm

How to make edge detector for the web?

Nowadays, it is possible to use modern web technologies to perform image analysis in a browser. The advanced HTML5 standard is widely supported in most popular platforms and this provides many opportunities.

Image filtering using CSS

The CSS filter property allows you to apply effects to images and various parts of HTML content such as borders and background.
Unfortunately this is not much of an use for image analysis and machine vision.

Edge detection with HTML5 Canvas

The HTML5 Canvas element is a powerful tool that provides access to raw image data. Due to this we can use vanilla JavaScript to manipulate and analyze images online.

Image analysis with WebGL

WebGL (Web Graphics Library) is another web technology that enables high-performance image processing. The best part about it is that it works without plugins, so we can use the direct JavaScript API to manipulate pixel data.

Online image processing apps

If you are interested in image processing, you can see these online graphics apps. In addition to applications, this project offers free tutorials and image analysis algorithms.

DEV Community: fiveko