DEV Community: PANDA

Wavelet Buffer In A Nutshell

Alexey Timin — Wed, 16 Nov 2022 08:37:45 +0000

Wavelet Buffer is a core technology for data processing in the PANDA|Drift Platform. It provides the following
features:

Storing data as wavelet subbands so that you can restore the smallest version of the signal
Denoising high-frequency spectrum
Compressing data with float-compression, so you can choose the proper “float” for your data with adjustable size of the mantissa.

The Simplest Explanation of Wavelet Decomposition

There is a beautiful and tricky math behind the wavelet decomposition, but for our engineering tasks it is enough to
consider the wavelet decomposition as a cascade of low-pass and high-pass filters + N/2 resampler. When we pass our
signal through the low-pass filter, we have twice smaller version of our signal (approximation or low frequency subband)
. When we pass
our signal through the high-pass filter, we have twice smaller subband with details and noise (detalization or
high-frequency subband).

So in the end we have two subbands of N/2 size that can be composed again, and we can have the original signal. This
approach has the following advantages:

We have noise in the high-frequency subband separately of the low frequency signal, and we can denoise it by setting the small values to zero and make the subband sparse. This is a critical part because when you measure some vibration of some machine that is currently stopped, you have only noise instead of the signal. So, we can reduce the size of this data almost to zero and store only the real signal on the disk.
We have a smaller version of the signal in low-level subband. For example, if we would like to show our signal in the browser and the monitor has resolution 1920x1200, it doesn’t make any sense to provide the signal with the size more than 1920 points. So, we can compose only some low-level subbands without reconstructing the whole original signal.

In the diagram, I showed only one step of the decomposition, but we can decompose the signal recursively, by applying
the wavelet decomposition on the low-frequency subband again. Therefore, we can decompose our signal to a few subbands:
N/1 H-freq subband (D), N/2 H-freq subband (AD), …. N/M L-freq subband (ADm), where N size of the original signal and M
– the number of the decomposition steps. After the decomposition, we have a representation of our signal as an array of
subbands e.ge for 3 steps it will be [AAAA, ADDD, ADD, AD, D]. When we make many steps of the decomposition, we do better
denoising and zooming because now we can get signal of sizes N/2, N/4 … or N/M if we restore only part of all the steps.

Wavelet Decomposition can be applied to 2-D matrices, the difference is that we have 4 types of subbands:

Low frequency approximation (A)
High frequency detalization by vertical (Dv)
High frequency detalization by horizontal (Dh)
High frequency detalization by diagonal (Dd)

Here you can see how it looks like for one decomposition step:

Structure

In a nutshell, Wavelet is just a wrapper around the wavelet subbands:

It has 4 main methods:

Decompose. It decomposes the input signal to the subbands and denoises the high-frequency ones.
Compose. It reconstructs the original signal from subbands. It may reconstruct only few steps of the decomposition so that we can get approximation of the signal N/2, N/4 or N/M (N size of original signal, M—number decomposition steps) and spare some computation time.
Serialize. However, subbands are sparse after denoising, they still contain zeros in the memory and occupy the same memory space as before denoising. This method compresses the subbands to binary data with Sparse Float Compression algorithm. We use it before saving a Wavelet Buffer onto the disk or sending it through network. You can read about it below.
Parse. Decompresses a Wavelet Buffer and restores subbands

Sparse Float Compression

Sparse Float Compression Algorithm is a specific compression algorithm to compress sparse float data. It does three
things:

Removing zeros and add leave only values.
Compressing the data, so that the more frequent values spend fewer bits for encoding.
Casts the values of the subbands to an adjustable float container, so that we can choose the size of float mantissa. We need it if we have lower resolution of our data, and 32-bit float is too big for this.

The current implementation of the compressor supports the following size of the mantissa:

Size in bits	Compression level in WaveletBuffer::Serialize
No compression. The algorithm is bypassed.	0
23 (float32)	1
21	2
20	3
...	...
7 (bfloat)	16

!!! warning
Because we use float values, we always have some error, which depends on the range of the values. You should adjust
compression level by using knowledge about your data.

Scalar Values

Some data don’t need any denoising and compression, for example GPS coordinates or slow data from automation systems. In
this case, we can switch off the compression and wavelet composition in WB and make it simple and stupid a vector or
matrix of the data.

Wavelet Functions

WaveletBuffer supports a few wavelet functions to tweak the wavelet decomposition:

Name	Description
NONE	No wavelet decomposition and denoising. See Scalar Values for detail.
DB1	Daubechies wavelet DB2 or Haar
...	..
DB5	Daubechies wavelet DB10

Denoising time series data by using wavelet transformation

Alexey Timin — Mon, 12 Sep 2022 15:34:32 +0000

A popular approach for conditional monitoring of mechanical machines is to embed vibration sensors into a machine and start "listening" to it. The data from the sensors must be stored somewhere. So, the more efficient we compress them, the longer history we can keep. When we deal with sensors we always have noise from them, and when a machine is stopped we have only noise which wastes our storage space. WaveletBuffer was developed to solve this problem by using the wavelet transformation and efficient compression of denoised data. However, a user must know many settings to use it efficiently. In this tutorial, you will learn how to find denoising parameters to get rid of white noise in your data.

Basics

For an educational purpose, I brought two samples from a vibration sensor which is maintained on a real machine. You can find them in the directory docs/tutorials. buffer_signal.bin contains a second of the signal when the machine is working. buffer_no_signal.bin has only noise because the machine is stopped. Both files are serialized buffer without any denoising, so they have equal sizes but different amount of the information!

We need to separate white noise and information, so we need the sample with the working machine. Let's deserialize it:

from wavelet_buffer import WaveletBuffer, denoise
import numpy as np
import matplotlib.pyplot as plt

blob = b""
with open("buffer_signal.bin", "rb") as f:
    blob = f.read()

print(f"Initial size {len(blob)/ 1000} kB")
buffer = WaveletBuffer.parse(blob)
print(buffer)

Output:

Initial size 194.918 kB
WaveletBuffer<signal_number=1, signal_shape=(48199), decomposition_steps=9, wavelet_type=WaveletType.DB3>

You can see that the signal has about 48000 points and the size of the file is about 200kB. Let's have a look at the signal inside:

signal = buffer.compose()

plt.plot(signal)
plt.show()

Looks noisy, right? When we've restored the original signal, we can use WaveletBuffer to denoise it. We use some random threshold for the demonstration. Later we learn how to find the optimal parameter. When we pass denoise.Threshold(a=0, b=0.04) in WaveletBuffer.decompose method it set to 0 all values in hi-frequency subbands which less than a*x+b where x is a decomposition step of the current subband.

buffer.decompose(
    signal, denoise.Threshold(a=0, b=0.04)
)  # this a*x + 0.04 where x is a decomposition step
denoised = buffer.compose()
plt.plot(denoised)
plt.show()

Information Loosing Detection

The denoised signal looks the same, but we actually don't know if we removed only the noise, or we removed some part of the signal. To figure out it, we can take the difference between the denoised and original signals and check if it has some information or only noise. It has some information and means we deleted with denoising.

err = denoised - signal
plt.plot(err)
plt.show()

To check if the difference (err) has something more than only noise we can find autocorrelation of error and test if all the values lie within [-1.96/sqrt(n), 1.96.sqrt(n)] where n is the size of sample. For more detail, see here.


def autocorr(x):
    result = np.correlate(x, x, mode="full")
    return result[int(len(result) / 2) :]


def test(corr: np.ndarray):
    thr = 1.96 / np.sqrt(len(corr))
    tests = np.where(np.abs(corr) <= thr, 0, 1)
    return tests.sum() == 0


corr = autocorr(err)
print(test(corr))

Output

False

Optimal Denoise Parameters

You can see that for denoise.Threshold(0, 0.04) we removed something important from our signal. However, the desnoised signal looked the same. To find the optimal threshold, we can start from 0 and increasing the threshold with a little step until we fail the test:

threshold = 0
step = 0.0001
for x in range(1000):
    threshold = step * x
    buffer.decompose(signal, denoise.Threshold(0, threshold))
    err = buffer.compose() - signal
    corr = autocorr(err)
    if not test(corr):
        threshold = threshold - step
        print(f"Optimal threshold: {threshold}")
        break

Output:

Optimal threshold: 0.0017000000000000001

You see that the threshold way less than our initial assumption. Let's check the size of serialized signal:

buffer.decompose(signal, denoise.Threshold(0, threshold))
print(f"Compression size: {len(buffer.serialize(compression_level=16)) / 1000} kB")

signal = buffer.compose()

plt.plot(signal)
plt.show()

Output:

Compression size: 58.528 kB

Looks like it got 3.5 times smaller after the demonising. However, the most interesting thing is compression of the sample when the machine was stopped:

blob = b""
with open("buffer_no_signal.bin", "rb") as f:
    blob = f.read()

buffer = WaveletBuffer.parse(blob)
buffer.decompose(buffer.compose(), denoise.Threshold(0, threshold))
signal = buffer.compose()

print(
    f"Compression (onlynoise) size: {len(buffer.serialize(compression_level=16)) / 1000} kB"
)

plt.plot(signal)
plt.show()

Output:

Compression (onlynoise) size: 0.41 kB

The compressed size is 500 times smaller now, because we don't have valuable information in the sample.

Conclusion

WaveletBuffer provides a pipeline wavelet transormation -> denoising -> compression which is useful for efficient compression of height frequency time series data. This approach has an additional advantage for data sources which don't provide valuable information all the time but are requested continuously.

References:

White Noise Model - Time Series Analysis, Regression and Forecasting
WaveletBuffer - A universal C++ compression library based on wavelet transformation
WaveletBuffer for Python - Python bindings for WaveletBuffer