Creating your own Processing Block

Super charge your Rune with pre and post processing functions

The Expressive Power of Rune

In our last newsletter we introduced you to the concept of Rune, our containerization technology for tinyML applications. To recap, we defined it as:

Rune is an orchestration tool for specifying how data should be processed, with an emphasis on the machine learning world, in a way which is portable and robust.

The main purpose of a Rune is to give developers in the fields of machine learning and data processing a way to declare how data should be transformed using a high level, declarative language.

Processing blocks (PROC_BLOCK directive in a Runefile) allow developers to express powerful processing pipelines in typical machine learning workflow, all declaratively.

What is the purpose of a Processing Block?

TinyML models trained to run on edge devices require data to be in a specific format. The models are also trained on fewer features that may not be generated in raw form by the device natively. We could build a model that is trained on raw features from the device but then the models could become really big and not work on constrained devices. It is easier to convert raw data into input feature data and from output features to output data via a processing pipeline, thus keeping the models as compact as possible.

As an example in the image below, the Rune pipeline we introduced in the earlier newsletter shows how we can take raw sound data and feed that into a microspeech tensorflow lite model to detect keywords. The FFT part of the pipeline takes audio signals and convert them to features that the model can predict on.

Processing Blocks (Proc Blocks) are the HOTG building blocks for constructing Machine Learning (ML) pipelines that can be run on the edge. In this blog post, we will implement an example STFT Proc Block to illustrate the process of creating, implementing, and sharing your own Proc Block.

What does a Proc Block look like?

A Proc Block is implemented as a struct in Rust code, with a transform method and optional other methods. The purpose of a Proc Block is to transform an input data item into an output data item using the transform method.

In the pseudocode example given below this FFT Proc Block performs STFT (Short Time Fourier Transform) in a sliding window manner over the clip when hard-coded:

use runic_types::Transform;

/// A [`Transform`] which implements the *Short Time Fourier
/// Transform*.
pub struct Fft {
    sample_rate: u32, 
    bins: usize, 
    window_overlap: f32,
}

impl Fft {
    /// Create a new `Fft` with some sane defaults.
    pub fn new() -> Self {
        Fft {
            sample_rate: 16000, 
            bins: 49, 
            window_overlap: 0.66,
        }
    }
}

In our example, we have hard-coded our Proc Block to have 49 STFT time bins, with a bin window overlap of 0.66 - to fit the TFLITE microspeech model often used in TinyML communities.

When given a waveform as input data, the Proc Block gives a normalized spectrogram of that waveform as output. Using examples of sinusoidal inputs of three different frequencies, we see input and output:

As expected, the higher frequency signals result in FFT responses in higher frequency bins.

How is a Proc Block tested?

Once coded, Proc Blocks must be tested for functional errors and performance. In order to do this, tests can be written in Rust that call the Proc Block. We write an example test where the Proc Block is called with a one-second length signal off all zeros. Return data is then verified to be of the right shape of 1960 elements (49 time bins by 40 frequency bins).

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn it_works() {
        let mut fft_pb = Fft::new().with_sample_rate(16000);
        let input = [0; 16000];

        let res = fft_pb.transform(input);

        assert_eq!(res.len(), 1960)
    }
}

Proc Blocks can also be called and tested through Python, through the use of maturin package.

How can Constant Generics be used to expand the input data format?

Constant Generics (const generics) can be used to enable a Proc Block to work for a greater scope of input types. A Const Generic allows a Proc Block to process input data of any type for which the Const Generic has been defined.

impl<const N: usize> runic_types::Transform<[i16; N]> for Fft {
    type Output = [u8; 1960];

    fn transform(&mut self, input: [i16; N]) -> Self::Output {
        Functionality to calculate spectrogram
        return out;
    }
}

What’s Next?

When we meet next time, we will walk through how to make a rust crate that can be attached to your Runefile project. We will also be releasing our open source code at the end of May so stay tuned!

Look out for a follow up post on how to build, test and publish an actual working proc block. Subscribe & follow us to check out more upcoming posts!