Generating Art with Edge AI and why it is a big deal for Rune Runtimes
Deploying multiple model ML pipelines to the Edge
Last week, we mentioned some work was done to “Runify” a Style Transfer model, but we didn’t go into too much detail on why this was such a big step forward.
The Style Transfer Rune is an ML pipeline which uses two models to generate a new image by applying the “style” from one image onto another. For example, you might give the Rune a painting and a photo from your phone’s camera, and the Rune will apply the painting’s color scheme and brush strokes to the photo.
Over the last two weeks, the team and our open source community on Discord hacked together several interfaces to make this work. In this blog post, we will explore one of our less talked about topics, the Rune runtime. The #rune-runtime channel on discord is where you can see some of the work we have been doing on the runtime.
New Additions for the Rune Runtime
As you may remember, a “runtime” is needed to execute WebAssembly code and provide the Rune with access to specific host functions.
Put simply, a host function is a Rune's lifeline to the outside world. Any data that is consumed by the Rune or passed to the application needs to be sent via host functions explicitly provided by the Runtime. Host functions are also required if the Rune wants to do anything other than pure computation.
To make Runes like Style Transfer work, the Rune runtime needed to be able to do inference on models that have multiple inputs and outputs. This was easy enough to do for our web and desktop runtimes, however our Runic Mobile app is a flutter application built on an older C++ runtime that links to TensorFlow Lite (for model inference) and Wasm3 (for interpreting WebAssembly) and bringing this new functionality to the mobile world required a fair amount of effort.
Hammer of the Gods is a Rust-first company and going forwards we will be pushing to make our Rust runtime (currently used on PCs and servers) work on the phone itself. This new work will bring a tremendous amount of consistency and safety to our mobile offerings, while letting us build in more stability and extensibility.
We are excited to use this extensibility to develop more domain-specific Runtimes for various verticals such as Crypto, SecureVM, and more. Stay tuned on our Github as we gear to start building the one Rust Runtime to rule them all.
The Art!
While it was a fun technical challenge, the real benefit of this work is that people now have a novel way of producing art by combining a style image (e.g. Vincent van Gogh’s Starry Night) with pictures take on their phone.
Check this out!
P.s. We plan to make some fun and games with the Runic Mobile app that will let the wider community also create some fun art using Edge AI. Until next time on TinyVerse.
Join our Community:
Twitter @hotg_ai and @hammer_otg | LinkedIn | Discord | Discourse