https://hackaday.com/2024/07/30/ai-image-generator-twists-in-response-to-midi-dials-in-real-time/

MIDI isn’t just about music, as [Johannes Stelzer] shows by using dials to adjust AI-generated imagery in real-time. The results are wild, with an interactivity to them that we don’t normally see in such things.

[Johannes] uses Stable Diffusion‘s SDXL Turbo to create a baseline image of “photo of a red brick house, blue sky”. The hardware dials act as manual controls for applying different embeddings to this baseline, such as “coral”, “moss”, “fire”, “ice”, “sand”, “rusty steel” and “cookie”.

By adjusting the dials, those embeddings are applied to the base image in varying strengths. The results are generated on the fly and are pretty neat to see, especially since there is no appreciable amount of processing time required.

The MIDI controller is integrated with the help of lunar_tools, a software toolkit on GitHub to facilitate creating interactive exhibits. 

That looks like a lot of fun! I am impressed by how fast the image is generated and altered as the MIDI parameters change. I wonder if we can use the MIDI controls/commands in existing software like Ableton Live to impact the image? Some of the MIDI effects tools, such as 'arpeggiator, inverse, reverse, humanise, etc. might create new unimagined possibilities. 

I would certainly like to have a go at doing that.