The ultimately customisable and expressive music controller

So a while back I had this idea.... from watching Dream Theatre's "score" dvd and in particular Jordan Rudess playing the intro to Octavarium.

Its there that I first saw what's now become known to me as a Haken continuum controller.

It was a marvelously emotive and inspiring musical performance and I too coveted such an expressive controller.

I doubt that I'll ever be able to afford one AND the Roland V-Synth though.

So after some email to and from Lippold Haken about vertical scaling, (he was most kind and helpful, so if you can afford a continuum, buy one from him)and potentially setting up the continuum to play scales vertically like guitar. He very kindly replied saying a continuum is not conducive to such a setup.

That was a bummer for me because I've been musically liberated by guitar, and its from that I cant scale horizontally on any keyboard.

So as usual, I got thinking with my improvisor brain. And then my improvising brain said "hey there are some interesting hacks and controller possibilties using the IR camera in a wii remote together with a programming language.

So the idea was partly born of an improvised need, people like Johhny Chung Lee's foresight about wii remote hardware potential, and my fascination with things like FTIR, the Reactable and multitouch surfaces.

So what we have here is... an endeavour by myself, new to programming and specifically Java programming, to write a program for all platforms, where you like me, can gain the expressive performance capabilties of something like a Continuum from a little DIY, a wii remote and my software.

So what can you do with it? Well here's what I want to be able to do with it, but first an explanation of what you will need to make it work.

1. One or two Wii remotes.
2. A USB bluetooth dongle OR built in bluetooth hardware(unless you wanna go the hardwired I2C route) 3. A bluetooth driver or stack(more on this later).
4. A glass or perspex frame on a stand with an IR LED frame (or an IR LED array with a center hole for the Wii remote to poke through)and at the bottom of the frame an adjustable mount for Wii remotes to see the surface.
5. A glove of sorts that will reflect IR.
6.A rear projection solution if you want visual feedback.

By the way, I'm going the instructables website route of removing an old LCD screen from its casing (because its backlight is broken, and placing it on top of an overhead projector. This I intend to then rear project onto the surface above the wiimote for visual feedback.

I want to be able to use this musically as well as to control and animate things in realtime in Autodesk Maya (via digital puppeteering-again, but more on that later, mostly cos that's also a wiimote/mocap software idea)

So musically here's the list.
1. A multi-touch controller outputting MIDI through virtual ports as well as out from my Edirol USB MIDI interface.

2. OSC output.

3. support for near enough the whole MIDI spec.

4. Through timed pulsing groups of LED's (like embedded in a resin puck and connected/synced via zigbee WIFI, a fiducial recognotion type functionality.

Okayn now consider that by having a four or three LED's embedded in a puck, seen by the wiimote below the glass/perspex surface and pulsing at a set rate with other pucks all in sync via zigbee, you can track the puck as a binary number and more pucks at the same 100 times per second but have (I hope), position AND rotation values for each.

So going the fiducial route one could control Reason like the Reactable but with cheaper processing overheads than a normal camera/rear projection processed feed.

Visual feedback will require building into the program a visual output module, I'll get to that when I have more java experience..

So normally a single Wii remote can track 4 IR points 100 times a second via bluetooth and 200 times a second by hardwired I2C, I'm led to believe.

So if we pulse all the LED's sequentially(which will need syncing probably) hopefully the number of tracking points can be increased to 16 points 25 times a second.

Also by hardwiring 2 adjacently aimed wiimotes and pulsing the IR LED's on the surface above, in a synchronised manner, we can get 32 multitracked points 25 times a second.

And by faster pulsing AND hardwiring wiimotes to the PC via I2C we can hopefully improve on that too.

Well that's loosley what I wanna do and I've made a start. As time goes on and I learn more about Java and things like arduino and I2C, zigbee etc. these listed things will become a reality I hope.

But hey, its MY idea and you read it here on my blog first.

So this will be my free software gift to the world(if you like me, can't afford a continuum). I already have psuedocode written and detailed features listed (for the software design) to offer a functionality like a continuum but it will also allow vertical scaling and guitar-like features.

But hey if it empowers you like it will me consider a donation please.

If anybody is interested and wants to be involved, get in touch. Many hands make light work.

Onward!!
Sent via my BlackBerry from Vodacom - let your email find you!