MIDI in the Browser... Let's Rock the Web!

Thanks for being here so late after this long, long day in this very, very hot room, but it's going to continue to be hot. I'm saluting people back at home also. I'm glad you're here while over there, so that's nice.

Yeah, my name is Jean-Philippe Cote and I'm here to talk to you about MIDI on the web. Now, okay, because I don't want to ruin this. I have sound, right? Given this is not a music conference some of you may wonder what MIDI actually is or not, I don't know. But to many people, I'm afraid, MIDI is this outdated technology from the 80s. It's this like... It's the reason for awful sounding track ripoffs like this one. That's awesome. You probably all heard things like this before, and just so we're clear, this talk is not going to be about those stupid MIDI standard MIDI files or karaoke files. So let's cross that out right now. You can always keep them for the after party if you want, but this talk is not going to be about that.

The talk is going to be about interacting with MIDI-compatible devices from your web browser. So I'm talking about synthesizers, and drum machines, and wind controllers, and percussions, and even the awesome keytar, my favorite. Even though the acronym stands for Musical Instrument Digital Interface, MIDI does extend way beyond music. The protocol can be used for, to control stage lighting and pyrotechnics, and it can be found in robotics, theme park, ride control, and all sorts of other contraptions and contacts. So, having access to MIDI from the browser really opens up awesome possibilities.

Now, just so we're on the same page, MIDI is a communication protocol. There is no audio involved. No audio is being transferred. The only data that's being transferred is actually just numbers, identifying the notes that are being played or stopped, and how strong they were played. Obviously, the protocol also has messages for pitch bands and program chains and time syncing and so on. But it's a very lightweight and efficient protocol that can be used on devices with really modest processing power. Think Arduinos, for example. You can do MIDI on an Arduino. MIDI has stood the test of time and is embedded in pretty much every single piece of electronic music, hardware and software made in the last 35 years. This is a long way for technology. The core MIDI standard was released in 1983 and it pretty much has remained the same since then. 37 years after that, in 2020, the MIDI 2.0 standard came out and it builds upon the original.

It doesn't replace it. It's just like it adds a bi-directional communication and a bunch of other stuff. But in our case, the year that's really the most interesting, I guess, is 2012. And the reason why is because this is the publication year of the Web MIDI API. At the time, midi.org touted the Web MIDI API as the most significant advancement of MIDI since MIDI itself. Way to talk about themselves. So, there you go. But really, it was quite a big thing.

All of a sudden, all your synths and sound modules and librarians and patch editors and what have you could all be accessed remotely from the browser. Theoretically. Obviously, the spec came out in 2012, but we had to wait a few more years for the first implementation, and this came in 2015 with Chrome 43. As of today, pretty much all browsers support it, except one notable exception, which is Apple, and, as you probably know, Apple has decided to block or not support a bunch of APIs over fingerprinting concerns. But, hey, pretty much all the other ones, as you can see, support it already. So this is roughly 87% of desktop browser traffic, which is really the target, well, the primary target for this API. Obviously, you can do MIDI on portable devices, but I think still the main target is desktop browsers.

So the question is, here, are you going to be amongst this first wave of developers creating awesome new projects with this API? What can we build with it? Well, you can do like Francois Georgi and build yourself a little webpage to edit your reverb pedal settings. Why not? Or you can go a bit further and build this old music education system that's online, considering the pandemic we just went through, this kind of makes sense now. Or perhaps what you want to teach about is music theory. And this is what the Chromatone project does. Or you're just a crazy team of people and you want to build your own jamming, live coding kind of thing? Well, there you go. This has been done already. Obviously, you can also create the, you can also use the WebMedia API to control your own web audio based instruments. So there are already several in such instruments in existence but not that many. And again, the point here is probably that it could be a great time for you guys to jump in and build those instruments, build those tools for the next wave of online MIDI music.

So, what is the experience of a user wanting to try out one of these software instruments? To demonstrate that, I'm going to use this awesome synth which is called simply WebSynths Microtunnel by Mitch Wells. It's one of the first ones. It's actually pretty good. It's a web audio-based so all the sounds you're going to hear are coming from web audio and I'm just going to control it from my little Akai controller. Now, we've had a few issues with the Wi-Fi. So, I think I'm just going to stick to using my own phone here to be on the safe side.

Let's connect to this guy and this should work, hopefully. All right. Crossing my fingers. Come on, come on, come on. Yes.

All right. All right. Okay. So this is this is oh, wait, wrong screen, oh, shit. Hold on. Where are you guys? Here. Okay. Sorry about that. Didn't realize you didn't see nothing.

All right. So this is live. This is as of now. And so basically from the user's perspective, you plug in your device. Basically pretty much all midi devices are midi over USB in 2020. So there's nothing much more than just plugging it in. And you can see on the little midi tab here, if I press a few buttons, yeah, I've got some traffic there. So it actually recognized it, and it's receiving some data. I'm gonna load a patch because you can save and load patches as JSON files. Here we are. Does that work? Yeah. All right. So this is great. And that's it. This is from the user's perspective. Well, say a few applause for that guy.

He's the genius. I'm just showing off here. But from the user's perspective, this is what it means. You go to a website, you plug in your device, and boom, you play. And that's it. And depending on what the developer has done, you can eventually save and share patches, play with other people, and so on and so forth.

This brings me to back to us, actually. From the developer's perspective, what does it look like? So I'm going to go through the, like, the bare minimum example of what would be required from a developer to use the Web MIDI API. Basically what we're going to do is whenever I press a note on the keyboard, I'm going to change something on the page. Let's see the background color. Just something super simple. But just so we can go through the motions.

So first thing is looking for this request MIDI access function. If it's there, we have support for MIDI. If not, well, move on. There is no way we can do anything else. So once we know we have MIDI support, we can actually call this request MIDI access function, and this is when we get the prompt. The user is prompted for authorization. If the user denies access, then there's nothing else we can do. But if he does grant access, obviously, then this method which is in the then clause will get called. And the on MIDI access function in this case receives a MIDI access interface objects, and this object allows you to list available devices and access them. So that's fairly straightforward. In this case, what I'm doing is simply going through all of them. It's in terrible and looking at the device name and seeing if it matches whichever device I want to use. And if it does, then I can assign a function to the on-MIDI message property. And this function will get called each time a new MIDI message is coming in from that device.

So, so far so good, simple stuff. So, let's say I press a note on my keyboard or a pad or electronic drum or whatever instruments you're using, MIDI message comes in. Now, it's a little bit trickier because the thing is, the Web MIDI API doesn't provide any semantics on top of the bare web MIDI, the bare MIDI messages. So, what that means is that what we get in the e.data property is the actual raw MIDI message in uint8 array.

So, I'm just going to do a little pause here and we'll look at a basic MEEDI message. It looks like this, it's usually three bytes. Byte number 2 and number 3 are respectively the note and the velocity between zero and 127. So, that's not too bad.

The first byte is broken down in two parts. The first part is the message type, which can be note on when you play the note, note off when you release it, but it can be a whole bunch of other stuff, program change message, pitch band, signal, whatever. It can be various things. And the second part of like the second 4-bit part of this first byte is the MIDI channel. So there are 16 channels in the MIDI system and that second number would identify that. All right.

So not too bad. I mean, we're diving into binary, but it's still managing. So, what this means for us is that we first need to look at the first byte of that data. And then we binary shift it right by four to get like this first part. And then we have to figure out what that number is. And if it happens to be 9, we're in luck because 9 means note on. How the hell did we know that? Well, you know, you have to read the specs. Oh, Jesus. All right. So, these are from the media.org site. And if you look around there, a hex value of 9, 0, it actually means a note on message on channel 1. Obviously, channels are numbered from 1 through to 16 when we speak but they're actually 0 through to 50. Anyway, at least we got that right. So, we come back and then we check the second byte of data.

And the second byte is as you all remember, yeah, I know. This is stupid. It's the note number. So, what is this 60 there? Well, let's say I want to check if it's middle C. If you're into music, like the middle C on the keyboard, this is C4, depends on the platforms, it's usually C4, and it turns out that it's note number 60. And you would know that not by looking at the MIDI specification, because it doesn't say, you would know that by looking at the MIDI specification and cross-referencing that with the scientific pitch notation.

There you go. Easy stuff, and then you change the background color. So, this was my first experience of using this API. At this stage, I was a bit bummed out. Maybe you are, too. Will we really have to do binary arithmetic and look up spec sheets just to react to a note being played? This kind of sucks.

The answer is, if you stick to the bare WebMIDI API, yes. Like I said earlier, it does not provide any semantics on top of the bare messages. So, to be fair, the WebMIDI API is like many other APIs in that regard. If you look at WebGL, for instance, it's not exactly easy to use. That's why we have libraries like Babylon or 3JS or libraries like that.

So this is basically what prompted me to create my WebMidi.js library. The idea was for web developers but also musicians who happen to also program to be able to use MIDI on the web with some kind of user-friendly interface or API. So, if we look back at the previous example but using the library, this is what it would look like. It's not necessarily that much shorter but I think it makes a little more sense. So first, we import library that's a given. Sorry. And then we call the libraries enable method which does two things, it checks if we have support for MIDI and it asks for authorization. Obviously, same thing as before if a user denies authorization, then we're done. But if a user authorizes us to get access to the MIDI subsystem, then the on enabled function gets called. This on enabled function basically fetches the device we want to use. So, get input by name simply like it says, it just retrieves the input and just specify the name of the device you want to use. And from there, we can add a listener for a note on event. So instead of having to look up which number matches what, we just get a bunch of pre-canned events to work with. And whenever a note on event is triggered, the on-note-on function gets called. And in this case, this function receives an event, and this event has got a whole bunch of information. One property is the note property which has an identifier subproperty, which is simply the note followed by the octave. So if I want C4, I ask for C4 simply. And then we can change the background color. That's in the gist what the library does for us.

Obviously, there's a whole quite a bit more stuff that it does do, but basically the idea is for the library to make it easier for receiving input, and also sending back messages to external devices. As far as outbound messages are concerned, the library has a matching function for pretty much all MIDI sending capabilities. This allows you to send a message to an outbound port without ever having to look at a spec sheet. Believe me, that's no fun. In the other direction, WebMD dispatches events for all commonly used outbound MIDI messages. These events are dispatched either at the device level or at the channel level depending on the types of messages. Obviously, there's no way for us to go through all of that in such a short talk. But we can still manage a small demo.

All right. Let's pray that demo gods are going to help us today. If I can find my cursor which is where? Oh, here we are. And you don't see anything right. Over here. Okay. Got it.

All right. So this is the HTML page that I'm going to be using. As you can see, there are four different modules that are being used. SVG inject which you might know of is simply going to be used because I have an SVG file for that device, and I want to manipulate it in real-time. I want to change the colors of the pads that I'm going to strike. So basically SVG inject just takes the SVG and inserts it into the DOM so we can manipulate it. So that's just a side usage there. TinyTween is I'll explain later, it's just a tweening library. Tone.js which you might have already heard of is this amazing Web audio library, but it's also a way to create instruments like synthesizers, samplers and stuff like that. So if you're looking to make music on the web, this is really one you want to look at. And obviously the last one is my own web meeting. A few styles, nothing fancy. This is our code which we're going to look at in a second. Basically as I said, this is the SVG of my keyboard which will be manipulating it as we play with it. If we look at the code itself, actually let's look at the page. Just show you what it looks like for now. Oh yeah, right. You probably know that we cannot play sounds as long as, well we need a user gesture before a sound is enabled. So that's the reason for that. But this is nothing to do with MIDI, it's just most browsers implement that. So this is the SVG file which we're going to just manipulate later on. All right. This is the code.

So super simple stuff, I think. This is just the listener for the click. When there's a click, we move here, we remove the prompt, we display the SVG, and we remove the listener. So boring shit.

All right. Now, the fun stuff begins there. This is where we use tone as library to create a sampler object. So in musical terms, a sampler is usually a device that when you strike a pad or a key plays some pre-recorded sound. That's just a bit. So in this case, I have eight sounds which are mapped to the eight pads on the device, and besides that, I have the note and octave matching each and every sound. That's quite simple to do. There's a little channelable method there which at the end just sends the audio to the master output because, obviously, we want to hear something.

Tone can be used in a live setup like I'm going to do today but it's used as a playback or sequencer, and, by default, it has a look-ahead time-out which is a bit long when you're trying to play live so I'm putting that to zero so it's a bit more responsive but that's the only reason that's there. And then we are ready to enable web MIDI, and, again, if the user authorizes, we trigger this on MIDI ready function. Now, my device is called an MPK Mini 3, so this is how it's reported by the MIDI subsystem, so if I do get input by name, I just passed that name in. Obviously you can list the inputs and outputs quite easily by logging web MIDI dot inputs or web MIDI dot outputs. It's easy to fetch. So I get the device. I place that in a constant called synth, and then on that device, I'm adding a listener for the note on event just like we saw earlier. This method here will get called every time a note on event is triggered, not every time a MIDI message is received, but only when a note on message is received. In this case, what I'm doing is using the trigger attack of the semper object which means play, play a note, and you need to simply pass it the note that you want to play in parentheses. In this case, I'm using the note identifier we saw earlier. So, basically, all of this pretty much lines with each other.

Now, so this is going to play an actual sound. Next, I'm going to flash the matching pads on the SVG while I'm playing so you get a bit of visual feedback also. And in this case, I just created a little map object there, so I've got eight pads and I'm just saying, well, E2 matches pad one and then F2 matches pad two and so on and so forth. And then I'm stupidly just bringing the opacity to one because it's currently transparent. So, that's all there is to it. And a little tiny tween library there is just because I'm fancy and I want a little fade out for the opacity. So, that's the only reason why that's there.

So, let's check this out. Actually, I'm not going to keep both of them. Let's just keep one. All right. Is this working? Let's do it. There you go. Easy stuff. Easy stuff.

So, that was the gist of it. If you'd like to try out Web MIDI, just go to the website. It's got a quick start section and you'll be able to play in no time. WebMidi.js is a passion project for me. It just happened to become popular, but I'm definitely not doing that for the money. I'm doing it for the fame. So, if you do build awesome stuff with it, just let me know or share it on the GitHub discussions page.

Thank you so much for this thing. Thanks. Thanks a lot. It was a lot of fun to see. I was curious, is it the WebMidi.js pack at all? Is it any commercial interest in it? Or is it mostly for people creating things like this? Which is awesome. Nothing against it. But, is it also some commercial interest in it?

Well, I'm really wishing that the big companies like the Yamahas and the Rolands and all those companies would actually take some form of interest in that. I'm not sure why they're not that interested so far. I mean, they're kind of there but I don't know. We're still at the beginning. That's why I was saying if you guys want to jump into that train and develop these projects, I think there is a lot of room to grow in there. But the majors, I think they're still a bit shy. So they'll need convincing, which I'm trying to do in my own way. Well, then we need to tag them on Twitter when we show them. Yeah, yeah, exactly, exactly.

And I imagine also in the Chiptune scene, there's a lot of interest in this, right? Yeah, yeah, totally. I mean, there's interest from various people. Chiptune is an example, this like a DJ VJ setup and be able to control our thing, but she's a web developer and she just wants to stick with web stuff. And this is the perfect match. So yeah, there's a whole bunch of stuff that can be done with this.

Yeah, and also live visualizations, man. Yeah, totally. Well, you basically did that. Well, you can go on steroids. Yeah, exactly. But really cool demo. So thanks a lot. If anyone wants to continue the conversation about webmedia, then you can go to the speaker's booth. And right now on the main stage, we're going to have a first screening of the new Ofrezen documentary about Svelte. So really excited about that too. One big round of applause for Jean-Philippe.