Unlocking Fun Experiments: Exploring the Web Speech API

Hello, JS Nation! I'm so happy to be here with you today. I'm going to show you how I've created my own gamified karaoke in a browser using Web Speech API. So allow me to introduce myself.

Hello, my name is Anna and I'm a frontend developer at the agency Hattar. I spend the majority of my free time on my blog, and I'm quite keen on the IndieWeb community. In fact, in the past, I've written a case for it on Smash Magazine. So please have a look and if you're in the chat, please send me your personal websites and blogs. I would love to read them.

Right, before we get started, I must mention one thing in particular. I do not represent any browser vendor. And for that, I haven't been to karaoke for years now. And I actually thought I would get more invitations after giving this talk a few times, but it hasn't happened. And also, I'm actually a big fan of the Rasmus. I'm not a stan, I'm just a fan. And the reason why I bring this up is that kind of the reason why this talk even popped into my head. I had a problem I needed to fix.

So whenever I go karaoke, there's only one song from Rasmus available. It's always in the chat in the shadows. And it's just not my favorite. So pop, for sure. But I needed more. So I start thinking, okay, what if we had more than just lyrics on the screen? What if we could gamify the experience? Because I know I would win. I know all the lyrics from my heart. So I set myself an adventure to create my perfect karaoke experience. And spoiler alert, it didn't go as well as I imagined. So let's see why. I searched for speech to text, and all the results that came up were all from private companies that offer this type of API as a service for a cost. But we know that there's a browser native speech recognition thing, and it's free. So I thought, no, no, I'm not spending money on this. So let's talk about Web Speech API, in particular, the speech recognition.

So the Web Speech API split into two, the speech recognition and the speech to text, and it's starting to be obvious how I'm going to be using the speech recognition here. But actually, one of the core ideas straight was built for karaoke, but it was to enable developers to use the speech recognition as an input for forms, continuous dictation, and control. In fact, there's quite an old draft that details how this API should work on input field. It's really old, not everything was implemented, but it's still a very interesting read.

In fact, you can see this vision live in a tool that many of us are familiar with. In Chrome, when using the Google Translate, you are able to use the microphone function to speak into an input field and see it translated into text. I thought, great, the code works in my head, so let's see what's the browser support. I was quite optimistic. I went to MDN, and it looks like it currently isn't supported in Firefox, and any other browser still requires a vendor prefix or a different name. And I thought, all right, one browser out, it's not the worst, let's go. Wrong.

Actually, this is a more realistic outcome, in my experience. And it turns out the explanation is fairly simple. This quote from the MDN page says, on some browsers like Chrome, using speech recognition on a webpage involves a server-based recognition engine. Your audio is sent to a web service for recognition processing, so it won't work offline. Right. Thinking about it, it makes sense. It needs access to a lot of data to train on. So browser vendors that belong to massive corporations have an easier time with all the necessary infrastructure, as well as that data to train on.

And this is an important note, browsers that use Chrome don't necessarily have access to the Google transcription service. It's a paid service from Google, which Chrome gets to use for free. Again, yeah, it does make sense. And when trying it on Safari, I was presented with a popover that said speech data from this app will be sent to Apple to process your requests. This will also help Apple improve its speech recognition technology. All right. At this point, I'm down to two browsers only. But I was in too deep, so I'm not going to give up. But I was curious, what's happening in Firefox? Unfortunately, I wasn't able to be in touch with someone who is currently working on this, but I was pointed to an enlightening thread that talks about their concerns about privacy and implementation. So it's a really interesting thread. I really recommend having a look at it, especially if you're curious to see why some browsers make the decisions they do.

And I was playing things out. There were other quirks that are quite important, turns out. You can't feed streams into this API, microphone only. Safari on mobile won't allow the speech recognition API once it is installed as a PWA. Also, on mobile phones, they may emit a sound of like a notification sound of when the microphone is being turned on and off, and you'll see why this is important.

Oh, dear. But let's see it in action. Let's work with what we have. And one of the easiest ways to actually see something like this in action is to look for a bookmarklet. So I've decided to find things done by other people, and I've actually found a really interesting one called Select and Listen. And the way this works is that you visit a webpage, whatever it is, select the text you want it to read out to you. For example, I'm going to the JS Nation website. I will select this text over here. And I'll click my bookmarklet Select and Listen. And it should do something.

Ana works as a front-end developer for the agency Hector. She started coding as a teenager building fan sites and has been working as a front-end developer for the last 11 years. Right. So this is the speech test. It just reads things. And this person, when they created it, they have different voices and languages as well. So it's really, really cool. I really enjoy that. So I've decided to build a little demo, just using a couple of words to see the speech recognition work. I was like, let me see if it detects just a few things. So I've built a demo. OK, let's pretend that you are in the kitchen. You want to cook something. Your hands are dirty and you want to touch your thing to check the recipe or instructions and going back and forth. So I built a little silly demo where I activate my voice navigation and I ask for the page to scroll based on what I say.

Let's give this a go. No hands, look. OK. Scroll to the ingredients. Scroll up. Scroll up. Scroll down. Back to the instructions, please. You have to say please. Scroll up. Scroll up. All right. Great. Great. I've got a demo. I just want to show off my cat. I've got a demo. It works. It picks up what I'm saying and it does those little instructions like scroll to a specific part of the page or scroll up and down.

Great. I think we're ready to build my karaoke then. Let's start simple. The way I envision this is that it will have a start and a stop button. We'll show our transcription if I want to just to double check things are working as expected. We'll load our audio, we'll load our lyrics and we'll load our JavaScript. We'll do some base styling with gradients and it looks something like this. Right. The good thing here is that you don't have to have mad design skills because anything you do will look better than your local bar. Trust me. We'll initiate our speech recognition.

We'll ask to show the interim results. Nothing too fancy. Great English, great British as a language. And append it to our doc. It will pick up what we're saying, put it on hidden div in this case and store it and that should be it.

So, this is the fun part. The speech recognition actually stops after a while. And it makes sense because you will want it to constantly run and consume resources if nothing is actually happening. Well, they don't know that, but it could make sense. So, what people usually do is that they add an event listener that when the recognition ends, we can make it start again. And it's a bit of a cheat, but do you remember when I said that the final quirk was that on mobiles it may emit sounds when the microphone is being on and off? This is what I mean. So, on mobile phones, it may become a bit of an annoying experience because the microphone is turning itself off automatically and we are forcing it to turn it back on. So, you may hear notification sounds of the microphone being turned on and off. It's a quirk to think about, but it is what it is. It's free, we're getting what we pay for.

Obviously, ideally, I would have used a song from the Rasmus, but copyright doesn't let me. So, let's go for nursery rhyme. We'll add our lyrics in a similar fashion of how subtitles in films work. You add the seconds the lines appear and disappear, and you add your line in particular. So, I picked twinkle twinkle little star because it's quite short. And what is the logic? Right. So, when the speech recognition is initiated, the song will also start. We'll check if the current time of the song falls between the start and the end of the expected lyric line. If so, let's add a class showing that it is a current lyric line. This is the logic. It translates roughly into this. Start speech recognition, start song, check for current time, add the CSS class to show that the lyric is currently in progress, and that should be it.

This is the logic. It translates roughly into this. Start speech recognition, start song, check for current time, add the CSS class to show that the lyric is currently in progress, and that should be it.

Same logic the other way around. If the current time is bigger than the end time of a lyric line, let's add a class that marks it as the past. Once that line is in the past, let's do some checks. It's not very fancy. Actually, after looking through our transcriptions, we mostly need to check if what you said, and actually what the computer captured matches word by word. And this actually includes doing a few knowing if-elses. Everyone said a lot of fancy include, but that said, everything else is just basically adding and removing classes.

Right. Let's give it a go. First, I'm going to read out the lyrics. Then, I'm going to read out the lyrics with slightly incorrect words to see if my error matching works. And then unfortunately, I'm going to sing, and I am so sorry in advance. I'm so sorry. Let's give this a go. Let's have a look at our demo. I love it. Bias. I've added the start and stop button. I also added the switch thing button, which is something that I actually thought was quite fun to do.

Right. Let's give this a go. I'm going to read it out first. Twinkle, twinkle, little star. How I wonder what you are. Up above the world so high. Like a diamond in the sky. Twinkle, twinkle, little star.

How I wonder what you are. Yay! Okay. I've changed the lyrics a little bit. So, let's see if this matches up. Twinkle, twinkle, little coat. How I wonder if you will load. Up above the browser high. Like a bug in the sky. Twinkle, twinkle, little star. How I wonder what you are. Yay! Right. So, it's not too bad. Like, it picked up. It marked high as being in the correct order, and it didn't give me any positive colors because I didn't say diamond and it got bug instead.

Let's try. Twinkle, twinkle, little star. How I wonder what you are. Up above the world so high. Like a diamond in the sky. Twinkle, twinkle, little star. How I wonder what you are. Up above the world so high. There we go. Right. Modeling mortifying, but let's forget about that. The yellow animation, the run text got you. So, what's the verdict? There's potential, but I don't think we're there yet. But I had fun. And look, there's a lot of fun demos and projects built around this API. Tony Edwards did a fantastic talk called Beat Rhymes and Edit Tests, and in his talk he wanted to see if the Web Speech API could help him jot down the rhymes he comes up with.

Similar to me, Tony noted that it wasn't perfect, but it was a great experiment, and I really recommend watching this talk. If you want to learn more, Stephanie Eccles built a fun set of chat for the 12 Days of Web Dev Challenge, and Wes Bowles, who was also speaking at JS Nation, has a chapter on the Web Speech API on his free JavaScript course. And there's plenty of other projects using this technology.

So, during the pandemic, the web captioner became a tool for many, many people to use at online meetups and conferences. It's been sunset, so the source code is all available on GitHub, so check it out. And there's also quite a few polyfills available, implementations with WebRTC, and as mentioned before, lots of private companies that offer you speech recognition as a service.

So what's next? I don't know. But I truly believe that it's events like these where these browser APIs can grow, if we're too interested in them. In the end of the day, voice interfaces aren't going anywhere. Meanwhile, I will share and support the current call to action to the speech recognition sibling. You should look up Leone Watson's amazing talk, Designing Voice Interfaces, where a call to action is presented to support CSS speech. I won't spoil this talk to you, so you really definitely should check it out. And there's also projects like the Common Voice from Mozilla, where you can donate your voice, and most importantly, at least from your accent, and also volunteer to do some quality control.

Let's be realistic. Are you going to use Web Speech API at work? Probably not. So you maybe think, okay, cool, whatever. How is this useful for my everyday job? Even for accessibility, the chances are you'll probably be asked to add one of those cursed inaccessible toolbars instead. But if you are, focus on voice interface design. And here are some things to consider. Avoid proposed content. Consider retention and speed to destination. Audit your content for how conversational and spoken it sounds like. Like, FAQs are a fantastic example of good voice-friendly content that is direct to the question and answer. Audit for how your content sounds in isolation. Things like click here and read more aren't usual culprits of not meaning anything when heard in isolation. Test how synthesized speech sounds like in different devices. So be on your laptop and your mobile, consider which prompts direct the conversation, and another good example is when you use your phone and you call a service and you are given multiple choices and it's so hard to memorize just by listening to the options. I didn't cover this in my demo, but consider error recovery. No speech timeouts, not understanding, and no match errors. Because like myself, not everybody's going to have perfect native English, which probably the data is trained on.

There's fantastic books about this topic. I really recommend having a look. And then again, why this? Look at what we did. We did a few working demos. I ended up spending a lot of time reading and experimenting with the browser API in this case, the live speech API, HTML media elements. It doesn't look like it, but there was a lot of JavaScript involved in this.

I ended up having content for a talk, CSS animations, and I just had a great time. And my goal with this talk was that you can learn by building useless things. And I only recently gave myself permission to build unproductive things that are not for my job. And in the past, I would feel guilty if I didn't do productive things after work. Side projects don't need to be monetized in order to be valid. They don't need to become NPM packages or open source projects. And if you ever think that your idea isn't good enough, ignore that thought. Boring is fine. Create your own safe space. And I had so much fun building this, and I felt so satisfied after I had it working.

So, thank you.