Emoji Encoding, � Unicode, & Internationalization

Hi, thanks for that great introduction. I'm Naomi Meyer, and I work as a software development engineer at Adobe, where I do localization and internationalization engineering for creative products like Adobe Fonts and Adobe Portfolio. So here's where you can find me online, and if there's anything that you feel passionate about or have strong opinions on, please let me know. I'd love to continue this conversation online.

So I'm sure most of us have seen this many times in the head tag of our HTML markup, and we all know to add this meta charset UTF-8 line. But I've been fascinated lately about the underlying details of what this UTF-8 thing really means and does. So I'm excited to share some more details about what it is and why I think it's so cool. Also, this connects with this JavaScript funniness that we can see here with these seemingly character emojis, when as strings in JavaScript have a length a lot longer than expected. And feel free to try this yourself in your dev tools. I know I wanted to test it out first when I first saw something like this. And part of my goal today is to talk about why this is, why you know familyEmoji.length is equal to 11 why that's true. And provide some more details about the underlying encoding happening here and what we can do to handle it correctly.

So speaking of emojis, I personally find them both delightful and intriguing from an engineering perspective, a linguistic perspective, a creative design perspective, a cultural sociological international perspective, and so much more. Our agenda for today is to start with a bit of encoding history to understand more of where we came from. Then we'll get into the Unicode Consortium and the UTF-8 algorithm. Then we'll talk about how this allows us to encode emojis and different languages across platforms, devices and operating systems. Overall, I think it's so important for us to keep these big ideas in mind when we build software products that are being used globally. This is kind of a timeline, a broad timeline, of what we'll touch on today. We've got a lot to cover in these 20 minutes. Let's jump right in with encoding to start.

Of course, when we're on our computers and we type an emoji, a letter, a character in any language, these are ultimately interpreted by the machine as zeros and ones. Let's get into a bit about how that works. In order to understand how it's working today, let's go back in the past to the 1960s of the first encodings. Back in the 60s, there were these big computers that filled a whole room. This is a picture of one from NASA. Engineers back then came up with a system. That system is called ASCII, the American Standard Code for Information Interchange. This image is from the first version that was published in 1963. ASCII was developed from telegraph code. It was originally built for more convenient sorting of lists, alphabetically by ascending, descending characters.

In 1963, ASCII was encoded in 128 English-only characters into 7-bit integers. So, I think ASCII is pretty cool because it makes sense on how it was built.

So first, we take a character, like the letter A, and we assign an ASCII decimal number, 65, which in binary is equal to 1, five zeros, one, in the original 7-bit system. And then after 65 goes 66, which is B, and we continue alphabetically, all the way to Z, which is ASCII decimal 90. Then to go from uppercase to lowercase characters, we only change one bit, 32 letter, 32 later, which is ASCII decimal number 97. And that continues alphabetically. So I think it's a cool system. And shout out to Tom Scott on this video that he has to explain it really clearly.

So ASCII makes sense, but then, and it kind of worked in America in the 60s. But over time, there were lots of bugs. And those bugs came because non-English characters, like those pictured here, include diacritics and additional accents that were added. So ASCII was originally worked with seven bits. But then computers moved to eight bits, and we went from 128 characters to 256 characters. And different countries and different language systems added more characters with those extra 128. And different languages like Japanese, for example, did its entire own thing. They had a separate multi-byte encoding system, you know, and Japanese, Russian, all these languages had a different encoding system. And that was fine when they worked independently. But then came the internet. And with the worldwide web, the internet kind of broke computers because there was this problem with no universal encoding system. When two different non-compatible encoding systems encountered one another, we got these types of errors like we see here with a lot of question marks and a lot of bugs.

So, in 1991, with the worldwide web, we got Unicode version 1. And Unicode was designed to be a universal encoding standard to solve this problem of conflicting character encodings. So, Unicode Today is a non-profit whose mission is that everyone in the world should be able to use their own language on phones and computers. Unicode version 1.0 was published in October 1991 and had 7,161 characters. To understand and think about Unicode, you kind of have to make a mental shift of your assumptions about language and characters. So, there's three kind of big ideas to think about. Props to Dimitri Pavloutin who has this great article I recommend called What Every Javascript Developer Should Know About Unicode. So, the first idea to keep in mind is abstract characters. So, instead of thinking about letters in an alphabet, it's good to think about abstract characters and Unicode deals with characters in these abstract terms. Second, we have code points.

A code point is a number assigned to a single character. So, this is an example of a code point where we have U plus 0041, an atomic unit of information. So, in a code point, we have 0041, which is a hexadecimal number, and then we have the prefix of U plus, where U equals Unicode, because each code point number is given a meaning by the Unicode standard, and Unicode character sets map each abstract character in the world to a unique number.

So, U plus 0041, we look it up in Unicode, we get Latin capital letter A. Currently, the Unicode standard defines over a million code points. And they all have a one-to-one mapping, so that ensures that there's no collision between alphabets of different languages. The third point to keep in mind is a plane. So, basically, long story short, Unicode divides over a million code points into 17 planes or groupings. These planes are represented here.

So, the first plane, plane 0, is the basic multilingual plane, also known as the BMP. And that's the unification of all the prior character sets. So, that includes ASCII, and Chinese, Japanese, and Korean characters. And this is what the BMP looks like. And I think it's fascinating to see the breakdown of different scripts included. You know, you can see East Asian scripts and the South, the Chinese, Japanese, Korean characters include a lot of additional code points. So, the BMP is four hexadecimal digits. And then, outside the BMP, BMP plane one is five hexadecimal digits. And plane 16 is six hexadecimal digits. And outside of the BMP is the astral plane, or the supplementary planes.

So, how does this relate to our head tag with UTF-8? That we've all, you know, are used to seeing. So, we know about code points. Abstract characters, like U plus 0041 is A. And we know about code units, or physical bits, because, you know, computers, at the memory level, don't use code points or abstract characters. They need a physical way to represent Unicode code points. So, computers translate from Unicode code points into physical bits using a character encoding translation, which takes the transformation from code point into physical bits. And Unicode has this popular open source character encoding translation algorithm called the Unicode Transformation Format or UTF, which does this job for us. And popular encodings of UTF are UTF-8, UTF-16, and UTF-32. So, UTF is really cool. It's reversible, so the conversions between all of them are algorithmically based. That's hard to say.

They're fast and support lossless round-tripping. UTF-8 is most common on the web. UTF-16 is used by Java and Windows. And both UTF-8 and UTF-32 are used by Linux and various Unix systems.

But JavaScript is weird. So, there's these two great articles I highly recommend from Matias Banards that go more into depth of JavaScript and Unicode. But basically, long story short, the ECMAScript engine itself exposes characters according to UCS2, not UTF-16. So, JavaScript engine encoding can have these strange negative effects when JavaScript works with Unicode.

And this is really important to keep in mind because, you know, as developers, we do a lot of string manipulation. We do sorting, filtering, searching, lookups. A lot of business logic involved with strings. So, it's great to understand how these strings are encoded to reduce bugs. This side looks weird. Let's move to the next one.

So, something that's a good example of how JavaScript is kind of strange with strings is the example of a combining mark. So, in Danish, you have the letter A with a little circle over it that is, in Unicode, two separate characters. Where the circle over it is a combining mark that is designed to modify the preceding character. So, if we console log these two Unicode code points, we get the combining mark A. This is one example with a combining mark.

Another example we have here is cafe with the accented E. And this one is different because if we define, you know, this variable as drink with cafe with the E, we can see that drink.length, while it looks like it should be four characters, actually comes out in JavaScript as five characters. And then if we split this string into an array, we can see that the final element of the array is that final accent. And so a lot of confusion can arise when we kind of ignore the concept of code unit sequences. So this is really important to get to keep in mind to avoid string bugs like we see here in cafe with an accent.

So there's a great solution that we can use and that's string.prototype.normalize, which is a method that returns the Unicode normalization form of the string. And there's the link to the MDN docs about it. I think it's fascinating. So if we take that same example with cafe with an accent and we normalize it, we see that the length is now four. And if we split it into an array, we don't get that accent as the final element. So this is a great tool to avoid localization bugs and to really think deeply about how we're building our apps so that they can be internationalized.

So remember that timeline. There's another important element in the timeline, and that's in 2010 on Unicode version 6.0, Unicode added support for emoji.

So let's move on to talk about emoji. So remember the Unicode planes. Outside of the BMP is the astral plane, and the astral plane or the supplementary plane is greater than four hexadecimal characters.

So because all emoji are outside of the BMP, they're in the astral plane, all emoji are five hexadecimal digits which require 21 bits to save in memory. But UTF-16 requires two code units of 16 bits in order to handle that. So with this example of the smiley face, which is the Unicode code point of U plus one f600, we can see that we require two code units. So they're split into surrogate pairs where we have a high surrogate code unit and a low surrogate code unit. And this is the math to get the surrogate pair where we input the astral code point of the smiley face and we return the two surrogate pairs of high and low. So because of this, if we take smiley face and split it we can see that there's two characters there that are garbled. And smiley face as a string.length is two, which is not what you would think. So, and this is true also for longer emoji.

Let me just quickly share a fun example. If we open up the console and you know you can do this in your browser if you want. Let me close all these warnings. So, if we do simple family, right? This is family. And on Mac you can do control command space to open up your emoji. We can see that family.length is 11. And if we try family.split, there's those 11 characters split into an array. And then if we try a different one, let's do like singer with darkskin.length, we see seven. And if we do singer.split, again, we see seven. And you can try this with lots of emoji, and the lengths are surprising, which is fun. So let me pull up that slide again. So the question is, why are these seven? Why are they 11? And the answer is multifaceted. But one of the… Let me close the inspector here. There we go. Okay. So one of them is the zero-width joiner. So the zero-width joiner is not an emoji itself, but it's used in emoji sequences as glue to attach different emoji code points together and create compound emoji sequences.

Another important emoji to keep in mind is the variation selector, which takes a plain text representation of emoji like this heart that we see here and stylizes it to display plain text characters with their colorful emoji representation. So we have a styled heart, but when we take the heart emoji plus the variation selector, which is U plus F E O F to create the styled heart.

Another emoji variant is the emoji modifier. And this is the skin tone modifier, and it's based on Fitzpatrick scale, which is actually used in dermatology about UV light exposure differences. So it's not really a race or ethnicity scale here. It's a dermatology scale. And this skin tone modifier was added in Unicode 8.0. So if we again, open up that singer example, we can see that the combination of the singer is we take woman emoji, which again, with the high and low surrogate pairs is a length of two plus skin tone modifier emoji, plus zero width joiner emoji, plus microphone emoji. We get the total combination of singer emoji equal to all of those combined, which I just think is so cool and awesome about emoji.

So let's end, you know, with emoji today. So emoji are like fonts and they have variation across platforms. Here we see in the browser in Apple, Google, Facebook, all these different emoji and how they can be represented differently just like fonts in the browser. And again with the hugging face, party face and face the tears of joy emoji, which is one of my favorites. And then I think this is funny because the cell phone emoji or the mobile phone emoji looks like the phone of the provider across different fonts. And the hamburger emoji, this was a big internet controversy because the placement of the cheese varied whether it was on the meat or on the bun. And people were very upset about it. And then the gun emoji, this one actually has sparked a lot of discussion among legal experts on whether or not emoji can be permissible as evidence in a court trial. And the different providers actually changed it from looking like sort of a realistic handgun to instead looking like a toy water gun to try and avoid this controversy.

So with that in mind, I think it's really important to know that anyone can propose a new emoji for the next version of Unicode. You can go on this link and it is a process but it's possible. So for example, these four emoji that we see here were all sort of externally proposed to Unicode. So the blood drop emoji we see here, it's featured in the documentary Beyond the Emoji, which I recommend. And the goal of this one was to break the stigma of menstruation. The woman with the hijab headscarf emoji, there was someone who said, hey, I wear a headscarf. I don't have any emojis that look like me. I want to get one into Unicode. And they made that happen. And Jennifer A. Lee, who's a really incredible emoji leader, she is from Chinese background and she wanted a dumpling that was part of her culture. So she worked hard to get a dumpling included in emoji.

So I've been to the Unicode conference, and while they're doing great work from an engineering the encoding is fascinating. The Unicode consortium, the people who are determining emojis and making changes for future versions of Unicode, it's pretty male and pale. So I think it's really important to have more transparency about this process so that we can get more emoji out there and encode them into bits because it's not just a matter of having kind of the right icon to describe what you ate for lunch. It's about digital acknowledgment of culture, about who gets to be represented in our future digital language. How do those representations take into account different things like ethnicity, religion, and more? I highly encourage people to learn more about Unicode, learn more about how they can propose a new emoji, and to get involved in the guts of the encoding process.

That's all I've got. Thank you so much.

Awesome, thanks so much. Hey, how's it going? Hi, doing well. Good, good, good. As I was saying to some of my co-chefs what an informative presentation that was. I'm looking forward to having a chat with my friends tomorrow and randomly bringing up a question as to why the family emoji is 11 characters long. You're just going to be like, oh Chris, you're so smart. I'm like, yep, I am. Nice.

But yeah, we do have quite a number of questions from the audience that I'll get into. So first, the first question is from Nix. Nix wants to know, so because some emojis is like a compound, can you replace elements of it to produce something different? Oh, that's a really fun question. You know what? I would love to try that. I don't see why not. It makes sense. Explore, give it a try. I think it would have to be a valid, it would need a design in the font face that you've selected. So if you're on a Mac, you know, there would need to be an Apple design for that emoji. And if there is, then you can probably change them using the different Unicode characters if you want. Instead of using the emoji picker. Cool.

And then this question was from Rin and Amir. They wanted to know what is your favorite emoji and why? I love emoji because there's so many different ways that you can use them. And I often will use a wide variety. And I try and use the least, the less popular emojis. But I think my all time favorite is probably the the face with tears, the joy face with tears, just because it's happy and fun. Do you have one, Chris? Oh man, mine is usually, the best one is just this. You know, the sign just saying like, OK, that's mine. That's mine.

Because pretty much it's always a long thread and I just can't get through it. And I'm just like, all right, cool. Yeah, right. Get right back at you. Yeah, I like the thumbs up one too. Exactly. Exactly.

People understand, like, I'm not busy, I'm just too lazy to respond.

We have a question from Cynthia and the question is, so if someone is building a Twitter like application and then we were limiting the characters in the message using string length to calculate character amount, calculate character count, how could you make sure that an emoji counts as one character instead of sometimes seven? Yes, so this is really interesting. I was thinking about doing a Twitter demo because Twitter counts each individual emoji as two characters. So you can try it out yourself, you know, in Twitter you can have a string of all As and then if you add an emoji at the end, you'll see that one emoji is two characters. So that's something to keep in mind. And another thing I find fascinating about Twitter emojis is that certain hashtags, you know, include an emoji. So for example, I've been using that, you know, hashtag Black Lives Matter a lot lately. And at the end, Twitter includes the sort of fist emoji. And so I don't know the exact number of characters, but you'd think it would just be the ABC characters and not the emoji when you're counting for character count. But the emoji is included as two characters in the hashtags that have emojis, if that makes sense.

And then a question I also had as well, because I was quite fascinated by your talk, is I wanted to find out what role a diversity of culture and traditions play in the design of emojis. A big role. That's something that I think is really important with emoji, is to try and keep them representative of the actual world of people who are using them. You know, if these little tiny pictures that exist on every phone, every computer, all across the world, all across different languages, I think to be more accessible, they should represent the actual people using them. And that's why, you know, using different gender modifiers or different skin tone modifiers is really important to include and to handle that in coding, because then we can, you know, make emojis as diverse as the people using the emojis. And I think it's really useful for everyone to learn more about Unicode and the sort of quiet consortium that's working behind the scenes to add new emoji, because things like, you know, the women with the hijab or the rickshaw or the dumpling emoji, these are representative of different cultures that maybe the original Unicode consortium didn't include, and I think it's really important to get more of those out there. For sure, for sure.

And then I have a question from Drum. And Drum wants to know, so can we use UTF-16 or UTF-32 in the web? Yeah, that's a good question. So it's recommended to use UTF-8 online, on the web. And it's important to remember that the web is encoding in UTF-8. So if you have a database or a backend process that's using UTF-32 or might be using ASCII or a different encoding altogether, to keep that in mind. Because when a string gets encoded and decoded, there could be conflicts if they're different systems.

So the benefit of UTF-8, UTF, is that it works if you're using UTF-32 or 16 or 8. But if you're using a different system, like we talked about, JavaScript under the hood has some UCS-2, that's where the problem arises. So always know at each point in your full app architecture what the encoding and decoding is so that you can remove any of those bugs.

Yeah, for sure. And I feel like this talk, just based off of, one, the fact that emojis are something that we all utilize, and two, just the fact that when they all represent us as individuals quite accurately, a lot of them as well. I feel like this talk is one that could go on and on and on and on, even beyond.

But, guys, Naomi is not going, she's simply going to be moved over to her own Zoom room. So, there are so many questions still piling in that we just don't have time to carry on, to finish with here. But, yeah, I guess for myself I just had one final question and I promise this would be, this is only food-related because we are doing a pseudo chef show but I guess myself and the other co-chefs wanted to find out what is your comfort food choice.

That's a great question, and my family is Jewish and we have matzo ball soup, which is like the Jewish chicken noodle soup, and that's definitely my comfort food, and I was thinking that could actually be another emoji, a matzo ball soup emoji. There you go, there you go. You know, ideas matter, ideas matter, now you know, now you know. But Naomi, thank you very much for an informative presentation. You've definitely, you're definitely going to make me, you know, I'm going to be much smarter in front of my friends because of it, for at least the next week or so, because I'm just going to be relaying everything you just mentioned, and then some. And yeah, please, she will be in, she'll be going to her Zoom room, so feel free to keep the question coming, you know, peeps. And yeah, Naomi, till we speak again. It's been a pleasure. Great, thank you so much, Chris. Thank you.