DEV Community: Nikki Massaro Kauffman

Converting UTF (including emoji) to HTML

Nikki Massaro Kauffman — Fri, 30 Jul 2021 20:09:59 +0000

Sometimes my coworker likes to mention things just to get my mind stuck on them. Take the text from this request:

Because of some limitations both in UTF-8 and mysql (less a concern for us now but still..) it would probably be good to have some kind of simple-emoji type of tag. Similar to how we have simple-icon, a simple-iconcould be used to provide minor tweaks / accounting for emojis in a consistent way.

So last night I worked on translating UTF (including emoji) into their HTML entities.

Basic Unicode to HTML Entity Conversion

I started with started with an adapted version of this conversion logic to convert any character that is not part of the 127 ASCII characters:

utf2Html(str){
  let result = '', 

    //converts unicode decimal value into an HTML entity
    decimal2Html = (num) => `&#${num};`,

    //converts a character into an HTML entity 
    char2Html = (char) => {
      //ASCII character or html entity from character code
      return char.charCodeAt() > 127 ? decimal2Html(char.charCodeAt()) : char;
    };

  //check each character
  [...str].forEach(char=>{
    result += char2Html(char);
  });

  return result;
}

If we want to check this function (quite literally by dropping a UTF-8 checkmark ✓ into the function), its character code 10003 is the same as it's unicode value so it can be used to generate correct HTML entity ✓

The Problem with Emoji Conversion

While the function above works on UTF-8 special characters, it won't work all of the emoji we have available today. I found a really good explanation for in a post called Unicode in Javascript.

Take the 🤯 emoji, for example.

The character code for this emoji is 55357, so the entity returned by the function above would be &#55357;, which does not work.

The unicode value for 🤯 is actually 129327 (or 0001 1111 1001 0010 1111 in binary). In order to express this character as in it's 16-bit form, it is split into a surrogate pair of 16-bit units, in string form as \uD83E\uDD2F (according this handy Surrogate Pair Calculator)--🤯

So in order to get the correct value, we need to know:

if a character is one of these surrogate pair emojis, and
how to calculate a surrogate pair's value.

Determining if an Emoji is a Surrogate Pair

The JavaScript string length for any type of character is 1.
It is the same for characters, symbols and emoji

JavaScript	Result
`'t'.length`	1
`'✓'.length`	1
`'🤯'.length`	1

But if I use the spread operator (...) to get length, I can see that my emoji is made of a surrogate pair.

JavaScript	Result
`[...'t'].length`	1
`[...'✓'].length`	1
`[...'🤯'].length`	2

That means that I can tell which characters are surrogate pairs if [...char].length > 1:

utf2Html(str){
  let result = '', 

    //converts unicode decimal value into an HTML entity
    decimal2Html = (num) => `&#${num};`,

    //converts a character into an HTML entity 
    char2Html = (char) => {
      let item = `${char}`;

      //spread operator can detect emoji surrogate pairs 
      if([...item].length > 1) {
        //TODO calculate a surrogate pair's value
      }

      //ASCII character or html entity from character code
      return char.charCodeAt() > 127 ? decimal2Html(char.charCodeAt()) : char;
    };

  //check each character
  [...str].forEach(char=>{
    result += char2Html(char);
  });

  return result;
}

Notice I left a //TODO comment about calculating the pair. We'll tackle that next...

Calculating a Surrogate Pair's Unicode Value

I couldn't find a good post for converting a surrogate pair to it's unicode value, so instead followed these steps for converting from unicode to surrogate pairs in reverse:

#	Step	🤯 Example
1	Get the value of each part of the pair.	55358 / 56623
2	Convert each value to a binary number.	1101100000111110 / 1101110100101111
3	Take the last 10 digits of each number.	0000111110 / 0100101111
4	Concatenate the two binary numbers a single 20-bit binary number.	00001111100100101111
5	Convert 20-bit number to a decimal number.	63791
6	Add 0x10000 to the new number.	129327

The Completed UTF (Including Emoji) to HTML Function

utf2Html(str){
  let result = '', 
    //converts unicode decimal value into an HTML entity
    decimal2Html = (num) => `&#${num};`,
    //converts a character into an HTML entity 
    char2Html = (char) => {
      let item = `${char}`;

      //spread operator can detect emoji surrogate pairs 
      if([...item].length > 1) {

        //handle and convert utf surrogate pairs
        let concat = '';

        //for each part of the pair
        for(let i = 0; i < 2; i++){

          //get the character code value 
          let dec = char[i].charCodeAt(),
            //convert to binary 
            bin = dec.toString(2),
            //take the last 10 bits
            last10 = bin.slice(-10);
            //concatenate into 20 bit binary
            concat = concat + last10,
            //add 0x10000 to get unicode value
            unicode = parseInt(concat,2) + 0x10000;
        }

        //html entity from unicode value
        return decimal2Html(unicode); 
      }

      //ASCII character or html entity from character code
      return char.charCodeAt() > 127 ? decimal2Html(char.charCodeAt()) : char;
    };

  //check each character
  [...str].forEach(char=>{
    result += char2Html(char);
  });

  return result;
}

Update

Thanks to a comment by LUKE知る, I have an even simpler way to do this:

export function utf2Html(str) {
  return [...str].map((char) => char.codePointAt() > 127 ? `&#${char.codePointAt()};` : char).join('');
}

Using Unbundled Web Components for Dynamic Imports

Nikki Massaro Kauffman — Fri, 23 Oct 2020 13:53:18 +0000

If you've been following along with this series of baby steps, you have already tried using a custom element with our CDN's "magic script", but now you have found other web components you want to use, so you'll need to make your own "magic script" to dynamically import unbundled web components. (See this series on unbundled-webcomponents for more info.)

This post will walk you through collecting your unbundled web components and generating a "magic script".

Set Up a Clone of unbundled-webcomponents

Install yarn.

Clone the unbundled-webcomponents project:

git clone https://github.com/elmsln/unbundled-webcomponents.git

Go to the unbundled-webcomponents directory:
```
cd unbundled-webcomponents
```

Find Your Components on npm

From npm search for the accent-card.
Note that accent-card is listed as @lrnwebcomponents/accent-card
Note that import statement that appears under Usage is import '@lrnwebcomponents/accent-card/accent-card.js'
Repeat steps 1-3 with other the components you wish to use.

Add the Web Components You Want

From your unbundled-webcomponents directory add accent-card to your list of dependencies in unbundled-webcomponents/package.json:
```
yarn add @lrnwebcomponents/accent-card
```
Inside unbundled-webcomponents/dist/app.js file add the Javascript import:
```
import '@lrnwebcomponents/accent-card/accent-card.js'
```
Repeat steps 1-2 with other the components you wish to use.

Building Your "Magic Script" and Curating Your Component Javascript Files

From your unbundled-webcomponents, install your components and dependencies:
```
yarn install
```
From your unbundled-webcomponents, build your dist folder:
```
yarn run build
```
Upload the local-path-to-your/unbundled-webcomponents/dist folder to a server location.

Using Your Very Own "Magic Script"

Add the following script to your page(s):

<script>
window.__appCDN="https://path.to.your/unbundled-webcomponents";
</script>
<script src="https://path.to.your/unbundled-webcomponents/build.js"></script>

Use the web components you selected to your HTML pages, just as you did in my first post:

<accent-card accent-color="red" dark horizontal image-src="https://webcomponents.psu.edu/styleguide/elements/accent-card/demo/images/image1.jpg">
<h1 slot="heading">Look at this card</h1>
<div slot="content">This is the card content.</div>
</accent-card>
``

The more you use web components, the more uses you will find for them. In the not too distant future, I'll put something together on how to make your own.

Finding Web Components

Nikki Massaro Kauffman — Thu, 22 Oct 2020 13:18:43 +0000

If you've been following along with this series of baby steps, you have already tried using a custom element. So far you've only used the web components that we have documented at webcomponents.psu.edu.

This post will get you to find other web components.

Finding Web Component Design Systems

If you're interested in finding a design system, the open-wc keeps a list of web component libraries.

From open-wc's list of web component libraries, click on the link for Adobe's Spectrum Web Components.
Use the left menu to find the card component.
Check out the documentation and demo for the card (sp-card).
Use the left menu to look at the rest of this library.
Go back to open-wc's list of web component libraries and explore more.

Browse some the list and try some of the demos.

Searching by Component

If you're searching for a component to solve a specific problem, try searching webcomponents.org.

From webcomponents.org, use the search to find a slider.
From the search results, select paper-slider. You will be taken to a page describing the paper-slider element.
From the paper-slider page, you can click the Demo link to see the slider demo and its HTML.
Search for webcomponents.org for other web component UI elements that you might want to use.

Once you've found the web components you want to use, you'll need to make your own "magic script" to dynamically import the Javascript files for your collection. Learn how in my next post.

Using a Custom Element

Nikki Massaro Kauffman — Wed, 21 Oct 2020 20:47:12 +0000

Recently I gave a lightning talk for the HighEdWeb Associations's annual conference called "Old School HTML with Modern Web Components" making the case for web components.

Since then I've been getting lots of follow-up questions about how to get started. I think what scares most people off of them is that we often start with how to make your own, which is honestly doesn't give you the immediate return on your learning investment. So what I've decided to do a series that starts at a much easier and more immediately rewarding point.

For newcomers dependencies, bundling, and polyfills are a bit intimidating, so for now, let's use a script where all of that has been taken care of for you. To that end, let's with the simplest most useful task: using an existing web component with a single script.

Add the Javascript. We have a CDN for the web components we use. Add the following script to any HTML page:

<script> 
window.__appCDN="https://cdn.webcomponents.psu.edu/cdn/"; </script>
<script src="https://cdn.webcomponents.psu.edu/cdn/build.js"></script>

Use the custom element HTML. For example, try accent-card:

<accent-card accent-color="red" dark horizontal image-src="https://webcomponents.psu.edu/styleguide/elements/accent-card/demo/images/image1.jpg">
<h1 slot="heading">Look at this card</h1>
<div slot="content">This is the card content.</div>
</accent-card>

Make some changes. Check out the accent-card demo to learn how to change the card's accent-color, orientation, heading, content, and more.
Try more custom elements. Browse our CDN to try more:

Next you'll learn how to make your own magic script for dynamically importing web components.