DEV Community: Martin Vejdarski

UTF-8, quick and easy

Martin Vejdarski — Sun, 17 Jan 2021 23:54:49 +0000

Text is simpler than it looks!

Every developer hits a wall working with text sooner or later, and diving into the complexity of manipulating it correctly can easily scare people away.

Why can't we have nice things?

Unicode greatly simplifies working with text by providing a standard representation, but it isn't used everywhere. UTF-8 is the most popular character encoding scheme, but, you guessed it, it also isn't used everywhere. For historic reasons, UTF-16 remains the default in JavaScript, Java, C#, Windows, Qt, and the ICU project. Visit http://utf8everywhere.org/ to find out more.

UTF-8 vs ASCII, UTF-16, UTF-32

Before jumping into UTF-8, here's a quick comparison with other encoding schemes:

UTF-8 is backwards compatible with ASCII, so everything in ASCII is already in UTF-8
ASCII encodes only 128 characters, which is a tiny fraction of Unicodes' 143,859 in v13.0
UTF-8 uses one to four 8-bit code units, giving it plenty of room for any additions to Unicode
UTF-16 uses one or two 16-bit code units
UTF-32 uses a single 32-bit code unit

Isn't UTF-16 safe in most cases?

No. Consider the two most frequent emoji used in Twitter, ❤️ and 😂. While most characters will be treated as having a length of one, both of these emoji occupy two UTF-16 units. Example in JavaScript:

Can't we just use UTF-32?

UTF-32 is inefficient and increases the space required to store text. Contrary to common expectations, UTF-32 also isn't a silver bullet for manipulating text. While it is fixed-width, it only represents a single Unicode code point, and many characters like emoji, for example, consist of a combination of code points. Example in JavaScript:

Code points ≠ characters

Many characters can be represented with a single code point, however, there are also many characters that span multiple code points.

For example, in Thai there are different tonal and vowel marks, อ อี อี้ อู้, which all consist of separate code points and can be typed and erased separately:

Different emoji also consist of combinations and variations:

If these aren't code points, then what are they? Find out more about grapheme clusters here.

How is UTF-8 stored?

UTF-8 encodes characters into one to four bytes and uses prefix bits to differentiate them. It can encode all characters in the range, U+0000..U+10FFFF (limited by the UTF-16 range).

(x represents code point bits)
0xxx-xxxx                                  1-byte sequence, 7-bit value
110x-xxxx 10xx-xxxx                        2-byte sequence, 11-bit value 
1110-xxxx 10xx-xxxx 10xx-xxxx              3-byte sequence, 16-bit value
1111-0xxx 10xx-xxxx 10xx-xxxx 10xx-xxxx    4-byte sequence, 21-bit value

Converting to UTF-32:

UTF-8                                   |  UTF-32
---------------------------------------------------------------------
0ABC-DEFG                               |  0000-0000 0000-0000 0000-0000 0ABC-DEFG
110A-BCDE 10FG-HIJK                     |  0000-0000 0000-0000 0000-0ABC DEFG-HIJK
1110-ABCD 10EF-GHIJ 10KL-MNOP           |  0000-0000 0000-0000 ABCD-EFGH IJKL-MNOP
1111-0ABC 10DE-FGHI 10JK-LMNO 10PQ-RSTU |  0000-0000 000A-BCDE FGHI-JKLM NOPQ-RSTU

Byte prefixes:

0 - 1-byte sequence
110 - start of 2-byte sequence
1110 - start of 3-byte sequence
11110 - start of 4-byte sequence
10 - UTF-8 continuation byte

It's not very common to have to implement UTF-8 iteration from scratch, given that there are well tested open-source solutions out there. However, it's still a useful exercise to understand how it works. Here's an example of UTF-8 iteration in C++:

constexpr auto UTF8UnitMasks = std::array{
    0b0011'1111, 0b0111'1111, 0b0001'1111, 0b0000'1111, 0b0000'0111};

int getUTF8Prefix(uint8_t c) {
    if (c < 0b1000'0000) return 1; // 1-byte (ASCII)
    else if (c < 0b1100'0000) return 0; // continuation
    else if (c < 0b1110'0000) return 2; // 2-byte
    else if (c < 0b1111'0000) return 3; // 3-byte
    else if (c < 0b1111'1000) return 4; // 4-byte
    else return -1; // invalid
}

// Returns the current code point and increments textBegin to the next one
int32_t nextUTF8(const char** textBegin, size_t size) {
    if (!textBegin || !size) return -1;

    auto& data = *reinterpret_cast<const unsigned char**>(textBegin);
    auto units = getUTF8Prefix(data[0]); // count code point units
    if (units < 1 || units > size) {
        ++data;
        return -1;
    }

    // verify all subsequent units are continuation bytes, getUTF8Prefix(c) == 0
    if (std::any_of(data + 1, data + units, getUTF8Prefix)) {
        ++data;
        return -1;
    }

    auto value = int32_t(data[0]) & UTF8UnitMasks[units];
    for (int i = 1; i < units; ++i) {
        value = (value << 6) + (data[i] & UTF8UnitMasks[0]);
    }

    data += units;

    // check for Unicode range and overlong encoding (e.g, ASCII in 2+ bytes)
    switch (units) {
        case 1: return value;
        case 2: return value >= (1 << 7) ? value : -1;
        case 3: return value >= (1 << 11) ? value : -1;
        case 4: return value >= (1 << 16) && value <= 0x10FFFF ? value : -1;
        default: return -1;
    }
}

void example() {
    auto text = std::string_view("สวัส\xFFดีครับ!"); // Hello in Thai + invalid
    for (auto begin = text.begin(); begin < text.end();) {
        std::cout << nextUTF8(&begin, text.end() - begin) << " ";
    }
    std::cout << std::endl;
    // Output: 3626 3623 3633 3626 -1 3604 3637 3588 3619 3633 3610 33
    //                             ^ 0xFF - invalid code point
}

This post only scratches the surface, but should help demystify some of the foundational bits.

DLL Symbol Visibility in C++

Martin Vejdarski — Fri, 15 May 2020 19:50:13 +0000

Static and Dynamic Libraries

Static libraries are a set of symbols (variables and functions), which get resolved and embedded into an application right after compiling, at link-time.
Dynamic libraries get resolved and linked at load-time, when the application starts, or later on, at run-time (e.g. as a plugin).
Having the symbols embedded reduces the load time, however, using dynamic libraries allows for these libraries to be changed without having to recompile the entire app.

Symbol Visibility

To allow for symbols to be linked dynamically, the dynamic library has to contain information that would map its symbols to the ones needed in the app.
This is achieved through the relocation and symbol tables.
By specifying which symbols are visible/exported (to be used by other libraries) and which ones are hidden (only used internally), the sizes of these tables can be greatly reduced, which results in much faster application load times.

DLLs and linking errors (`LNK2001`, `LNK2019`, and `LNK1120`)

Working with dynamic-link libraries on Windows requires that symbols shared across boundaries be explicitly marked as exported or imported. Failing to do so results in errors, such as error LNK2019: unresolved external symbol ... referenced in .... This typically comes as a surprise at first, especially when coming from a different OS where all symbols are exported by default and limited symbol visibility is an option.

How are symbols exported/imported?

The most common way is by using the keywords __declspec(dllimport) and __declspec(dllexport) to mark every symbol that should be usable by other libraries.

For example:

// foo.h
__declspec(dllexport) void bar();

When compiling the DLL, the compiler must see the __declspec(dllexport) keyword. However, when later on using that DLL in another library or executable, the compiler has to see the __declspec(dllimport) keyword for those same definitions instead.

// foo.h - compiling library A
__declspec(dllexport) void bar();

-------------------------------------------------------------------------------

// foo.h - compiling library B, which links to library A
__declspec(dllimport) void bar();

The easiest way to achieve this is by wrapping these keywords in macros so that they can be flipped conditionally.

// common.h

#if defined(A_IMPLEMENTATION)
#    define A_API __declspec(dllexport)
#else
#    define A_API __declspec(dllimport)
#endif

// foo.h
A_API void bar();

When compiling library A, A_IMPLEMENTATION would be passed as a compiler definition, and when compiling library B, it would be left out.

While this macro is sufficient when only targeting Windows, if the library were to be used on other OSs, the macro would have to be extended to handle other platforms accordingly.

In CMake, there is a GenerateExportHeader utility, which, as the name would suggest, generates all the necessary macros automatically. CMake also automatically defines <libname>_EXPORTS (analogous to A_IMPLEMENTATION) for each library.

Where should the macros be placed?

The answer depends on the symbol type, and while this is not an exhaustive list, it should cover most of the typical use cases:

// free functions declarations - before the return type
A_API void freeFunction();

// class or struct definitions
//   to export the entire class with all of its members, 
//   add the macro after the class keyword
class A_API FullClass {
public:
    void method();
    static void staticFunction();
    int memberVariable;
}
//   to export only some of the members, decorate each member
//   Note: exporting the whole class is more common and less prone to issues
class PartialClass {
public:
    A_API void method();
    A_API static void staticFunction();
    A_API int memberVariable;
}

// extern variables - same as free functions
A_API extern int externVariable;

// inline functions - these will compile with or without an export,
// however, if an inline function hasn't been exported, its address
// will be different depending on which library accesses it
// See References below for more information.
A_API inline void inlineFunction() {}

// template functions without definition in the header:
// each specialization that must be available from the DLL, must be 
// explicitly instantiated and exported
//   foo.h - declaration
template<class T>
void templateFunction(T);
//   foo.cpp - definition and exports
template<class T>
void templateFunction(T) {}
template A_API templateFunction<int>(int);
template A_API templateFunction<double>(double);

// template classes without implementation in the header:
// same as the template functions above - each specialization must be exported
//   foo.h
template<class T>
class TemplateClass {
public:
    TemplateClass();
}
//    foo.cpp
template<class T>
TemplateClass::TemplateClass() = default;
template class A_API TemplateClass<int>;
template class A_API TemplateClass<double>;

References:

Prettifying Debug Variables in C++ with LLDB

Martin Vejdarski — Thu, 14 May 2020 06:22:00 +0000

Prerequisites

This post is a quick introduction to variable formatting in LLDB, and so it assumes some familiarity with LLDB and C++ debugging.
To learn more about LLDB, please visit either of the following:

What is variable formatting?

LLDB has ways of quickly representing the values of variables we see when debugging either through an IDE or the command line.

For example, this is how debug variables and their values look like in CLion:

Why do we need to customize them?

While the defaults work perfectly well in the majority of cases, there are times when having a custom summary can significantly enhance readability and save time.

Consider the following example with color:

using Color = uint32_t;
Color color = 0xFF880022; // ARGB

Without any formatters, the value would appear as an integer:

We can format Color types as hex by running the following command in an LLDB command-line session:

type format add -f hex Color

The value is now more readable:

The ARGB channels can also be displayed via Python scripting (more below):

How to add custom summaries?

LLDB commands may be run either through the command-line (also available during a debug session in the IDE) or via a .lldbinit file in the home directory.

It is also possible to automatically load .lldbinit files from the current directory by specifying settings set target.load-cwd-lldbinit true in ~/.lldbinit , which is useful for project-specific customizations.

Additional .lldb and .py files may be loaded via:

command script import /path/to/script.py
command source /path/to/script.lldb

The ARGB summary above can now be defined as a Python function:

def ColorSummary(valobj, internal_dict):
  value = valobj.GetValueAsUnsigned(0);
  alpha = (value >> 24) & 0xFF
  red = (value >> 16) & 0xFF
  green = (value >> 8) & 0xFF
  blue = (value >> 0) & 0xFF
  return 'A: {}, R: {}, G: {}, B: {}'.format(alpha, red, green, blue)

To register it, first import the file, Summaries.py, and associate the ColorSummary function to the Color type:

command script import ./Summaries.py
type summary add -F Summaries.ColorSummary Color

More information about defining function summaries can be found here.
The valobj variable is just a wrapper around the C++ variable and its type is an lldb.SBValue.

Synthetic Children

LLDB also supports defining custom children for any type. Instead of having the color channels be part of the summary, let's define them as synthetic children.

class ColorSyntheticProvider:
  def __init__(self, valobj, internal_dict):
      self.valobj = valobj

  def num_children(self):
    return 4

  def get_child_at_index(self, index):
    if index < 0 or index > 3:
      return None

    value = self.valobj.GetValueAsUnsigned(0);
    value = (value >> 24 - index * 8) & 0xFF
    name = ['A', 'R', 'G', 'B'][index]
    childType = self.valobj.target.GetBasicType(lldb.eBasicTypeInt)
    return self.valobj.CreateValueFromData(name, 
                                           lldb.SBData.CreateDataFromInt(value), 
                                           childType)

Reference: Synthetic Children, lldb.eBasicTypeInt, lldb.SBData

And the command to register the provider inside the .lldbinit file:

type synthetic add -l Summaries.ColorSyntheticProvider Color

Where to go from here?

The official documentation is excellent and covers all the details.
Filters may be used to hide children that are not useful for debugging.
Regex Typenames are particularly handy for registering summaries for templated types.
More examples can be found on the LLVM repository.
For specific information about any command, simply type

help <command> <sub-command> .

3 Free Ways of Shortening URLs with Firebase

Martin Vejdarski — Fri, 07 Feb 2020 06:45:02 +0000

Shortening URLs makes long and obscure URLs look better and allows for going back and quickly editing the URLs they replace. Tweets, emails and many other things can’t be edited. However, URLs don’t have to be set in stone. The following are three ways of shortening URLs with Firebase.

Prerequisites

Sign in to Firebase and create a new project.
Go to the Hosting tab from the sidebar and add a new site.

(No need to setup CLI or install anything at this point.)

1. Dynamic Links

As the official successor to Google URL Shortener (migration guide), Dynamic Links provide a quick and easy, no code required, approach to setting up custom URLs.

Steps:

Go to Dynamic Links in the Grow section of the Firebase Console’s sidebar and follow the Get Started instructions.
Click the New Dynamic Link button.
Choose a prefix for the new URL.
Enter the original/long URL in the Deep link URL field.
Enter a memorable name for the link in the Dynamic Link name field.
Skip the rest and click Create.
All done. The URL is now live.
Note: Using hosting/multiple sites with Dynamic Links requires additional steps, so keeping a separate Firebase Project might be easier.

2. firebase.json

This approach takes advantage of the project configuration file and after a quick setup, allows for adding and modifying URLs by merely editing the firebase.json file.

Follow the Hosting Quickstart steps to set up the CLI and create the project folder. The firebase.json file inside the project folder will contain something like:

{
  "hosting": {
    "public": "public",
    "ignore": [
      "firebase.json",
      "**/.*",
      "**/node_modules/**"
    ]
  }
}

To add a link, let’s configure the redirects (more here):

{
  "hosting": {
    "public": "public",
    "ignore": [
      "firebase.json",
      "**/.*",
      "**/node_modules/**"
    ],
    "redirects": [
      {
        "source": "/google",
        "destination": "https://google.com",
        "type": 302
      },
      {
        "source": "/",
        "destination": "https://firebase.google.com",
        "type": 302
      }
    ]
  }
}

The above forwards example.com/google to google.com and example.com (the root URL) to firebase.google.com. (Assuming example.com is the domain set in the Hosting tab on Firebase Console).
Run firebase deploy --only hosting in the console to deploy the new configuration. The links should now be live.

Multiple sites

When working with more than one site, the configuration will look a bit different. Example:

{
  "hosting": [
    {
      "target": "app",
      "public": "app",
    },
    {
      "target": "links",
      "public": "links",
      "redirects": [
        {
          "source": "/google",
          "destination": "https://google.com",
          "type": 302
        },
        {
          "source": "/",
          "destination": "https://firebase.google.com",
          "type": 302
        }
      ]
    }
  ]
}

Each site becomes a target, which needs reflecting in the firebase.json file. Instead of an object, hosting now contains an array of objects. Each target needs setting up in the CLI by associating its name with its “resource name” as defined on Firebase Console’s Hosting tab.

Setting up the targets in the CLI:

Create a new target: firebase target:apply hosting links name-on-firebase
Deploying a specific target: firebase deploy --only hosting:links

More info is available on the official docs.

3. Realtime Database + Javascript Redirect

This approach follows an excellent tutorial from Abe Haskins, which consists of setting up the Realtime Database as well as deploying a single page that immediately forwards to the corresponding long URL by using Javascript.

This allows for easy editing of the URLs directly from the Database tab on the Firebase Console. However, unlike the first two approaches, it relies on client-side redirects instead of server-side ones (more on the differences can be found here).

DEV Community: Martin Vejdarski

UTF-8, quick and easy

Text is simpler than it looks!

Why can't we have nice things?

UTF-8 vs ASCII, UTF-16, UTF-32

Isn't UTF-16 safe in most cases?

Can't we just use UTF-32?

Code points ≠ characters

How is UTF-8 stored?

DLL Symbol Visibility in C++

Static and Dynamic Libraries

Symbol Visibility

DLLs and linking errors (LNK2001, LNK2019, and LNK1120)

How are symbols exported/imported?

Where should the macros be placed?

References:

Prettifying Debug Variables in C++ with LLDB

Prerequisites

What is variable formatting?

Why do we need to customize them?

Consider the following example with color:

How to add custom summaries?

Synthetic Children

Where to go from here?

3 Free Ways of Shortening URLs with Firebase

Prerequisites

1. Dynamic Links

Steps:

2. firebase.json

Multiple sites

3. Realtime Database + Javascript Redirect

All Done! Time to make some URLs! 🌍

DLLs and linking errors (`LNK2001`, `LNK2019`, and `LNK1120`)