DEV Community: Patrick Stival

Understanding Geohashes

Patrick Stival — Mon, 24 Jul 2023 21:31:03 +0000

Hello there.

Are you curious about how geographic locations can be represented as a short string of letters and numbers? Or how this technique helps in fast retrieval of location data?

This post will take you on a deep dive into geohashes, explaining what they are, how they work, and their practical uses. Whether you're a data scientist, a software engineer, or just someone with a knack for geography and data, this post will help you grasp the concept of geohashes and their significance.

The purpose of this post is to explain in depth how a geohash works and its subtleties step-by-step.

What is a geohash?

Chat GPT says:

A geohash is a location encoding system that represents a geographic location using a short string of letters and numbers. It works by dividing the world into a grid of squares, assigning each square a unique hash. The length of the hash controls the precision, with longer hashes corresponding to smaller, more precise squares.

That gives us a general idea. You use a geohash to identify areas on a map. The easiest way to visualize them is with an interactive map

Why geohashes

Databases are better optimized for text than geolocation queries. If you need fast retrieval over a large collection of data, R-Tree indexes might not be enough.

Quoting Chris Hewett's website:

Geohash avoids/minimises the use of (usually) slow geospacial functions. Who you going to call when your ST_CONTAINS() is slow and an EXPLAIN shows it's already using an index...

Also, geohashes can be used as an alternative to tile coordinates in order to load chunks of a map and possibly cache them on the client side.

Different precisions

As chat gpt said, a geohash length describes its precision. For instance, dr7 is a geohash with precision 3. That means that its parent, dr (precision 2), encloses a larger area that also contains its neighbors.

2 types of geohashes

There are 2 types of geohash grids (width x height): 8x4 and 4x8. That means that for any given zoom level, there are 32 possible squares inside.

We start with an 8x4 grid with the whole world inside. Each time you go one precision deeper, the type of grid you get alternates. This is made this way, because if we kept grids in the same format for all zoom levels, we would eventually get super thin areas.

Since there are 8x4 and 4x8 grids only, we use these 32 characters to encode each precision: 0123456789bcdefghjkmnpqrstuvwxyz.
They range from 0 to z, but beware, there are a few letters missing in this specific encoding, such as a;i;l and o.

Keep dividing by 2

If you want to find out the geohash in which a point lies into, you have to bisect it 5 times. Why 5? Because 2^5 is 32, which is the total amount of geohashes for each precision level.

Let's do a dry run of the algorithm to figure out in which geohash1 a location L in argentina lies into.

purple: divide the world in 2. L is in the first half
red: L is beneath
orange: L is to the right
yellow: L is above
green: L is to the left
We have run 5 tests already, so the geohash is 6

Notice that we always have two possible moves for each step: to choose between the first and the second half. The moves alternate between latitude and longitude, so we are able to move in 2D.

Also, we have to somehow translate the accumulated result of those moves into an index for the string 0123456789bcdefghjkmnpqrstuvwxyz.

The rules for that are:

We start with an index valued 0
If the location lies in the first half (lower than mid), multiply the index by 2
If it's the second half (higher than mid), multiply the index by 2 and add 1
Repeat 4 more times

That means we are using even numbers to identify first-half moves and odd numbers for second-half moves.

The highest value we can get is 31:

0 * 2 + 1 = 1
1 * 2 + 1 = 3
3 * 2 + 1 = 7
7 * 2 + 1 = 15
15 * 2 + 1 = 31

Since our index starts at 0, that would be 32 possibilities.

This is exactly what the code is doing:

// (c) Chris Veness 2014-2019
let latMin = -90, latMax = 90
let lonMin = -180, lonMax = 180

while (geohash.length < precision) {
  if (evenBit) {
    // bisect E-W longitude
    const lonMid = (lonMin + lonMax) / 2
    if (lon >= lonMid) {
      idx = idx * 2 + 1
      lonMin = lonMid
    } else {
      idx = idx * 2
      lonMax = lonMid
    }
  } else {
    // bisect N-S latitude
    const latMid = (latMin + latMax) / 2
    if (lat >= latMid) {
      idx = idx * 2 + 1
      latMin = latMid
    } else {
      idx = idx * 2
      latMax = latMid
    }
  }

  evenBit = !evenBit

  if (++bit === 5) {
    // 5 bits gives us a character: append it and start over
    geohash += base32.charAt(idx)
    bit = 0
    idx = 0
  }
}

console.log(geohash)

Then we stop once the precision is met.

Alternatives to geohash

If you check the h3geo comparison page, you should see plenty of alternatives to geohash, such as s2 or even h3 itself.

Glossary

ST_CONTAINS: postgis function that finds geometries inside areas;
EXPLAIN: postgres statement that calculates query cost

Habbo: Avatar Rendering Basics

Patrick Stival — Fri, 18 Sep 2020 22:49:42 +0000

Hey there!
In this article, I'm going to explain how you can render a pretty figure like this one:

The avatar images in this article were rendered using Open Hotel Client. If you like habbo or games development in general, please consider contributing to the project or joining the team. My e-mail is at the bottom of this article.

Oh right, and it can also walk, swim, lay and face 8 different positions! Sounds like a lot of work, and it's indeed something susceptible to a bunch of edge cases.

The goal here is to learn how Habbo handles their assets to build an avatar.

Naming convention

An avatar figure is built using a combination of multiple body and clothing parts. You can use the habbox standalone avatar imager to try some combinations and get a string that describes your character.

At Open Hotel, we provide the same options as the habbox standalone imager for the avatar rendering (which is also the default habbo api pattern). Those are:



{
  look: 'hd-180-1.hr-110-61.ch-210-66.lg-280-110.sh-305-62',
  action: 'mv,respect',
  direction: 2,
  head_direction: 2,
}

Figure parts

In this example, our encoded avatar figure is:



hd-195-1.hr-679-61.ha-1012-110.ch-804-1341.lg-275-110.sh-3089-110

Each figure part is separated by a ., and each one of these parts can be described as:
figureType-imageID-colorID1-colorID2...-colorIDn

Actions

Actions change the way we build the avatar figure. Take by example the std, laugh and mv actions:

Notice that multiple actions can occur at the same time, like sitting and waving. According to the action applied, some body parts might stay the same, while others are changed.

Figure Parts Example

Combining parts

Let's try to render them separately at Open Hotel and see what we get:

hd-195-1: Body + face with key 195 and color 1

hr-679-61: Hair with key 1012 and color 61

ha-1012-110: Hat with key 1012 and color 110

ch-804-1341: Shirt with key 804 and color 1341

lg-275-110: Trousers with key 275 and color 110

sh-3089-110: Shoes with key 3089 and color 110

All figures combined:

Notice: the figure part key is different from the figure part id, as you can see in the Figure Data section of this article.

Take a look below at the hairs hr-110-61, hr-677-61, hr-3048-61, hr-165-61. All of them with color 61:

Figure Data

Open Hotel provides a file called figuredata.json. This file holds information we need to get the right image for each one of our figure parts.
It is based on habbo's figuredata.xml, but we converted it to json to make it easier to use.

By using it, we can selectively lazy-load the image files as we need, since it wouldn't be practical to load everything in memory at once.

It basically holds two first-level keys:

pallete is a dictionary that maps a palleteid to a dictionary of colors.
settype keeps track of the palette, the metadata (like gender) and the images we need to use for each figure part.

Let's try a step-by-step render for the hr-679-61 figure (hair 679 with color 61). Since it's a hair, it's held under the hh_human_hair lib.



// figuredata.json
{
  "palette": {
    // 3. get color "61" hex from palette "2"
    "2": {
      "32": { "color": "DFA66F" },
      "61": { "color": "2D2D2D" }
    },
  },
  "settype": {
    // 1. Access the figure type, which is "hr"
    "hr": {
      // 2. Access the palette with id 2
      "paletteid": "2",
      "set": {
        // 4. Get hair with key 679
        "679": {
          // Both genders accept this hair
          "gender": "U",
          "parts": [
            {
              // 5. Since type is hr, get this part 
              // id and move to figureMap.json
              "type": "hr",
              "id": 27,
              "colorable": 1,
              // for part sets with more than one
              // color, multiple color indexes can be used
              "colorindex": 1
            },
            {
              // hrb is used when the avatar is using a hat
              "type": "hrb",
              "id": 27,
              "colorable": 1,
              "colorindex": 1
            }
          ]
        },
      }
    }
  }
}

Figure Map

The figuremap.json holds the libraries names mentioned at the Naming Convention section.

At figuremap.json, the first-level keys are:

libs
An array with all the libraries names.
parts
A dictionary that maps a partset name and a part id to an index of the libs array.

To get the lib we need, let's follow the steps:



// figuremap.json
{
  "libs": [
    // ...
    // 3. Get the lib id from the 1004th position of the array
    { "id": "hh_human_hair" },
  ],
  "parts": {
    "ha": {
      // ...
    },
    // 1. Access the "hr" partset
    "hr": {
      // 2. Access the part with id 27 and get its lib index
      "27": 1004,
    }
  }
}

Now we have all the information we need to get our hair image, we just need to build the image file name.

Image files naming

The resultant file name should be:
hh_human_hair_h_std_hr_4_2_0, where each part of the file indicates:

hh_human_hair: this file is a part of the human hair lib
h: the image size (it could be sh if it were zoomed out). At Open Hotel we're not using sh images, since zoom is handled by pixi-viewport
std: the image's action. std is the standard, but it could be wlk for walking, or sml for smile.

For some reason, the action mv matches the files with the wlk action name. This mapping happens at animations.json. I might cover animations in detail in a future article.
hr: the figure part, which in the case is hair.
4: the figure part id for this specific hair.
2: the position, which can vary from 0 to 7 rotation clock-wise

0: the animation frame. Actions like std only have one frame (frame 0), but for animations like walking (mv) and waving(wave), more frames are required.

Conclusion

This tutorial might be confusing, but that's because a lot of mappings are required, and animating can get even more complicated.

The goal here is to provide a general idea of how the rendering process works at Habbo and also encourage people to contribute to Open Hotel.

The current client active branch is structure-migration. If you run this branch, you should be able to look at the code that actually rendered the images present in this article.

If you're interested in contributing or have any questions, you can contact me at trickstival@gmail.com.

Thanks!!

How to classify log status on DataDog

Patrick Stival — Mon, 13 Jan 2020 20:30:44 +0000

DataDog is a great tool for Data Analysis and Log Management. I started using it recently, and the tools they provide really impressed me.

First things first, you will need to collect your logs before monitoring them. There are many ways to do so, through preexisting integrations or by collecting them from files.

By default, every log message is classified as info, but it can be changed to any Severity Level described by syslog.

The possible log status are:

Emergency
Alert
Critical
Error
Warning
Notice
Informational
Debug

Once you have collected your logs, it's time to classify them.
To do so, you have to create a pipeline.

Pipeliiines

Pipelines are workflows that hold one or more processors inside. Processors are predefined data manipulators that can be executed in line. For this post we are going to use 3 Processors:

Grok Parser
String Builder
Status Remapper

Creating a Pipeline

Go to Logs > Configuration on the sidebar menu
Click on "New Pipeline" at the upper corner of the page
Name your Pipeline
Once you created the pipeline, you should be able to add processors:

Parsing the logs

The next step is to parse the logs. For that purpose you can use the Grok Parser and extract information from your text. You can find more information about parsing rules by clicking here.

Choose the Grok Parser as the processor.
Then, provide some log samples (you can get those on the Data Dog Logs Page) and write your own Parsing Rules.

A Parsing Rule is defined by the following structure:



<Name of the Rule> Pattern %{matcher1:attribute}...

You can also use filters combined with matchers

<Name of the Rule> Pattern %{filter1} Pattern2 %{matcher1:attribute}

Read more about matchers and filters.

Check out the Grok Parser configuration:

In most cases, just one rule should be enough. In the example there is just one rule called ImmaRule and we collect the attribute timestamp as a date and username as a word, by using matchers.

Once you extracted all the data you needed, a JSON representation of the attributes and values extracted through parsing you specified with the data samples will show up on the bottom of the screen:
.
Name your processor and click save.

Classifying the logs

The Grok Parser will keep forward only with the logs that matched the pattern you specified earlier. Since it's a pipeline, you can safely classify all logs filtered as errors, warnings, or any other status.

In this case I want to classify them as errors. For this purpose, we're adding a String Builder processor to the pipeline beneath the Grok Parser.
This way we can store the log level in a new attribute and officialize it as the status attribute later by using the Log Status Remapper processor.

So I'll add the String Builder processor and add the attribute level to it, with the value error:

And finally set the new level attribute as the status attribute, by adding the Log Status Remapper processor after the String Builder on the pipeline:

And finally, our pipeline is done!

Now you should get your logs on the Log Explorer flagged with the level you chose previously

Thanks for reading.
I hope this tutorial to be somehow useful for you.