DEV Community: Paul Götze

The
Of a Dropdown

Paul Götze — Sat, 30 Apr 2022 08:49:33 +0000

How to build a beautiful dropdown panel with just HTML and CSS

I would by no means consider myself an expert when it comes to HTML & CSS. But I thought I just share what I found to work really well for me, when I tried to build a dropdown panel without any JavaScript.

tl;dr: You can build a fully functional dropdown panel by using plain HTML and CSS with no JavaScript involved by leveraging the <details> and <summary> tags. Here’s a minimal example that you can use and customize to your needs:

If you want to see more advanced dropdowns built with this approach and want to learn about what’s going on here, then read on.

There is an HTML tag that might count among the little known but most powerful tags in the world of websites: the <details> tag.

It usually appears together with another tag that makes the <details> tag shine: the <summary> tag.

It basically does what it says — it shows you a summary. And you can click on it to see some details. The best feature is, that it comes out of the box with each browser, no JavaScript needed for toggling the details. Just plain old HTML. Here’s what it looks like by default:

Wow, not sparking too much joy, yet. But we’ll get there.

Let’s Spice It Up 🌶️

With a bit of styling, the plain <details> and <summary> snippet from above can already be used for such exciting things as questions (summary) and answers (details) on your FAQ page.

But let us explore some even more exciting use cases. Let’s build a dropdown menu that might live in the navigation on your web page.

First, we put a container around our <details> snippet which will represent the dropdown element. Then we replace the former summary text with a hamburger menu icon and insert a list as the actual dropdown content. Last, we wrap the whole dropdown with a <nav> tag and give all of that some quick color styles. Et voilà:

Hiding The Disclosure Widget

The <details> tag comes with an open attribute and a disclosure widget (▶, ▼) which indicates whether the dropdown is opened and details are visible. Let’s hide this marker, so that the click area looks more like a general button.

In Firefox we can do so by setting list-style: none; for our summary. In other browsers you need to apply display: none; for the summary’s pseudo-class ::-webkit-details-marker.

We’ll also fix the cursor behavior for the <summary> on the fly. So, this is what we end up with after applying these fixes:

Detaching The Details

This might already serve well as a simple dropdown menu. But we can do much better.

Right now, when opening the details, the content enlarges the details container and therefore also pushes down any other content that lives below our navigation bar. In more complex navigations we might like to have a dropdown panel that is detached from the triggering summary.

So, next up, we’ll add some styles to display the actual menu content in a position-wise independent panel.

We can get an independent menu content by adjusting its position property. Hence, we wrap our content into a <div> container and give it an absolute position. With this, the opened menu content is now displayed directly below the summary, which is the opening trigger for our dropdown menu.

In order to define if the panel is opened to the left or the right of the triggering summary, we apply a position: relative; for the dropdown

container. You can now customize the dropdown panel’s anchor by applying right: 0; or left: 0; (which is the default) to the summary.

When using a relative position, we also need to make sure the dropdown panel has a minimum inline size of its maximum content width – else our menu items will have unwanted line breaks.

By giving the dropdown <div> container an additional display: inline-block; we make sure the dropdown panel only opens when clicking the hamburger menu icon directly.
Similar to before, we also add some list styles to make it already look like a menu and make it more distinct from the navigation and the text content below:

Closing The Menu

Opening and closing the dropdown works nicely and it looks like a navigation menu alright. However, there’s still an issue if we open our menu and decide to not click anything in it but leave for interacting with other content on the page. Then our menu will still be wide open and cover underlying content.

So, we need to figure out how to close the <details> again, whenever we click somewhere else outside the menu area.

We can use a neat little trick to reach this behavior — again without any JavaScript involved. When the <details> container is in the open state, then clicking any <summary> content will hide the menu content.

We can leverage this behavior by making sure, that the only area we can click on outside the menu content will always be the <summary> area. So, a click anywhere else on the page would always trigger closing the menu.

Technically this is possible by enlarging the summaries ::before pseudo-class to the full view size. This is done by giving it a fixed position and expanding it to all four view corners (setting top, right, bottom, and left to 0).

In order to fill the whole screen we also need to set the content of the ::before pseudo-class. By default the details content is displayed with a higher z-index than the related summary content, so we don’t need to care about this. You can still interact with the menu content.

The next example applies a transparent background color for the summary::before’s content, so that we can see the area covering the entire screen behind the menu content:

Conclusion

This approach can be used for a multitude of different dropdown panels, including navigation menus, sharing widgets, and all sorts of buttons that open a panel with further details or actions.

In fact, GitHub is using a similar approach for their clone button and the branch select panel — which is also where I took inspiration from to kind of reverse-engineer the described approach:

GitHub’s “Code” & “select branch” dropdown is built using <details> and <summary> tags

With some minor additions and a couple more CSS styles we can make our example into a shiny dropdown menu, that works just as you would expect a dropdown menu to work:

As long as you don’t want to put any more complex interactions into the dropdown, the described approach does not need any JavaScript.

However, if you don’t have a page reload after clicking a link in the dropdown panel, you would need to add some JavaScript to toggle the <details>s open state. The same applies for any additional actions from within the dropdown panel that should change its open state.

I hope you learned some useful details about how to built dropdowns. We can give the positive summary that you might not always need JavaScript to build interactive web components.

HTML and CSS might have you covered in more cases than you think. For dropdown menus and dropdown panels it certainly does.

This post was originally posted on my personal blog — Easter egg included ;)
https://paulgoetze.com/2020/05/09/the-details-of-a-dropdown

Credits: header image by Fabrizio Conti.

Testing your JSON API in Ruby with dry-rb

Paul Götze — Fri, 29 Apr 2022 12:21:42 +0000

When writing a JSON API with Ruby in your favorite web framework, you will certainly come to a point, where you have to decide how to test your endpoints. In this article I’m going to show one of my preferred ways to test the structure of JSON responses in a clear and readable way—by making use of the dry-schema gem and some other dry-rb libraries.

There’s Different Ways

When it comes to testing your JSON API endpoints in Ruby, you can go for plenty of different approaches. The simplest probably is to check the status code and explicitly assert each value of you response data.

Let’s assume we want to test a GET /todos endpoint, which returns to-do items like this:

Then a simple test for this response could look like this:

However, this approach is a bit cumbersome for large response bodies and can result in a lot of accesses to nested hash keys, until you get to the value you actually want to check. It will also take you quite some effort to adjust the tests as soon as some keys or values in the JSON response change.

So, instead of checking for each exact value, you could move up one level of abstraction and only check for the right structure and data types in your response body. Again there’s different ways to do that.

You could use JSON Schema files and validate your response against them, as is described here:

Validating JSON Schemas with an RSpec Matcher

Use RSpec and JSON Schema to create a test-driven process in which changes to the structure of your JSON API drive the implementation of new features.

thoughtbot.com

The JSON Schema approach works quite well. However, writing JSON Schema files can get a bit tricky and confusing, especially when you have to deal with complex and nested JSON data. There’s ways to handle this complexity, e.g. by splitting up and reusing schemas by referencing them.

But maybe there’s a more legible and maintainable way than JSON Schema validation. Let’s explore the realm of dry-rb for this purpose.

What is dry-rb?

dry-rb is a collection of Ruby libraries, whose goal is to encapsulate common tasks–like data validation, data transformation, Ruby object initialization, and others–in small, reusable, and framework-independent Ruby gems.

For our purpose of validating the structure and types of JSON data it provides us with the dry-validation, dry-schema, and dry-types gems. These gems give us all we need for setting up a schema to validate JSON data of almost any complexity.

So let’s have a look at some validation use cases and let’s see how we can leverage the mentioned dry-rb gems for a simple JSON structure. We will also dig into a more complex one afterwards.

A Simple JSON Response

The dry-schema gem comes with a Dry::Schema module which has a Params method to define schemas for validating hashes. Built on top, it also has a JSON method for defining JSON schemas with coercible data types, like Date or Time. This is the one we want to use for our response validation.

A schema for one of our to-do items from above would look something like this:

With these definitions we make sure that an id, title, and the done flag is present in our to-do item JSON object–and that each property has the given type.

Instead of required() we can also use optional(), to allow the object to not have this property. Instead of value() we can also put maybe() to allow nil values.

Having this schema defined, we can now run the validation in our test for each to-do item by calling the schema, instead of checking for each value explicitly as we did before:

Dry-schema also allows us to nest schemas and to pass schemas as property types. So, let’s build a schema for our entire JSON response:

Putting all of this together in our test case we can now validate our response body to match the ResponseSchema:

Of course you might still want to check some of the JSON data’s properties explicitly, e.g. to make sure the right to-do ids are involved.

Nevertheless, using dry-schemas to validate the response structure makes your test code a bit more concise already and also makes it easier to understand what the actual JSON response looks like. You can have a quick look at the involved schemas to get an idea about required or optional properties and their (maybe nullable) types.

A More Complex JSON Response

You might think: “Well, nice little examples, but my real-world JSON data is way more complex. Can I use this for these structures, too?”.

I think you can. Let’s see how.

Let’s replace some parts of our simple to-do JSON with more complex ones. Instead of the done flag we will have a done_at timestamp, we will add a created_at timestamp and a priority (which could be one of high, default, or low). We will also assume that we have a collaborative to-do list, where we want to include the creator and assignee for each to-do item. Last, let’s have an arbitrary list of tags that can be assigned for a to-do item. This is how our new JSON response might look like:

A very basic schema for this response would be something like the following:

Note that we used maybe() for the done_at property, which allows to have nil values.

This only validates the presence and the top level type of each property. The dry-types gem that is used by our schema, however, provides us with a number of built-in types, like Types::JSON::DateTime (which will be used under the hood if we type :date_time), Types::Array, and Types::String.enum. We can use these to refine our timestamps and tags types in the schema. In order to create custom types we need to create a Types module and include the dry types. Similar to what we already saw for the simple JSON response, we can also create and reuse sub-schemas for the creator and assignee properties:

This now looks already more like a real-world use case!

Still, we can go one step further and check for the integrity of certain properties. With dry-validation (which is powered by dry-schema) we have a tool at hand, that allows us to define custom rules for our JSON data on top of our existing schema definitions.

Before we can define custom rules we need to change our schema definition from the Dry::Schema.JSON block to a response class that inherits from the Dry::Validation::Contract class:

The Dry::Validation::Contract base class comes with a rule method that we can use to raise custom validation failures if certain conditions match.

Inside a rule we have access to a values variable, which holds the value(s) of the configured property for that rule. As soon as we assign a failure message (by calling key.failure("some message")) we will make the validation fail.

Let’s say we want to validate the done_at DateTime to always be after the created_at DateTime, and each tag in tags to only appear once. Then these might be our rule definitions:

You can define whatever rules you can come up with to customize your validation contract and to adjust it to your needs. Dry-rb already provides the most often used data types, but you can also use fully customized ones as we saw in the examples above.

Note that our first ResponseSchema definition, where we made use of Dry::Schema.JSON, gave us an instance of the schema class already. When using the Dry::Validation::Contract we need to create an instance of our ResponseSchema ourselves, before we can call its call() method to validate our data.

For more details and dry-validation features I encourage you to checkout the dry-validation documentation.

Some Assertion Syntactical Sugar 🍬

One issue with directly asserting if a schema validation succeeds is that you won’t see what went wrong if the validation failed. Dry-schema provides detailed error messages, so we can make use of these to debug our failing response validation.

We can fix the visibility issue of failure messages by setting up a small test helper method:

If the validation fails you will now be pointed to the erroneous properties and you will see a message about what went wrong. On top you will get the original JSON response, so you can spot the validation issue in no time.

Conclusion

We saw that dry-rb gems, specifically the dry-schema and dry-validation gems provide us with some useful tools to make JSON response tests in Ruby more legible and maintainable.

On top of making our JSON response tests more concise, the schemas we set up with this approach also allow us to have a quick overview about the structure and data types of our endpoint’s response.

Schemas can be reused in other schemas and tests. They are easy to change and framework agnostic. So, although I used minitest assertion syntax in my code examples, with some small adjustments the shown approaches can also be applied if you use RSpec or any other Ruby testing library.

I hope you enjoyed this little journey into JSON response testing with dry-rb—thanks for reading and I’m curious to hear about your experiences with using dry-rb gems or similar approaches in your testing environment.

Happy coding, happy testing!

Credits: dryer icon in the header image by photo3idea_studio from flaticon.com

Experiencing The Stroop Effect With a Ruby CLI

Paul Götze — Wed, 24 Jun 2020 18:58:26 +0000

You’ve probably seen these weird lists of color words before. Words whose text color does not match their content. Maybe you’ve even tried to read the text colors out loud quickly and have struggled to do so?

These lists are part of psychological tests to demonstrate the so-called Stroop effect. It is named after the American psychologist John Ridley Stroop, who first published the effect in English in 1935.

Wikipedia tells us:

"The Stroop effect is the delay in reaction time between congruent and incongruent stimuli."

But what does that actually mean? 🤔

I built a tiny command line interface with Ruby – just because it was fun and so that you can experience the Stroop effect yourself. You can install and run it with:

gem install stroop
stroop

Congruent? Incongruent?

A Stroop tests consists of a small task, like reading color words. This task can be done with different kind of stimuli, namely: neutral, congruent, and incongruent.

Let’s take a list of color words with a neutral stimulus, which means that all words are written in the same text color:

Neutral Stimulus – All color words are written in a neutral text color (black, or here gray)

Reading all words out loud should not be hard to do.

Next let’s take the same list of color words, but let’s print them in a color that matches the respective text. That’s a congruent stimulus:

Congruent Stimulus – The word content matches the text color.

If you read he words again, it might even be easier than with the neutral stimulus.

Last, let’s print the same word list again, but this time we use another text color than the color that is represented in the text. That’s an incongruent stimulus:

Incongruent Stimulus – The word content and the text color are different.

Reading the colors words might feel a bit slower now. However, the Stroop effect is much more apparent, if we slightly change our task:

Instead of reading out loud the color words, try saying the text color for each word as fast as possible!

With the incongruent stimulus you probably need much longer now and you might even make one or the other mistake.

And that's the Stroop effect: the delay that appears if we speak the text color out loud for the incongruent word list compared to the congruent word list.

Isn’t it fun – and really exhausting?

🌱 The above word lists were generated using the stroop gem:

stroop neutral
stroop congruent
stroop incongruent

with the seed: 134414671674842647560860440639024210370

(FYI: I found this tiny stroop CLI to also be perfect for creating colorful artsy-fartsy wallpapers & online profile backgrounds 😉)

Building a City Search with Elixir and Python

Paul Götze — Thu, 28 May 2020 21:37:10 +0000

The other day I was wondering whether there was an easy self-made local alternative to something like the Google Places API, that I could use in a Phoenix app. I wanted to search for a city and wanted to get back the city itself, its state, and the country.

I found the free GeoLite2 city dataset, provided by Maxmind, which I could use to create a city search index.

In case you directly want to dive into the programmatic materialisation of what I came up with, it is available on Github.

I did a quick search and stumbled upon the searchex project by @andyl. This actually looked like it was exactly what I was searching for. However, there is very little documentation yet. So, unfortunately, I couldn’t really figure out how to get it working.

Then, while thinking about how to approach this, Whoosh, a Python package that I have used at work, came to my mind. Whoosh is a library for indexing text and searching the index. It is pretty easy to set up and delivers great search results with little effort.

With this in my mind, I was wondering whether there was a way to call Python code from Elixir. After some further research and some articles later I found the Erlang library erlport, which allows you to call Ruby and Python code from Elixir.

There is also an Elixir wrapper for it, bearing the sounding name Export. You could also use erlport directly in Elixir, but Export gives you some convenient functions on top and a more Elixir-like feeling.

Setting Up a New mix Project & Python virtualenv

In order to get started with our custom city index, let’s set up a prototype mix project, called elixir_python:

mix new elixir_python

Head to the mix.exs file and add the export dependency:

# mix.exs
# ...
defp deps do
  [{:export, "~> 0.1.0"}]
end

Then install the dependencies with:

mix deps.get

For setting up a Python environment you can use virturalenv to create a local virtual environment. Also keep in mind to activate it after creating:

virtualenv -p python3 venv
source venv/bin/activate

We will use Whoosh, so we need a requirements.txt next to our mix.exs that defines the Python dependencies:

# /requirements.txt

whoosh==2.7.4

Install the requirements with:

pip install -r requirements.txt

Next, we need a directory where our Python code will live. Let’s create a lib/python directory where we will put the *.py files later on. You can really put them wherever you want, you just have to link to the directory when using Export.

In your lib/python directory create a geolite2.py file. This is where we will put the code for our city search index. Next, download the GeoLite2 CSV files from dev.maxmind.com and put the English city locations in the /lib/python/data directory. For our Python requirements we will also need a requirements.txt file in our project’s root directory.

Our Elixir code will live in lib/elixir_python/geolite2.ex.

The overall project structure should now look like this:

└── elixir_python
    ├── config
    ├── lib
    │ ├── elixir_python
    │ │ └── geolite2.ex
    │ ├── python
    │ │ ├── data
    │ │ │ └── GeoLite2-City-Locations-en.csv
    │ │ ├── __init__.py
    │ │ └── geolite2.py
    │ └── elixir_python.ex
    ├── mix.exs
    ├── requirements.txt
    └── …

The Python Part

Our geolite2 Python module will have an API composed of two functions:

# lib/python/geolite2.py

def create_index():
    # We will add code here in some minutes...
    pass

def search(query, count=10):
    # We will add some code here soon...
    pass

The first one creates our search index using the GeoLite2 city CSV file. The second lets us search for cities, states or countries and will pass the results back to Elixir.

Indexing the City Data

For each Whoosh index you can define a certain structure, its schema. The schema defines which data you want to store in the index and which fulltext – or content – you want to run the search on.

Our city schema looks like this:

from whoosh.fields import SchemaClass, TEXT
from whoosh.analysis import NgramWordAnalyzer

class CitySchema(SchemaClass):
    city = TEXT(stored=True)
    state = TEXT(stored=True)
    country = TEXT(stored=True)
    content = TEXT(analyzer=NgramWordAnalyzer(minsize=2), phrase=False)

We want to store the city, the state, and the country. The content field will hold the fulltext to search in, in our case it will be the joined city, state, and country name. This allows us to also search for cities, states or countries and provide multiple query terms to narrow down our results.

We use an NgramWordAnalyzer and set the phrase argument to False in order to save some space (see this whoosh recipe for more details).

Before creating the index let’s define our directory names and files we want to use along with some handy functions for building the absolute paths to these files:

# lib/python/geolite2.py

import os

# The base directory where out data lies, relative to this file
DATA_BASE_DIR = 'data'

# The actual city data file
CITY_DATA_FILE = 'GeoLite2-City-Locations-en.csv'

# Our base directory where the index files are stored
INDEX_BASE_DIR = 'index'

# The name of our index
CITY_INDEX = 'city'

def index_path(index_name):
    """ Returns the absolute index path for the given index name """

    index_dir = '{}_index'.format(index_name)
    return os.path.join(current_path(), INDEX_BASE_DIR, index_dir)

def data_file_path(file_name):
    """ Returns the absolute path to the file with the given name """

    return os.path.join(current_path(), DATA_BASE_DIR, file_name)

def current_path():
    """ Returns the absolute directory of this file """

    return os.path.dirname(os.path.abspath( __file__ ))

Armed with these helpers we can now go ahead and define the actual index creation function. We read the CSV file line by line, and create the schema from it. Some lines in the CSV do not represent cities but states or countries, so we skip these lines, unless there is a value in the city column:

# lib/python/geolite2.py

import csv
import shutil

from whoosh.index import create_in

# ...

def create_index():
    """ Create search index files """

    path = index_path(CITY_INDEX)

    _recreate_path(path)

    index = create_in(path, CitySchema)
    writer = index.writer()

    with open(data_file_path(CITY_DATA_FILE)) as csv_file:
        reader = csv.DictReader(csv_file)

        for row in reader:
            _add_document(data=row, writer=writer)

    writer.commit()

def _recreate_path(path):
    """ Deletes and recreates the given path """

    if os.path.exists(path):
        shutil.rmtree(path)

    os.makedirs(path)

def _add_document(row, writer):
    """ Writes the data to the index """

    city = row.get('city_name')

    if not city:
        return

    state = row.get('subdivision_1_name')
    country = row.get('country_name')
    content = ' '.join([city, state, country])

    writer.add_document(
        city=city,
        state=state,
        country=country,
        content=content
    )

Note that the content ("<city> <state> <country>") is the actual text we analyse and put into the index. The rest of the schema properties is just stored data, which we can access again later on in our results and pass on to our Elixir app.

Searching for Cities

Let’s now implement the function for making a search request:

# lib/python/geolite2.py

from whoosh.qparser import QueryParser
from whoosh.query import Prefix

# ...

def search(query, count=10):
    """ Searches for the given query and returns `count` results """

    index = open_dir(index_path(CITY_INDEX))

    with index.searcher() as searcher:
        parser = QueryParser('content', index.schema, termclass=Prefix)
        parsed_query = parser.parse(query)
        results = searcher.search(parsed_query, limit=count)

        data = [[result['city'],
                 result['state'],
                 result['country']]
                for result in results]

        return data

First, we get the city index that we created with create_index(). Then we build an instance of whoosh’s QueryParser in order to parse our query using our city schema. We use the termclass=Prefix here to only match documents that contain any term that starts with the given query text (see the whoosh.query.Prefix docs).

The parsed query is then passed to a searcher which finally runs the search and compiles the results for us. In order to keep it simple we collect the needed data in a list of lists. This will be the data we are going to receive from our Elixir function in a moment.

The Elixir part

Our Elixir API will look pretty much the same as the Python API:

# /lib/elixir_python/geolite2.ex

defmodule ElixirPython.GeoLite2 do
  def create_index do
    # We will add code here in some more minutes...
  end

  def search(query, count \\ 10) do
    # We will add some code here later...
  end
end

Calling Python

To prepare for calling our Python functions from Elixir, add a python_call/3 function to the ElixirPython module. It creates a Python instance for us and runs the Python code we provide it with.

# lib/elixir_python.ex

defmodule ElixirPython do
  use Export.Python

  @python_dir "lib/python" # <-- this is the dir we created before

  def python_call(file, function, args \\ []) do
    {:ok, py} = Python.start(python_path: Path.expand(@python_dir))
    Python.call(py, file, function, args)
  end
end

We make use of Export’s Python module. Python.start/1 returns a tuple including a Python instance. In order to pick up our modules we pass the path to our Python directory as base path. Python.call/4 takes care of calling the given Python function from the respective module file.

Creating the City Index

We use the python_call/3 function we just defined to run the create_index function in Python:

# lib/elixir_python/geolite2.ex

defmodule ElixirPython.GeoLite2 do
  import ElixirPython, only: [python_call: 2]

  @python_module "geolite2"

  def create_index do
    python_call(@python_module, "create_index")
  end

  # ...
end

Searching for Cities

To run a search query we use our python_call/3 function again to call the Pythonsearch function we defined. The returned value is a list of lists holding the stored index data. We just loop over it and create Maps from it:

# lib/elixir_python/geolite2.ex

defmodule ElixirPython.GeoLite2 do
  import ElixirPython, only: [python_call: 2, python_call: 3]

  @python_module "geolite2"

  # ...

  def search(query, count \\ 10) do
    results = python_call(@python_module, "search", [query, count])

    for [city, state, country] <- results do
      %{city: "#{city}", state: "#{state}", country: "#{country}"}
    end
  end

Running a Search

And we are done with our hunt for a city search index and we can use it now. Make sure you activated your Python virturalenv, then open up iex and give it a try:

source venv/bin/activate
iex -S mix
iex(1)> ElixirPython.GeoLite2.create_index()

⌛️

iex(2)> ElixirPython.GeoLite2.search("Berlin", 3)
[%{city: "Berlin", country: "Germany", state: "Land Berlin"},
 %{city: "Berlingen", country: "Belgium", state: "Flanders"},
 %{city: "Falkenberg", country: "Germany", state: "Land Berlin"}]

Yay. It works!

Let’s try another one:

iex(3)> ElixirPython.GeoLite2.search("San José")
[]

Hm, why is that? We couldn’t find any results, although San José is definitely in the index. It’s because our system does not normalise special characters and accents yet. Let’s do this in a final next step.

Handling Special Characters

On the Elixir side this is easy to do. There is an Erlang lib called iconv. Let’s just add it to our mix.exs:

# mix.exs

#...
defp deps do
  [{:export, "~> 0.1.0"},
   {:iconv, "~> 1.0"}]
end

Then install the dependency with:

mix deps.get

Let’s now preprocess the query before we pass it to our Python function:

# lib/elixir_python/geolite2.ex

defmodule ElixirPython.GeoLite2 do
  # ...

  def search(query, count \\ 10) do
    query = clean_text(query)
    # ...
  end

  defp clean_text(text) do
    :iconv.convert("utf-8", "ascii//translit", text)
  end
end

When we rerun our query now we get the wanted city in the results:

iex(4)> ElixirPython.GeoLite2.search("San José")
[%{city: "San José", country: "Costa Rica", state: "Provincia de San Jose"},
# ...
]

We still have some problems with e.g. German cities, like Görlitz, that use Umlauts, So let’s transform them to their ASCII counterparts before creating the index:

# lib/python/geolite2.py

import unicodedata

# ...

def _add_document(row, writer):
    """ Writes the data to the index """

    # ...

    # clean up the content that goes to the index by using _cleanup_text():
    content = _cleanup_text(' '.join([city, state, country]))

    writer.add_document(
        city=city,
        state=state,
        country=country,
        content=content
    )

def _cleanup_text(text):
    """ Removes accents and replaces umlauts """

    replaces = [
        ['ä', 'ae'],
        ['ö', 'oe'],
        ['ü', 'ue'],
        ['Ä', 'Ae'],
        ['Ö', 'Oe'],
        ['Ü', 'Ue'],
        ['ß', 'ss']
    ]

    for original, replacement in replaces:
        text = text.replace(original, replacement)

    text = unicodedata.normalize('NFKD', text)
    text = ''.join([char for char in text if not unicodedata.combining(char)])
    text = text.encode('ascii', 'ignore').decode('ascii')

    return text

If we recreate our index and run a Görlitz query now, we will get some fitting results.

Wrapping Up

We managed to build a small local fulltext index for a city search without too much effort. Our system returns great search results and we allowed to search for cities with or without using special characters.

All in all, it does not scale very well, though. I tried to build another, more sophisticated city index including about 4.4 M cities and villages world-wide and the location coordinates (latitude and longitude) and time zone for each place.

If you are interested you can find the script for combining the city data and location data in this gist. It took quite some time to build the index (about 40 minutes on my laptop) and resulted in an index file of 1.3 GB size (compared to ~29 MB for the GeoLite2 index).

Although it also worked well and you will get fitting search results, it takes about 10 seconds to finish a single request. This approach would need some additional caching and further optimisation in order to be useful in any kind of way.

So, eventually, I ended up using the Google Places API anyway 😉. But, hey: “Wieder was gelernt.”

Adoptoposs.org – Find Co-maintainers For Your Open Source Project

Paul Götze — Mon, 18 May 2020 23:12:26 +0000

I am happy to announce Adoptoposs.org, an open source app that connects open source software maintainers with people who want to help keep projects and maintainers healthy in the long term.

Adoptoposs offers a platform to put your open source projects up for adoption and make initial contact with potential co-maintainers.

The source code is available on GitHub:

adoptoposs / adoptoposs

Finding co-maintainers for your open source software project.

If you’d like to learn more about the reasons why I built Adoptoposs, the problem of unmaintained open source software, and what might be part of the solution, then read on.

Open source software (OSS) incontrovertibly runs our world. Enterprise open source is considered to continue to grow in 2020. Most of the internet is running on open source software and millions of developers build and maintain hundreds of thousands of open source software packages in more than 250 programming languages these days.

The Problem of Unmaintained OSS Projects

Whereas most developers use and contribute to open source software at work, the maintenance of a great amount of OSS projects is done by volunteers in their spare time.

Many projects grow out of a personal side-project into popular and widely used libraries. With a lot of attraction, more issues and pull requests are opened by the community. A maintainer’s growing responsibility is now to orchestrate incoming requests and changes, and steer the whole project.

There are loads of stories about the challenges and struggles of OSS maintainers. They include emotionally exhausting community management and the great effort that has to be put into triaging of issues and pull requests.

When hearing these stories of mostly voluntary efforts, it’s no surprise that maintainers feel overwhelmed by the amount of work and turn their back on their projects.

Reasons for dedicating less time to or even leaving an open source project are many. Maintainers leave their company or lose interest. Changes in their personal lives give them less time to take care of the project or they stop their activities in the open source scene entirely because of burnout or illness. In the worst case, they have passed away.

All these cases are leaving behind projects that often have no one but the original author with administration rights or access to the publishing accounts.

Bug reports stack up, pull requests don’t get merged, compatibility issues with the surrounding ecosystem emerge. Then, at some point, someone in the community steps up and opens the infamous “Is this project abandoned?” Issue.

Eventually, the project might be forked and fixes are being made by new volunteers from the community. New versions of the project are published under a new name because of lacking publishing access.

All this leads to confusion about the state of the project and whether it can and should still be used and considered as a stable piece of software.

So, What’s the Solution?

Simply put, each and every open source project of considerable popularity should have a team of maintainers.

What does “considerable popularity” mean? When do you know your project is an important dependency for others? You will probably know really soon by its download rates and the popularity of its direct dependents.

However, if it already gained popularity and you are still operating the project on a single-maintainer policy then maintenance issues are likely to loom on the horizon sooner rather than later.

Having a team of multiple maintainers, however, is likely to result in a better-maintained project for several good reasons.

1. The workload can be split up between all maintainers

Every one of us has different preferences on what to work on, different interests, and skills. And probably none of us is highly skilled and interested in working on all the tasks that open source maintenance demands.

Thus, it makes sense to build a team and split up the entire workload to let every maintainer work on the part they like and can do best.

Less workload will also mean less risk to burn out and will allow maintainers to better look after themselves. Just like what teams in a company are built for.

2. It’s easier to make decisions as a team

Not only does it take the pressure off every maintainer when decisions have to be made. It also potentially leads to more stability and more informed decisions.

Different points of view are taken into account by default. As the old saying goes: “A fresh pair of eyes prevents many bugs’ rise.”

3. There will be no single point of failure

In our daily lives of software development, we aim at reaching availability and fault tolerance by introducing redundancy. Our clusters consist of redundant servers, we use Git to decentralize repositories, and we (hopefully) create backups of our personal documents.

All of these measures try to prevent a single point of failure. By working in a team of co-maintainers, where multiple people have full access to the project, this single point of failure can be extinguished.

Communication with the community can happen, pull requests can be merged, releases can be published, and access can be granted even if one or more co-maintainers are not available at times.

4. It is possible to step down as a maintainer

It’s important to allow maintainers to leave the project when they decide to do so. If you leave your projects as a solo maintainer then it is abandoned. Unmaintained at a moment’s notice.

The community has to put extra effort into keeping it alive or bringing it to life again. If nobody has administration and publishing rights, this process will even get harder.

Having a team of co-maintainers allows you to think about how to deal with someone leaving the project and how to onboard new co-maintainers.

When having multiple maintainers in place, if someone wants to leave the project, they can just do that — without inevitably putting the project into an unmaintained state.

What Can I Do?

As a maintainer of open source software, you should remove the single point of failure of your project by building a team of co-maintainers. Granting multiple people full access to the project will ensure that there is always more than a single person who can administrate the project.

It doesn’t stop at giving access to the repository or GitHub organization, though. You should also give your co-maintainers push access to package registries and all other related accounts in the publishing process of your project.

Co-maintainers do not necessarily need to be people who are working a lot on the project. They might just be the one in charge in case of an emergency. This can also involve giving new maintainers access when former maintainers want to step down. Similar to a custodian.

If you want to leave a project, you should communicate that clearly and inform the involved people and the community. Try to educate people about your reasons and how they can help with maintaining the project.

Don’t forget to share and document knowledge while working on the project, so that if someone leaves, there is little to no unknowns for the rest of the team.

You can use community-driven projects that help with maintaining open source software, such as the wonderful CodeTriage project, collaborative communities like Jazzband (read an interview on opensource.com), and programs like Adopted Ember Addons.

And now there’s Adoptoposs, too, hopefully helping with keeping your projects and yourself healthy while maintaining open source software.
Check it out, and happy maintaining.

By the way — Adoptoposs is always happy to hear about your feedback. It’s built with Elixir, Phoenix LiveView, and Tailwind CSS, and is currently hosted on Heroku. If you want to help, don’t hesitate to check our open issues and discuss new features with us. Issues can be reported in the GitHub repository.

References

Here are some insightful links and references that I came across when doing research for building Adoptoposs and writing this post.

nayafia / awesome-maintainers

Talks, blog posts, and interviews about the experience of being an open source maintainer

Situations where projects could not be maintained anymore

A widely used Python package, where a former maintainer left the company and was off-boarded from the team, had no access anymore, and could not merge existing PRs with bugfixes:

Fix for Python 3.7 #243

hb-alexbotello posted on Jul 26, 2019

Fixes #233

According to PEP 479, if raise StopIteration occurs directly in a generator, simply replace it with return.

This is both backwards and forwards compatible code.

View on GitHub

Another Python package, flask-security, that was forked multiple times and is now maintained by Chris Wagner and lives on as Flask-Security-Too:

Flask-Middleware / flask-security

Quick and simple security for Flask applications

Flask-Security

Quickly add security features to your Flask application.

Notes on this repo

This is a independently maintained version of Flask-Security based on the 3.0.0 version of the Original

Goals

Regain momentum for this critical piece of the Flask eco-system. To that end the the plan is to put out small, frequent releases starting with pulling the simplest and most obvious changes that have already been vetted in the upstream version, as well as other pull requests. This was completed with the June 29 2019 3.2.0 release.
Continue work to get Flask-Security to be usable from Single Page Applications such as those built with Vue and Angular, that have no html forms. This is true as of the 3.3.0 release.
Use OWASP to guide best practice and default configurations.
Be more opinionated and 'batteries' included by reducing reliance on abandoned projects and bundling in support for common use cases.
Follow…

View on GitHub

Ryan Bates, the creator of railscasts.com, paused software development because of burnout. His CanCan Ruby gem lives on as CanCanCan:

NOTICE: cancan is moving! #994

codyolsen posted on Feb 21, 2014

"parting is such sweet sorrow"

As discussed for the last month in the comment thread of pull https://github.com/ryanb/cancan/pull/989 it is time to move cancan forward before it fizzles. While the community support behind this gem is high, and pull requests have been added daily, the last merge into the official gem was in September of 2013. We both love and support the work that @ryanb has given to this gem, and in hopes to keep his legacy alive we have the following solution:

Enter `cancancan`

This will be the new gem repository going forward. Please put issues and pull requests in the new repo, as we will now be able to merge fixes into the main gem. Thanks to @bryanrite, as he has spearheaded the effort for the new repo.

For more details on the transition, please look at the readme on the new repo and the comment thread.

View on GitHub

Michael Kessler (netzpirat), the maintainer of the haml-coffee Ruby gem, passed away:

Attention: @netzpirat passed away in April '14 #97

janv posted on May 05, 2014

I, together with @sebastiandeutsch were the first committers on the project, before handing it over to Michael (@netzpirat).

I have just learned that Michael has passed away in the beginning of April. I'm putting up this issue to bring this to the attention of HAML-Coffee users.

This also means that haml-coffee needs a new maintainer. Unfortunately my free time doesn't allow for maintenance of this project in the extent Michael worked on HAML-Coffee.

</div>
<div class="gh-btn-container"><a class="gh-btn" href="https://github.com/netzpirat/haml-coffee/issues/97">View on GitHub</a></div>

DEV Community: Paul Götze

The Of a Dropdown

Let’s Spice It Up 🌶️

Hiding The Disclosure Widget

Detaching The Details

Closing The Menu

Conclusion

Testing your JSON API in Ruby with dry-rb

There’s Different Ways

Validating JSON Schemas with an RSpec Matcher

What is dry-rb?

A Simple JSON Response

A More Complex JSON Response

Some Assertion Syntactical Sugar 🍬

Conclusion

Experiencing The Stroop Effect With a Ruby CLI

Congruent? Incongruent?

Building a City Search with Elixir and Python

Setting Up a New mix Project & Python virtualenv

The Python Part

Indexing the City Data

Searching for Cities

The Elixir part

Calling Python

Creating the City Index

Searching for Cities

Running a Search

Handling Special Characters

Wrapping Up

Adoptoposs.org – Find Co-maintainers For Your Open Source Project

adoptoposs / adoptoposs

Finding co-maintainers for your open source software project.

The Problem of Unmaintained OSS Projects

So, What’s the Solution?

1. The workload can be split up between all maintainers

2. It’s easier to make decisions as a team

3. There will be no single point of failure

4. It is possible to step down as a maintainer

What Can I Do?

References

nayafia / awesome-maintainers

Talks, blog posts, and interviews about the experience of being an open source maintainer

Situations where projects could not be maintained anymore

Fix for Python 3.7 #243

Flask-Middleware / flask-security

Quick and simple security for Flask applications

Flask-Security

Notes on this repo

Goals

NOTICE: cancan is moving! #994

Enter cancancan

Attention: @netzpirat passed away in April '14 #97

The
Of a Dropdown

Enter `cancancan`