DEV Community: Perry Janssen

Fix slow Docker databases for Apple Silicon M1 chips

Perry Janssen — Thu, 04 Mar 2021 21:23:26 +0000

Reading and writing to databases in Docker on Apple Silicon laptops is currently quite slow. Dealing with large databases can be hard or even impossible to work with. While Docker and Apple are working to fix this issue, we need a temporary solution for our local environments. Luckily, running databases outside of Docker containers isn't slow at all!

In this little tutorial we'll be creating a database on our localhost and connecting a Drupal installation inside a Docker container to it.

Homebrew, MariaDB and another terminal

In this example we'll be setting up a MariaDB database. MariaDB can be installed using Homebrew, but MariaDB isn't compatible with the new M1 chip yet, so we will install it using a Rosetta version of Homebrew.

Rosetta Terminal

First, we'll need to create a new terminal application to run our commands using "x86_64 instructions", which emulates the Intel processors of previous Macs. The emulator is called Rosetta 2. You can run programs in this mode, but also create applications that always start up using Rosetta.

In Finder, navigate to your Applications folder.
Inside the Utilities folder, duplicate the Terminal application and call it something like "Rosetta Terminal".
Click on "Get Info" on the newly created Application.
In the General tab, check the box that says "Open using Rosetta".

You can change the application icon by copying an image, clicking on the small icon at the top of the "Get Info" window, and pasting. This way you can tell the two terminal applications apart. Check sites like www.flaticon.com for free icons.

You can also change the background color of the terminal in the application's Preferences > Profiles, so you always know which terminal you've got in front of you.

Now all commands executed in this terminal are executed using the Intel architecture. Now let's install Homebrew.

Rosetta Brew

Install Homebrew using this command:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

This will install the brew command in /usr/local, which is a different location than the M1 Homebrew installation. This means we've got two Homebrews, so we need to differentiate between the two by creating an alias command for the new installation.

You can do this by running: alias ibrew='arch -x86_64 /usr/local/bin/brew'

The arch -x86_64 part makes sure the command is run using Rosetta, even if you use it outside of the Rosetta Terminal, but I advise to just always use the Rosetta Terminal when working with x86_64 packages.

If you run ibrew, it will run the Rosetta brew command 🎉

As you will probably bump into more incompatible brew packages in you developing life, you will get a lot of use out of our ibrew command.

If you don't want to run the alias command every time you want to use the ibrew command, put the line in your terminal profile. For the ZSH shell this is ~/.zshrc.

MariaDB

Install MariaDB in your Rosetta Terminal.

Run brew install mariadb and let the installation complete.
Run brew services start mariadb to start the MySQL database.
Run sudo mysql to log into MySQL as the root user.
Run CREATE DATABASE db_name; to create a database.

Now we have a database service and created a database which we can connect to. But before we can connect to it outside of our terminal, we have to create a user.

When still in the MySQL command line:

Run CREATE USER 'username'@'%' IDENTIFIED BY 'password'; to create a user with username "username" and password "password" for all hosts. Replace username and password with you desired credentials.
Run GRANT ALL PRIVILEGES ON *.* TO 'username'@'%' IDENTIFIED BY 'password'; to grant all permissions to the user, so it can read and write to the database.

The percent sign '%' after the username makes sure the user can log in from any host. Setting this to 'localhost' will only make the databases accessible from the localhost domain, for that user. You want to specify this host in production environments for security reasons, but locally I advise using the % wildcard, especially because we want to connect to it from out Docker container.

Now we have set up everything to connect to our database, so let's do that now!

Connecting to an "external" database

When inside a Docker container, you cannot reach the localhost of you machine by just using localhost or 127.0.0.1 as the host. Luckily Docker has made a couple of hosts we can use that proxy to our localhost 😄

There are multiple hosts for some reason. One or all might work for you, just try them:

host.docker.internal
gateway.docker.internal
docker.for.mac.host.internal

In Drupal 8/9 the database config looks something like this:

$databases['default']['default'] = [
  'driver' => 'mysql',
  'database' => 'db_name',
  'username' => 'username',
  'password' => 'password',
  'host' => 'host.docker.internal',
  'port' => 3306,
];

You should be able to connect to your external database now!

Download Sequel Pro to make creating and maintaining databases a little more visual.

While this isn't a perfect solution, it actually works better than I expected, at least for the project I'm working on now! For small databases you can still use database services inside Docker, but if you're talking huge, content-filled Drupal or Wordpress databases, you will have a hard time not using an external database like this. Let's hope Docker and/or Apple will fix this very soon!

Getting random tracks using the Spotify API

Perry Janssen — Thu, 20 Jun 2019 14:05:21 +0000

While the Spotify API offers quite a lot of ways to interact with one of the most used streaming services, it lacks the ability of accessing all Spotify tracks as a whole. This might be because of the enormous amount of available songs. Grabbing a list of all wouldn’t be very efficient. While I was creating an application with which a user can discover new music, I wanted to show the user seemingly random tracks using Spotify. In this article, I’ll go over a couple of ways I tried to achieve this.

This is not a tutorial on the basics of the Spotify API, but on kind of a obscure, specific issue. Check out the ‘Get Started’ guide: https://developer.spotify.com/documentation/web-api/quick-start/

Developing using (Node)JS? This module really made connecting with Spotify and especially oAuth, a lot easier: https://www.npmjs.com/package/spotify-web-api-js

Genres

The first method I tried made use of Spotify’s genre system. By fetching a list of all genres, picking a random genre and getting a track by that genre would result in a random track, or so I thought. First, Spotify doesn’t assign a genre to each track. A genre is linked to an artist, instead. Secondly, the list of available genres does not match the genres linked to the artists.

According to the list of ‘available genre seeds’ only contain about one- or two hundred genres. These genres are all very general. The genres linked to artist accounts are usually more specific. This makes it impossible to search all artists, and therefore all tracks, by using genres.

This problem has already been noticed by other developers. You can check out the issue on GitHub: https://github.com/spotify/web-api/issues/868

Search

The second method I tried made use of a simple search query. Using Spotify’s Search API, you can search through all available tracks, artists, playlist and more. This method worked best for me. According to Spotify’s documentation, a wildcard character can be used to search. A wildcard is a character that can be anything. For instance, a search query like this: My % will return all tracks that begin with the word My.

Now I had to think of a way of creating random searches, and getting a random track from that search. The best way I could think of was having a random character prefixing the wildcard character, like this: d%. This would return all tracks that start with a d. You could also put the wildcard at the beginning, or at both the beginning or end.

Because Spotify shows the more famous artists and tracks at the top of the search results, I also had to randomize an offset. According to documentation, a maximum limit of 10.000 can be applied. The offset will be the index of the first returned item, or the first returned page if a limit is set.

Check out the Spotify Search documentation: https://developer.spotify.com/documentation/web-api/reference/search/search/

Getting a random search query

To get a random search query, you’ll need to grab a random character from a list of characters and add the wildcard character to the string. I created a simple function that would do just that:

function getRandomSearch() {
  // A list of all characters that can be chosen.
  const characters = 'abcdefghijklmnopqrstuvwxyz';

  // Gets a random character from the characters string.
  const randomCharacter = characters.charAt(Math.floor(Math.random() * characters.length));
  let randomSearch = '';

  // Places the wildcard character at the beginning, or both beginning and end, randomly.
  switch (Math.round(Math.random())) {
    case 0:
      randomSearch = randomCharacter + '%';
      break;
    case 1:
      randomSearch = '%' + randomCharacter + '%';
      break;
  }

  return randomSearch;
}

To create a random offset, just generate a random number between 0–10.000:

const randomOffset = Math.floor(Math.random() * 10000);

The Spotify Search endpoint allows you to filter on different types. To get tracks, just add type=track to the request.

Now that we have all the required parameters, we can make a GET request to the Spotify Search endpoint, like this:

GET https://api.spotify.com/v1/search
type=track
q=getRandomSearch()
offset=randomOffset

Locality issue

To use the Spotify API, Spotify requires the user to be logged in. A logged in user is linked to a certain location. This means that searching, even through the API, can be a little biased towards tracks originating from the user’s country. There isn’t really a way around this, but it is possible to define a market. A market is, kind of, the country the user is from. You can set the market the same way as any other parameter, but setting a market means you’ll only get results from artists from that specific country. Of course, you could randomize this as well.