Send records in batch to Algolia with free plan

Manon Simonin — Fri, 29 Sep 2023 09:16:08 +0000

The free plan of Algolia is more than enough for basic needs, development testing or even some little websites, internal or not. Just keep in mind you need to credit Algolia when using in production.

You get 10,000 search requests every months and can store 1 million records. But there's a restriction on the size limit of a single record which makes it difficult to add these records in the first place. The only thing reason we would go for the first priced plan is that limit being increased from 10 to 100 Kb.

In my case, I wanted to use Algolia as a search engine for Wiki.js. The main problem I faced is that I had already hundreds of pages when I decided to do so, and Wiki.js only supports indexing your entire database at once. Obviously this was too large and resulted in an error.
Once the index is built, every changes or addition is sent individually, respecting the size limit for the most part.

I just needed to build an initial index of my existing pages, so I wrote a script that sends the records in batch instead.

All you need is node.js installed, for instance in MacOS :

brew install nvm
echo 'source $(brew --prefix nvm)/nvm.sh' >> ~/.zshrc
mkdir ~/.nvm
nvm install node

Create a directory and add the following packages :
This is designed for a Postgre database, but you could make the same with a MySQL database with mysql12 and some tweaks.

npm install pg
npm install algoliasearch

Then create a file app.js and don't forget to replace your Algolia credentials as well as your database connection :

const { Client } = require('pg');
const algoliasearch = require('algoliasearch');

const dbConfig = {
    user: "username",
    password: "password",
    host: "localhost",
    port: 5432,
    database: "db-name",
  };

  const algoliaConfig = {
      appId: 'appID',
      apiKey: 'apiKey',
      indexName: 'index-name',
  }

const client = new Client(dbConfig);

client.connect()
  .then(async () => {
    const query = `
      SELECT title, path, content, description, LENGTH(content) / 1024 AS size_kb
      FROM pages
    `;

    const result = await client.query(query);

    const algoliaClient = algoliasearch(algoliaConfig.appId, algoliaConfig.apiKey);

    const index = algoliaClient.initIndex(algoliaConfig.indexName);

    const sendRecordToAlgolia = async (record) => {
      try {
        await index.saveObject(record);
        console.log(`Record sent to Algolia with objectID: ${record.objectID}`);
      } catch (error) {
        console.error('Error sending record to Algolia:', error);
      }
    };

    for (const row of result.rows) {
        if (row.size_kb.toFixed(2) < 10) {
            const record = {
                objectID: row.path,
                path: row.path,
                title: row.title,
                content: row.content,
                description: row.description || '',
                locale: 'fr',
              };
              await sendRecordToAlgolia(record);
        }
    }
  })
  .catch((error) => {
    console.error('Error:', error);
  })
  .finally(() => {
    client.end();
  });

Run the script :

node app.js

You should then be able to find those records on Algolia, don't forget to check if some pages were too large with a SQL query for example. In my case I only had 2, so I just split them and edited them manually.

How to use Docker to work locally on WordPress websites with different PHP versions in MacOS

Manon Simonin — Fri, 01 Apr 2022 11:53:53 +0000

This article will help you to use Docker mainly for local development.

Maybe you are handling a WordPress website all by yourself, and don’t have a staging website to try all sorts of updates and plugins. This could be a great, easy and free alternative.

Download Docker

First of all, you need to download Docker Desktop, which will come with a fresh installation of Docker as well as a great UI to keep track of things without having to use too much command lines. But as we will need some, you can prepare yourself by opening the terminal, either by querying terminal.app in the search bar or from the launchpad.

Download Docker Desktop

When this is all set up, you can check that Docker is installed with this command line :

docker --version

You will get something like this :

Docker version 20.10.12, build e91ed57

Get a copy of your website

Before configuring Docker, you need to get a copy of your website on your computer.

The entire sources
An export of the database

This is the part where you might ask your web hoster to do this for you, or maybe use one WordPress plugin to create backup, I won’t focus on this.

Configure Docker for your local website

First of all, you need to create a new file to your WordPress repository.

This file will be at the root of the folder and named : docker-compose.yml

Basically, this file is used to create a container which will put different services and volumes together, in this particular case we need WordPress and MySQL.

WordPress will be fetch using an official image, it means it will execute some code on the container that was created by WordPress itself. Thanks to that, our job gets much easier because it already comes with everything that WordPress needs to work, including PHP.

MySQL is also an image, but as we need its data to be persisting we’ll put it in a volume. This is the purpose of a database, but volumes can be used for other things, for example logs.

Let’s that a closer look at everything that will be in the file :

version: "3.8"

This is the version of the Compose file, we will not mess with this and just get the latest as of March 2022 : https://docs.docker.com/compose/compose-file/compose-versioning/

services:
    wordpress:
        container_name: my-website
        image: wordpress
        restart: always
        ports:
          - 8001:80
        environment:
          WORDPRESS_DB_HOST: db
          WORDPRESS_DB_USER: my-user
          WORDPRESS_DB_PASSWORD: my-password
          WORDPRESS_DB_NAME: my-website
        volumes:
          - ./:/var/www/html

container_name : if you do not name you container_name, it will be randomize. But this is useful to find in the desktop app or to target it from command lines.
image : a lot of official images can be found on the Docker Hub like WordPress :https://hub.docker.com/_/wordpress
restart : just making sure the container stays up in an error comes.
ports : here you just bind the client host (80) to your own host (8001), this means your website will be on at http://localhost:8001
environment: here you can set some global variables that can be useful, for instance we set everything related to our database, but as it’s also set in the wp-config.php file, I’d say it’s not that important. Just keep in mind the database host because it’s particular. The full list of available environments can be found on the image on Docker Hub.
volumes : this indicate that the sources are in the current folder (./), and Docker maps this with its path /var/www/html, so that our sources are used for the container.

 db:
    container_name: db-wp
    image: mysql:5.7
    ports:
    - 3307:3306
    restart: always
    environment:
      MYSQL_DATABASE: my-website
      MYSQL_USER: my-user
      MYSQL_PASSWORD: my-password
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    volumes:
      - db:/var/lib/mysql

image : Official image for MySQL 5.7, amongst other that you can find here : https://hub.docker.com/_/mysql. Also, if your database isn’t in MySQL, this is doable with other database management systems like MariaDB, Postgre...
ports: you don’t actually need to specify this, the database will be created and stored, but I personally choose to show the port so that I can connect to it using TablePlus.
environment : same as WordPress but with specific environments for MySQL
volumes : again, this maps our own database with the one from the WordPress image.

You also need to set the volumes outside the services :

volumes:
  db:

Remember the database host “db” ? Here it is. We just named it. Now we can use it in wp-config.php and Docker will resolve it with an IP address.

That being said, this is what our file looks like with everything we discussed :

/docker-compose.yml

version: "3.8"

services:
  wordpress:
    container_name: my-website
    image: wordpress
    restart: always
    ports:
      - 8001:80
    volumes:
      - ./:/var/www/html
 db:
    container_name: db-wp
    image: mysql:5.7
    ports:
    - 3307:3306
    restart: always
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    volumes:
      - db:/var/lib/mysql

volumes:
  db:

Now, our website is ready.

The last step is to launch everything, so that Docker builds the local website. You need to go to the root of your website from the terminal, probably something like that :

cd /Users/me/my-website

And then the final command which will mount everything :

docker-compose up --build

The website can be accessed from a browser at http://localhost:8001 (or any port you mentioned)

Now let’s get a little bit further.

WordPress is written in PHP. It comes in many versions, you might not have the latest installed and want precisely to test an update. Same goes for MySQL even if it’s less often.

WordPress official Docker images allow us to mess with the versions.

You can try the following images to alter the PHP version :

wordpress:php7.4-fpm
wordpress:php7.3
wordpress:php7.2
wordpress:php7.1
wordpress:php7.0
wordpress:php5.6

To end this, just know that we can do anything with Docker. That includes starting from scratch and recreating the WordPress image, but that requires a lot of stuff that comes prebuilt here, like a nginx or Apache server, many PHP extensions... If we can, let's stick with official images.

This article is an extend to this introduction, published on the website of my agency (French) : https://www.pix-associates.com/blog/docker-definition-et-usages-on-vous-en-dit-plus

DEV Community: Manon Simonin

Send records in batch to Algolia with free plan

How to use Docker to work locally on WordPress websites with different PHP versions in MacOS

Download Docker

Get a copy of your website

Configure Docker for your local website