DEV Community: Oshan Wisumperuma

Elasticsearch Tune for indexing speed

Oshan Wisumperuma — Wed, 18 Nov 2020 05:27:59 +0000

Increase the refresh interval and disable replicas

On some projects, indexing happens at a different pace, not while the application is in use. In my current project, we follow a similar approach. So I'm disabling the refresh interval because we are not going to search while indexing is on. Also, I'll enable replication when all the indexing complete.

curl --location --request PUT 'localhost:9200/<index>/_settings' \
--header 'Content-Type: application/json' \
--data-raw '{
    "settings": {
        "index": {
            "number_of_replicas": "0",
            "refresh_interval": "600s"
        }
    }
}'

call GET endpoint to confirm the new variables.

curl --location --request GET 'localhost:9200/<index>/_settings'

collecting these values initially will be helpful, so you can reset the configurations after indexing is complete.

bulk insert

adjust the batch size of the elasticsearch-model import function argument.

API: https://rubydoc.info/gems/elasticsearch-model/Elasticsearch/Model/Importing/ClassMethods#import-instance_method

more tips can be found from here: https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html#tune-for-indexing-speed

How to create faster smaller swap space

Oshan Wisumperuma — Sat, 17 Oct 2020 19:37:00 +0000

Even though swap is not necessary for the most of use cases, but developers might need swap space when doing a big compilation, browsing on chrome or on slack. but it is a really painful experience using a swap partition on Disk space. ZRAM helps us allocate small storage from RAM as swap space and store 3,4 times of data on that space. its even faster than the SSD swap partitions because this is a ram drive.

Since zstd has an excellent compression ratio, 2GB of zram space will have total swap capacity around 6GB. please note that these commands are only tested on Fedora 32. All the changes will be lost in the next reboot.

$ modprobe zram num_devices=2
$ echo zstd > /sys/block/zram0/comp_algorithm
$ echo 2G > /sys/block/zram0/disksize
$ mkswap --label zram0 /dev/zram0

enable the swap partition with higher priority

$ swapon --priority 1 /dev/zram0

Tadaa! 🎉 now we can open another chrome tab ;-)

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           15Gi       7.3Gi       1.8Gi       1.5Gi       6.4Gi       6.4Gi
Swap:         2.0Gi       5.0Mi       2.0Gi

to disable the swap storage

$ swapoff /dev/zram0

update: zram is enabled by default