DEV Community: Matthieu SB

Self-Hosting Wikipedia (and other sites ...) with Kiwix

Matthieu SB — Thu, 13 Mar 2025 13:16:47 +0000

In this article we'll be deploying Kiwix using docker and docker compose. You can look at the github repositories and contribute to its many projects (server, mobile, browser and desktop apps).

Here is an example of what it looks like:

Disclaimer

ℹ️ This article is helpful if you want to self-host an OFFLINE version of different sites (Wikipedia, Wikitionnary, Stack Exchange, DevDocs ...) in your home, or on a portable server to bring it with you (It should work on a Raspberry-Pi).

⚠️ This article is not for you if you want to host and edit your own version of Wikipedia, but there should be resources on MediaWiki out there. You also don't need this if you just want to read .zim files directly on your local computer or phone, kiwix client apps are largely enough.

Setup Instructions

This installation was tested on ubuntu 24.04, both server and desktop, but it should work on any system with docker installed. Most of this tutorial is a step-by-step guide largely inspired by joinboiser github repository

Pre-requisites

A system with docker installed
Your favorite command line
Your favorite editor

Create the project structure and empty files:

mkdir -p kiwix/zims
mkdir -p kiwix/scripts

cd kiwix

touch Dockerfile
touch docker-compose.yml
touch scripts/provision.sh
touch scripts/entrypoint.sh

chmod +x scripts/provision.sh
chmod +x scripts/entrypoint.sh

Get static sites data

Kiwix uses .zim files, you can find a nice repository on the official kiwix downloads repository.

💡 To begin I would advise using tiny zim files to try out your setup, a full english wikipedia with images can take around 100GB and takes a while to download

For this example, we'll be downloading the following (From the kiwix folder:

wget https://download.kiwix.org/zim/devdocs/devdocs_en_ansible_2025-01.zim -O zims/devdocs-ansible-en_2025-01.zim
wget https://download.kiwix.org/zim/devdocs/devdocs_en_typescript_2025-01.zim -O zims/devdocs-typescript-en_2025-01.zim

If you still want to download wikipedia, I am currently using this version (🚨 Very long to download):

wget https://download.kiwix.org/zim/wikipedia/wikipedia_en_all_maxi_2024-01.zim -O wikipedia-en_2024-01.zim

Setting up scripts

Add the following content to the empty scripts:

provision.sh

#!/bin/bash

TGTDIR="${1:-/usr/local/bin}"

VERSION="${VERSION:-3.1.2-4}"
MACHINE="${MACHINE:-$(uname -m)}"

URL="${URL:-"https://download.kiwix.org/release/kiwix-tools/kiwix-tools_linux-${MACHINE}-${VERSION}.tar.gz"}"

DLPATH=${DLPATH:-/tmp}
FILE="$DLPATH/${URL##*/}"

# Install kiwix-tools from binary tarball
set -x -e
wget -O "$FILE" "$URL"
tar -xvf "$FILE" -C "$TGTDIR" --strip-components 1
rm -f "$FILE"

entrypoint.sh

#!/bin/bash

ZIMDIR=${ZIMDIR:-/zims}
LIBRARY_XML=/library.xml

for f in "$ZIMDIR"/*.zim; do
  if [[ -f "$f" ]]; then
    ( set -x; kiwix-manage "$LIBRARY_XML" add $f )
  fi
done

kiwix-serve --port 8080 --library "$LIBRARY_XML"

Setting up Dockerfile

Add this to your empty dockerfile:

FROM ubuntu:24.04

ENV TGTDIR=/usr/local/bin

RUN env LANG=C LC_ALL=C apt update
RUN env DEBIAN_FRONTEND=noninteractive LANG=C LC_ALL=C apt-get -y full-upgrade
RUN env DEBIAN_FRONTEND=noninteractive LANG=C LC_ALL=C apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install wget

VOLUME /zims

COPY ./scripts/*.sh $TGTDIR/

RUN provision.sh

EXPOSE 8080
ENTRYPOINT ["entrypoint.sh"]

💡 For testing purposes, you can comment the following line as it can take a while to execute: RUN env DEBIAN_FRONTEND=noninteractive LANG=C LC_ALL=C apt-get -y full-upgrade

Setting up the docker-compose file

Add this to your empty docker-compose:

services:
  kiwix:
    build: .
    container_name: kiwix
    ports:
      - '8080:8080'
    volumes:
      - ./zims:/zims:ro
    networks:
      - kiwix_net

networks:
  kiwix_net: 
    driver: bridge

Your kiwix folder should now look like this:

kiwix
├── docker-compose.yml
├── Dockerfile
├── scripts
│  ├── entrypoint.sh
│  └── provision.sh
└── zims
   ├── devdocs-ansible-en_2025-01.zim
   └── devdocs-typescript-en_2025-01.zim

Launching and testing

To launch our kiwix server, just execute:

docker compose up --build

The service should now be available at <server-hostname-or-ip>:8080. Use localhost:8080 if you deployed locally.

If you only added devdocs mentioned above, you should see something like this:

To stop the service:

docker compose stop

If you add new zim files, you just need to restart the service, a docker compose restart should suffice

[OPTIONAL] Using a systemd service

This will help you if you want to enable the service on boot and administer it through systemd.

First, create the log folders and unit file with:

sudo mkdir -p /var/log/kiwix
touch kiwix.service

Add and adapt the following to your kiwix.service file:

[Unit]
Description=Kiwix
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
User=<YOUR USER>
ExecStart=/usr/bin/docker compose -f <path-to-kiwix-service>/docker-compose.yml up --build
ExecStop=/usr/bin/docker compose -f <path-to-kiwix-service>/docker-compose.yml stop
StandardOutput=append:/var/log/kiwix/kiwix.log
StandardError=append:/var/log/kiwix/kiwix-err.log

[Install]
WantedBy=multi-user.target

Copy it to systemd configs:

sudo cp kiwix.service /etc/systemd/system

And then start and enable the systemd service:

# Reload configurations
sudo systemctl daemon-reload
# Enable start on boot and start service immediatly
sudo systemctl enable kiwix.service && sudo systemctl start kiwix.service

This should do it ! Have fun with your custom static sites.

Integration testing on existing routes with Apache Camel (and Spring and DBUnit)

Matthieu SB — Mon, 07 Aug 2017 16:02:25 +0000

Subject and Target Audience

This article is here to show how to do integration tests with Apache Camel in a Spring App, since it is not described precisely on the documentation. It can be read by both beginners or developers acquainted with Apache Camel features. Still, a bit of knowledge makes it easier to understand.

Introduction

Using this method, you will be able to launch an existing route from beginning to end (With or without you real database), intercepting the exchange between each part of the route and check if your headers or body contain correct values, or not.

I have been doing this on a classic Spring project with xml configuration and DBUnit for database mocking. Hopefully this will give you a few leads.

Setup

If it is not already done on your project, you should add testing dependencies for camel. See below for maven users :

<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-test</artifactId>
  <version>${camel.version}</version>
  <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-test-spring</artifactId>
    <version>${camel.version}</version>
     <scope>test</scope>
</dependency>

Camel route example

The following route has a simple goal :

First, it checks if and ImportDocumentProcess object is present in the database and adds it as an exchange header
Then, it adds an ImportDocumentTraitement (Which is linked to the previous ImportDocumentProcess) in the database

Here is this route's code :

@Component
public class TestExampleRoute extends SpringRouteBuilder {

    public static final String ENDPOINT_EXAMPLE = "direct:testExampleEndpoint";

    @Override
    public void configure() throws Exception {
        from(ENDPOINT_EXAMPLE).routeId("testExample")
            .bean(TestExampleProcessor.class, "getImportDocumentProcess").id("getImportDocumentProcess")
            .bean(TestExampleProcessor.class, "createImportDocumentTraitement").id("createImportDocumentTraitement")
            .to("com.pack.camel.routeshowAll=true&multiline=true");
    }

}

The id on the routes are not mandatory, you can use the bean strings afterwards too. However I think using ids can be considered a good practice, in case your route strings change in the future.

Camel processor example

The processor just contains just contains the methods needed by the route. It is just a classic Java Bean containing several methods. You can also implement Processor and override the process method.

See the code below :

@Component("testExampleProcessor")
public class TestExampleProcessor {

    private static final Logger LOGGER = LogManager.getLogger(TestExampleProcessor.class);

    @Autowired
    public ImportDocumentTraitementServiceImpl importDocumentTraitementService;

    @Autowired
    public ImportDocumentProcessDAOImpl importDocumentProcessDAO;

    @Autowired
    public ImportDocumentTraitementDAOImpl importDocumentTraitementDAO;

    // ---- Constants to name camel headers and bodies
    public static final String HEADER_ENTREPRISE = "entreprise";

    public static final String HEADER_UTILISATEUR = "utilisateur";

    public static final String HEADER_IMPORTDOCPROCESS = "importDocumentProcess";

    public void getImportDocumentProcess(@Header(HEADER_ENTREPRISE) Entreprise entreprise, Exchange exchange) {
        LOGGER.info("Entering TestExampleProcessor method : getImportDocumentProcess");

        Utilisateur utilisateur = SessionUtils.getUtilisateur();
        ImportDocumentProcess importDocumentProcess = importDocumentProcessDAO.getImportDocumentProcessByEntreprise(
                entreprise);

        exchange.getIn().setHeader(HEADER_UTILISATEUR, utilisateur);
        exchange.getIn().setHeader(HEADER_IMPORTDOCPROCESS, importDocumentProcess);
    }

    public void createImportDocumentTraitement(@Header(HEADER_ENTREPRISE) Entreprise entreprise,
            @Header(HEADER_UTILISATEUR) Utilisateur utilisateur,
            @Header(HEADER_IMPORTDOCPROCESS) ImportDocumentProcess importDocumentProcess, Exchange exchange) {
        LOGGER.info("Entering TestExampleProcessor method : createImportDocumentTraitement");

        long nbImportTraitementBefore = this.importDocumentTraitementDAO.countNumberOfImportDocumentTraitement();
        ImportDocumentTraitement importDocumentTraitement = this.importDocumentTraitementService.createImportDocumentTraitement(
                entreprise, utilisateur, importDocumentProcess, "md5_fichier_example_test", "fichier_example_test.xml");
        long nbImportTraitementAfter = this.importDocumentTraitementDAO.countNumberOfImportDocumentTraitement();

        exchange.getIn().setHeader("nbImportTraitementBefore", Long.valueOf(nbImportTraitementBefore));
        exchange.getIn().setHeader("nbImportTraitementAfter", Long.valueOf(nbImportTraitementAfter));
        exchange.getIn().setHeader("importDocumentTraitement", importDocumentTraitement);
    }
// Rest of the code contains getters and setters for imported dependencies
}

Not much to say here, except that we use the exchange to transfer objects from one part to another. This is the way it is usually done on my project, since we have really complex processes to handle.

Camel test class

The test class is going to trigger and run tests on the example route. We also use DBUNit to mock a database with some predefined values.

First, we use an abstract class in order to share common annotations between each Camel Integration test. See the code below :

@RunWith(CamelSpringRunner.class)
@BootstrapWith(CamelTestContextBootstrapper.class)
@ContextConfiguration(locations = { "classpath:/test-beans.xml" })
@DbUnitConfiguration(dataSetLoader = ReplacementDataSetLoader.class)
@TestExecutionListeners({ DependencyInjectionTestExecutionListener.class, DirtiesContextTestExecutionListener.class,
        DbUnitTestExecutionListener.class })
@DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD) // Not always relevant,  See explanation below
public abstract class AbstractCamelTI {

}

Careful not to forget any annotation or your DAOs won't be injected correctly, and that will lead to NullPointer exceptions when you run your tests. DBUnit annotations can be removed if you are using another database mocking system.

IMPORTANT NOTE : The @DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD) annotation is here to reload the camel context for EACH TEST. This way, each test method has a fresh reinitialized context. But you need to redefine the mocked endpoints for each test too. If you need to refactor a large camel processor and test each part independently, this can help. Otherwise, just remove this annotation as it has a few downsides :

The tests are not real integration tests anymore, and you have to remove and weave your route parts for each test, which can be a bit heavy
Reloading the context for each test takes a while and can significantly increase your tests overall running time. This can be frustrating in the long run

Test class configuration

@DatabaseSetup(value = { "/db_data/dao/common.xml", "/db_data/dao/dataForImportDocumentTests.xml" })
public class TestExampleProcessorTest extends AbstractCamelTI {

    @Autowired
    protected CamelContext camelContext;

    @EndpointInject(uri = "mock:catchTestEndpoint")
    protected MockEndpoint mockEndpoint;

    @Produce(uri = TestExampleRoute.ENDPOINT_EXAMPLE)
    protected ProducerTemplate template;

    @Autowired
    ImportDocumentTraitementDAO importDocumentTraitementDAO;

    // -- Variables for tests
    ImportDocumentProcess importDocumentProcess;

    @Override
    @Before
    public void setUp() throws Exception {
        super.setUp();

        importDocumentProcess = new ImportDocumentProcess();
        //specific implementation of your choice
    }
}

First test

This tests triggers the first part of the route and leads it to a mockEndpoint so that we can test if the ImportDocumentProcess has been correctly selected and put into the headers :

@Test
public void processCorrectlyObtained_getImportDocumentProcess() throws Exception {
    camelContext.getRouteDefinitions().get(0).adviceWith(camelContext, new AdviceWithRouteBuilder() {

        @Override
        public void configure() throws Exception {
            weaveById("getImportDocumentProcess").after().to(mockEndpoint);
        }
    });

    // -- Launching the route
    camelContext.start();
    template.sendBodyAndHeader(null, "entreprise", company);

    mockEndpoint.expectedMessageCount(1);
    mockEndpoint.expectedHeaderReceived(TestExampleProcessor.HEADER_UTILISATEUR, null);
    mockEndpoint.expectedHeaderReceived(TestExampleProcessor.HEADER_IMPORTDOCPROCESS, importDocumentProcess);
    mockEndpoint.assertIsSatisfied();

    camelContext.stop();
}

Second test

This tests simply tiggers the whole route :

@Test
public void traitementCorrectlyCreated_createImportDocumentTraitement() throws Exception {
    camelContext.getRouteDefinitions().get(0).adviceWith(camelContext, new AdviceWithRouteBuilder() {

        @Override
        public void configure() throws Exception {
            weaveById("createImportDocumentTraitement").after().to(mockEndpoint);
        }
    });

    // -- Launching the route
    camelContext.start();

    Exchange exchange = new DefaultExchange(camelContext);
    exchange.getIn().setHeader(TestExampleProcessor.HEADER_ENTREPRISE, company);
    exchange.getIn().setHeader(TestExampleProcessor.HEADER_UTILISATEUR, null); // No user in this case
    exchange.getIn().setHeader(TestExampleProcessor.HEADER_IMPORTDOCPROCESS, importDocumentProcess);

    long numberOfTraitementBefore = this.importDocumentTraitementDAO.countNumberOfImportDocumentTraitement();

    template.send(exchange);

    mockEndpoint.expectedMessageCount(1);
    mockEndpoint.assertIsSatisfied();

    camelContext.stop();

    long numberOfTraitementAfter = this.importDocumentTraitementDAO.countNumberOfImportDocumentTraitement();
    assertEquals(numberOfTraitementBefore + 1L, numberOfTraitementAfter);
}

Good practices

If you want to avoid wasting time on simple errors, some guidelines can be followed.

Routes ids

In this example, I use the following line to get my routes and weave my test endpoints :

camelContext.getRouteDefinitions().get(0).adviceWith(camelContext, new AdviceWithRouteBuilder() { [...] });

HOWEVER A better way would be to affect ids to your routes and use the following line to get and weave them :

camelContext.getRouteDefinition("routeId").adviceWith(camelContext, new AdviceWithRouteBuilder() {  [...]  });

This method is a bit cleaner as you don't have to manually acknowledge which route you want to test. Refactoring a route is a bit easier too, since you can change your route's strings without changing their ids.