DEV Community

Cover image for Tracking Optuna Experiments with JournalStorage and Optuna Dashboard
Haruki Niwa
Haruki Niwa

Posted on Edited on AI-assisted

Tracking Optuna Experiments with JournalStorage and Optuna Dashboard

#optuna #python #machinelearning #tutorial

Keeping track of hyperparameter optimization becomes more difficult when multiple processes or machines contribute trials to the same experiment. Each worker needs access to a consistent study history, and the resulting trials need to be easy to inspect without manually parsing logs.

Optuna addresses these needs with persistent storage backends and Optuna Dashboard. JournalStorage records changes to study and trial state in an operation log, while Optuna Dashboard provides a browser interface for inspecting that history. Together, they offer a practical workflow for tracking experiments without writing custom visualization code.

This guide combines my notes on JournalStorage and Optuna Dashboard, with a small illustrative Python example added to make the workflow reproducible. The original notes discuss shared NFS/NAS storage in a research computing environment. For this article, the toy script's study creation and journal reloading were checked in an isolated environment using Optuna 4.8.0. The dashboard interface and multi-node NFS behavior have not been newly tested, and no performance benchmarking was conducted.

1. How JournalStorage Works

Optuna separates an optimization study from the storage that persists its state. A study contains trials, suggested hyperparameters, objective values, and execution states.

The commonly used RDBStorage stores this information in relational database tables, using databases such as PostgreSQL, MySQL, or SQLite. JournalStorage takes a different approach: it persists a sequence of operations that can be replayed to reconstruct the current state.

For example, the journal records events corresponding to operations such as:

  • Creating a study or a trial.
  • Recording suggested hyperparameter values.
  • Recording an objective value.
  • Changing a trial's state to COMPLETE, PRUNED, or FAIL.

These are conceptual examples of operations, not literal event names or lines copied from an Optuna journal file. JournalStorage keeps its reconstructed state in memory and can rebuild it by replaying saved operations after a process restart.

For file-based storage, the two main classes are:

from optuna.storages import JournalStorage
from optuna.storages.journal import JournalFileBackend

storage = JournalStorage(JournalFileBackend("optuna_journal.log"))
Enter fullscreen mode Exit fullscreen mode

JournalStorage manages study state; JournalFileBackend reads and writes the underlying journal file. The API shown here follows the names used in Optuna 4.0 and later.

2. JournalStorage vs. RDBStorage

Consideration JournalStorage with a file backend RDBStorage
Persistence Append-only operation log Relational database tables
Additional database service Not required for a file backend Required for PostgreSQL/MySQL; not for SQLite
Study access Optuna's storage API Optuna's storage API, plus database-specific tools
Direct SQL queries Not available on the journal file Available
Concurrency File locking and storage behavior matter Depends on the database and configuration
Scaling to multiple machines Requires careful filesystem/locking validation PostgreSQL/MySQL are generally the safer default

A journal file is attractive when an existing shared filesystem is available and setting up a database server would be inconvenient. It is not a drop-in guarantee of safe or efficient access from any number of compute nodes. In particular, SQLite database files should not be used as a substitute for a properly configured multi-node backend on NFS.

3. Create a Small Optimization Example

The following example works with a local journal file. It uses a simple mathematical objective rather than a trained machine-learning model so that the storage and dashboard workflow is the focus.

Install Optuna and Optuna Dashboard

Use Python 3.10 or newer, as required by the current Optuna Dashboard documentation. On Linux or macOS:

mkdir -p optuna-journal-demo
cd optuna-journal-demo

python3 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install optuna optuna-dashboard

python -m pip show optuna optuna-dashboard
Enter fullscreen mode Exit fullscreen mode

The last command prints the installed package versions. Record those versions if you intend to reproduce an experiment later. The installation commands above are documentation-based examples, not a newly tested environment specification.

Write the optimization script

Create optimize.py:

from pathlib import Path
from time import sleep

import optuna
from optuna.storages import JournalStorage
from optuna.storages.journal import JournalFileBackend

LOG_PATH = Path(__file__).resolve().with_name("optuna_journal.log")
STUDY_NAME = "journal_dashboard_demo"


def objective(trial: optuna.Trial) -> float:
    x = trial.suggest_float("x", -10.0, 10.0)
    y = trial.suggest_float("y", -10.0, 10.0)

    # Only makes progress easier to observe in the dashboard.
    sleep(0.5)
    return (x - 2.0) ** 2 + (y + 3.0) ** 2


def main() -> None:
    storage = JournalStorage(JournalFileBackend(str(LOG_PATH)))

    study = optuna.create_study(
        study_name=STUDY_NAME,
        storage=storage,
        direction="minimize",
        load_if_exists=True,
    )

    study.optimize(objective, n_trials=20)

    print(f"Study: {study.study_name}")
    print(f"Recorded trials: {len(study.trials)}")
    print(f"Best value: {study.best_value}")
    print(f"Best parameters: {study.best_params}")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Then run:

python optimize.py
Enter fullscreen mode Exit fullscreen mode

Optuna creates optuna_journal.log alongside the script and records the study's operations there. After the run, the script prints the number of recorded trials and the best observed parameters. The numerical results depend on the sampled trials and are not reported here as measured performance.

Because load_if_exists=True is set, rerunning the script with the same STUDY_NAME and LOG_PATH joins the existing study rather than intentionally starting a new one. Each execution requests 20 additional trials; n_trials limits a single call to study.optimize(), not the total number of trials in the shared study.

Verify the saved study without running new trials

You can also reload the study without starting another optimization run. From the project directory, execute:

python - <<'PY'
import optuna
from optuna.storages import JournalStorage
from optuna.storages.journal import JournalFileBackend

storage = JournalStorage(JournalFileBackend("optuna_journal.log"))
study = optuna.load_study(
    study_name="journal_dashboard_demo",
    storage=storage,
)
print(f"Recorded trials: {len(study.trials)}")
print(f"Best value: {study.best_value}")
print(f"Best parameters: {study.best_params}")
PY
Enter fullscreen mode Exit fullscreen mode

Unlike running optimize.py again, this inspection step does not create any new trials.

4. Share a Study Across Multiple Processes

Two processes on the same machine can run the same optimize.py and use the same journal file. From the project directory on Linux or macOS:

python optimize.py > worker1.log 2>&1 &
worker1_pid=$!
python optimize.py > worker2.log 2>&1 &
worker2_pid=$!

wait "$worker1_pid"; status1=$?
wait "$worker2_pid"; status2=$?
printf 'Worker 1 exit code: %s\nWorker 2 exit code: %s\n' "$status1" "$status2"
Enter fullscreen mode Exit fullscreen mode

Check both exit codes and the corresponding log files. Exit code 0 indicates that the Python process finished successfully; nonzero codes require investigation. Since each worker can finish at a different time, its printed trial count may not represent the final total until both workers have completed.

Both workers use the same file path and study name. Assuming both finish normally, they request 20 trials each, adding 40 trials in total. The trial numbers are allocated through the shared storage; the work does not need to be divided manually into fixed trial-number ranges.

The example uses the local filesystem, where JournalStorage is documented for multi-process use. Do not assume that the same default locking behavior is safe when the file is moved to NFS and accessed concurrently from different machines. The distinction matters for HTCondor, Slurm, and other multi-node setups.

5. Inspect the Study with Optuna Dashboard

Optuna Dashboard can open a file-based journal directly. In a separate terminal, activate the same virtual environment and run the following from the project directory:

source .venv/bin/activate
optuna-dashboard --host 127.0.0.1 --port 8080 ./optuna_journal.log
Enter fullscreen mode Exit fullscreen mode

Open http://127.0.0.1:8080 in a browser on that machine.

The dashboard can display trial lists, objective-value history, parameter relationships, and related visualizations. Available plots depend on the study data and the dashboard version. If the optimization is still running, refresh or live updates can reveal newly recorded trials as the dashboard reads updated storage state.

Use a different port

The default port is 8080. If that port is occupied, or if you want to run a second dashboard instance, choose another port:

optuna-dashboard --host 127.0.0.1 --port 8081 ./optuna_journal.log
Enter fullscreen mode Exit fullscreen mode

You can then open http://127.0.0.1:8081. To compare different journal files, start separate dashboard processes with different file paths and ports. A single journal file may also contain multiple named studies, which can be selected within the dashboard.

Binding to 127.0.0.1 keeps the web interface local to the host. Avoid exposing research experiment metadata on a public network without appropriate access controls. For a dashboard running on a remote compute server, an SSH tunnel is preferable to opening the service to everyone on the network.

6. Using a Shared NFS/NAS Journal in a Cluster

The original notes illustrate a journal path on mounted shared storage, for example:

/mnt/nas/optuna/optuna_journal.log
Enter fullscreen mode Exit fullscreen mode

For this layout, every participating worker must be able to read and write the same underlying file, and the directory must already exist. Different mount points, permissions, or locking semantics can make a seemingly shared configuration behave incorrectly.

Optuna exposes explicit file-lock implementations. For example, its JournalFileSymlinkLock class is documented for synchronization on NFSv2 or later:

from optuna.storages import JournalStorage
from optuna.storages.journal import JournalFileBackend, JournalFileSymlinkLock

log_path = "/mnt/nas/optuna/optuna_journal.log"

storage = JournalStorage(
    JournalFileBackend(
        log_path,
        lock_obj=JournalFileSymlinkLock(log_path),
    )
)
Enter fullscreen mode Exit fullscreen mode

Important: In Optuna 4.8.0, JournalFileBackend already uses JournalFileSymlinkLock by default. Specifying it explicitly makes the locking strategy visible, but does not by itself offer additional protection against NFS race conditions. Optuna also provides JournalFileOpenLock, documented for NFSv3 or later. The appropriate strategy depends on the filesystem and its behavior; neither option guarantees reliable multi-node operation without validation.

This is a configuration example, not a validation of a particular NAS or NFS deployment. All workers must use a compatible locking strategy. Before relying on it for long-running cluster experiments, test concurrent writers, interruption and restart behavior, and journal readability on the actual mounted filesystem.

There is an important documentation nuance: Optuna's file-based JournalStorage recipe describes NFS-based distributed optimization, but the Optuna 5.0 parallelization guide warns that file locking across machines can cause race conditions and recommends RDBStorage for multi-node optimization. For a production cluster, PostgreSQL or MySQL through RDBStorage is the more conservative choice unless the shared-file approach has been validated in the target environment.

When using HTCondor or another scheduler, each worker can participate in the same study by using the same study name and shared storage. The scheduler still needs to arrange execution, dependencies, software environments, and resource requests; JournalStorage does not replace those functions.

7. Operational Considerations

Journal growth. A journal accumulates operations over time. For long-lived studies, plan for backups and storage capacity. Do not truncate or manually rewrite an active journal file to reduce its size.

Restart and recovery. A restarted process can reconstruct persisted study state from the journal. This does not mean that a process crash automatically resumes an interrupted model-training computation from an exact checkpoint. Model checkpoints and trial-level failure handling are separate concerns.

Write contention. Frequent trial-state updates and high worker counts can increase locking and I/O overhead. File-backed JournalStorage is not designed for unlimited concurrent writers.

Troubleshooting an empty dashboard. Verify that the dashboard is reading the same journal file that the optimization script writes, that the file exists, and that the selected study contains trials. When files are on shared storage, also check permissions and mounts.

Data access. A dashboard may reveal hyperparameters, intermediate results, and experiment metadata. Keep its network binding and file permissions appropriate for the data.

Moving studies. Optuna's copy_study API can copy a study between supported storage backends when a different storage architecture is needed. It is not necessary to parse and import individual journal lines manually. For a stable migration snapshot, stop the optimization workers before copying: the API also permits copying a study while trials are still running, which can leave those unfinished trials in the destination.

8. Conclusion

JournalStorage provides a persistent operation log for Optuna studies, and JournalFileBackend makes that log easy to store as a file. Optuna Dashboard complements it by making trials and optimization history visible through a browser.

For local multi-process experiments, the combination is a straightforward way to preserve and inspect results without a dedicated database server. For distributed experiments across machines, however, storage consistency and filesystem locking deserve explicit validation; RDBStorage remains an important alternative.

References

AI disclosure: This article documents my own technical setup and results. An LLM was used to help draft, organize, and edit the English text. I remain responsible for its technical accuracy.

Top comments (0)