DEV Community

Lawson Dong
Lawson Dong

Posted on AI-assisted

Managing My First AI Experiment: Checkpoints, Logs, and Python Virtual Environments

#python #linux

In my previous note, I set up Ubuntu with WSL 2, created a Python virtual environment, trained a small XOR neural network with PyTorch, and saved its parameters.

This time, I wanted to make that experiment easier to manage: organize the model files, save the training output, and understand what actually changes when I activate a virtual environment.

This is a hands-on continuation using the same project, ~/research/first-neural-network. You will need the existing train.py, .venv, and xor_model.pth from the previous tutorial.

1. Open Ubuntu and return to the project

Start a new Windows Command Prompt. It may initially look like this:

A new Windows Command Prompt

Enter:

wsl -d Ubuntu
Enter fullscreen mode Exit fullscreen mode

The prompt now belongs to Ubuntu:

Ubuntu shell after entering WSL

Notice that WSL may initially open in a mounted Windows directory, such as /mnt/c/Users/linxi. Navigate to the Linux project explicitly:

cd ~/research/first-neural-network
pwd
ls -la
Enter fullscreen mode Exit fullscreen mode
  • cd changes the current working directory.
  • pwd prints its absolute path.
  • ls -la lists the directory contents in detail, including hidden entries such as .venv.

Project directory containing the virtual environment, training script, and saved model

2. Organize the model files

I wanted to separate the Python environment, model parameters, and training logs:

The project organized into a training script, Python environment, checkpoints, and logs

Create two directories and copy the existing model file:

mkdir -p checkpoints logs      # Create both directories if they do not exist
cp xor_model.pth checkpoints/  # Copy the saved parameters into checkpoints/
ls checkpoints                # List the files in checkpoints/
Enter fullscreen mode Exit fullscreen mode

The copied xor_model.pth file listed inside checkpoints

Seeing xor_model.pth in the output confirms that the file is present in checkpoints/. In this project, it stores the trained model's parameters, including weights and biases, saved by PyTorch. The .pth extension alone does not tell us what every PyTorch file contains; that depends on what the training script saved.

Because I used cp, the original file is still in the project root:

The original model file and its separate copy in checkpoints

These are two separate files. If train.py saves a new xor_model.pth in the project root later, the copy in checkpoints/ will not update automatically. For now, that copy is a snapshot of the earlier run.

Inspect the file sizes 

ls -lh xor_model.pth checkpoints/xor_model.pth
Enter fullscreen mode Exit fullscreen mode

You can write -lh as -l -h:

  • -l uses the long listing format, showing permissions, owner, size, modification time, and filename.
  • -h displays sizes in human-readable units, such as K, M, and G.

Explanation of ls, the long-listing option, and human-readable file sizes

The size is one field in a longer line of file information:

How to find a model file's size in an ls long listing

Here are the two files side by side:

The original and copied model files have the same displayed size

Matching displayed sizes are a useful quick check, although they do not prove that two files have identical contents.

3. Activate the Python environment

Before running the training script, activate the project's existing environment:

source .venv/bin/activate
Enter fullscreen mode Exit fullscreen mode

source executes the activation script in the current shell. Activation puts .venv/bin at the beginning of the shell's PATH, so commands such as python and pip use the project's environment.

The structure and meaning of source .venv/bin/activate

Install NumPy if the project needs it

NumPy is a Python package, not a separate environment. To install it into this virtual environment and check its version:

python -m pip install numpy
python -c "import numpy; print(numpy.__version__)"
Enter fullscreen mode Exit fullscreen mode

The version attribute is __version__, with two underscores on each side. Using python -m pip runs pip with the same Python interpreter selected by python.

Installing NumPy may resolve a missing-NumPy warning from PyTorch, but NumPy is not required to read a text log. We will use Linux commands for that.

4. Save the training output to a log

Run the training script and capture its output:

python train.py > logs/training.log 2>&1
Enter fullscreen mode Exit fullscreen mode

The training command broken into Python, script, output redirection, and error redirection

There are two main output streams:

  • Standard output (stdout, descriptor 1): ordinary output, such as printed epoch numbers and losses.
  • Standard error (stderr, descriptor 2): errors, warnings, and other diagnostic output.

> logs/training.log sends standard output to the file. Then 2>&1 sends standard error to the same destination as standard output. The order matters: redirect standard output first, then make standard error follow it.

How stdout and stderr are captured with and without 2 greater-than ampersand 1

The terminal may appear quiet while training runs because the output is going into the file. The command returns to the shell prompt when the script finishes.

Each run with > overwrites the existing log. To append instead, use:

python train.py >> logs/training.log 2>&1
Enter fullscreen mode Exit fullscreen mode

A log records what the program writes; it does not automatically save the trained model. Model saving still depends on the code in train.py.

5. Read and monitor the training log

After training finishes, display the log:

cat logs/training.log
Enter fullscreen mode Exit fullscreen mode

cat is short for concatenate. It can combine files, and when given a single file, it prints that file's contents directly in the terminal.

For longer logs, these commands are useful:

Commands for displaying, browsing, and monitoring text files

head logs/training.log       # Show the first 10 lines
tail -n 5 logs/training.log # Show the last 5 lines
less logs/training.log       # Browse the file interactively; press q to quit
Enter fullscreen mode Exit fullscreen mode

Here is a screenshot from my session, showing a NumPy installation followed by the training log:

Installing NumPy and displaying the recorded training output

Installing a package after a run does not change a warning already recorded in that run's log. To capture a new run with the updated environment, run the training command again.

Follow a longer training run

For a long experiment, use a second terminal to watch new log lines arrive.

In the first terminal, start training with unbuffered Python output:

cd ~/research/first-neural-network
source .venv/bin/activate
python -u train.py > logs/training.log 2>&1
Enter fullscreen mode Exit fullscreen mode

-u makes Python's output unbuffered, which helps printed progress appear promptly in a redirected log.

In a second Windows Command Prompt, enter Ubuntu and follow the file:

wsl -d Ubuntu
cd ~/research/first-neural-network
tail -f logs/training.log
Enter fullscreen mode Exit fullscreen mode

tail -f displays the last 10 lines and continues showing newly appended content. Press Ctrl+C in the monitoring terminal to stop following the log; training in the first terminal continues.

6. Understand the virtual environment and the file system

A Python virtual environment gives a project its own installed packages and access to a Python interpreter through the environment's executables. In this project, .venv is simply a directory inside first-neural-network.

The important paths are:

Path Purpose
.venv/bin/python Python interpreter executable, which may be a copy or symbolic link
.venv/bin/pip Python package installer entry point
.venv/bin/activate Shell script used to activate the environment
.venv/lib/pythonX.Y/site-packages/ Installed Python packages; X.Y depends on the Python version
train.py Neural network training script
xor_model.pth Model parameter file originally saved in the project root
checkpoints/xor_model.pth Separate copy of that parameter file
logs/training.log Recorded training output

Activating .venv does not change your working directory or prevent Python and the shell from accessing the rest of the project. It is not a separate Linux file system or a file-access sandbox.

Experiment: check what activation changes

Start with a fresh Ubuntu shell, or run deactivate first if the environment is already active. Then execute this block:

# Step 1: Navigate to the project directory
cd ~/research/first-neural-network

# Step 2: Check the current directory and Python executable
pwd
which python3

# Step 3: Activate the virtual environment
source .venv/bin/activate

# Step 4: Check the directory, Python executable, and model file again
pwd
which python
ls xor_model.pth
Enter fullscreen mode Exit fullscreen mode

Before and after activation: unchanged working directory, a different Python executable path, and an accessible model file

Observations:

  • pwd prints the same path before and after activation, showing that the current working directory has not changed.
  • which python now points to the executable inside the project's .venv/bin/ directory.
  • ls xor_model.pth still finds the model file in the project root.

Conclusion: Activating a Python virtual environment changes which Python executable and project packages are used. It does not create a separate Linux file system or change the current working directory.

To exit the environment:

deactivate
Enter fullscreen mode Exit fullscreen mode

The project files and saved logs remain where they were.

References

Top comments (1)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.