DEV Community

YMori
YMori

Posted on • Edited on • Originally published at zenn.dev

Track SIGNATE Competition Scores with W&B: signate-wandb-sync

#python #machinelearning #wandb #cli

Background

I previously built signate-deploy — a CLI tool to automate SIGNATE competition submissions via GitHub Actions.

The missing piece was experiment tracking: after submitting, there was no easy way to record the SIGNATE score back into W&B alongside the training metrics.

So I built a companion tool.

signate-wandb-sync 

pip install signate-wandb-sync
Enter fullscreen mode Exit fullscreen mode

PyPI: https://pypi.org/project/signate-wandb-sync/
GitHub: https://github.com/yasumorishima/signate-wandb-sync

Records SIGNATE submission scores to W&B run summaries with a single command.

The Full Pipeline

Combined with signate-deploy, the complete workflow looks like this:

[GitHub Actions]
  1. Download data via SIGNATE API (signate-deploy)
  2. Run train.py — W&B run created, metrics logged
  3. Submit to SIGNATE (signate-deploy)
  → W&B run URL printed to Actions log

[Local]
  4. Check score on SIGNATE leaderboard
  5. Record score to W&B with signate-wandb-sync
Enter fullscreen mode Exit fullscreen mode

Add W&B to train.py

A few lines in your training script:

import wandb

run = wandb.init(
    project="my-signate-project",
    config={"model": "...", "params": "..."},
)

# ... training and inference ...

wandb.log({"oof_score": oof_score})

print(f"W&B run URL: {run.url}")  # visible in Actions log
wandb.finish()
Enter fullscreen mode Exit fullscreen mode

Set WANDB_API_KEY as a GitHub Secret and pass it to the training step:

- name: Train and predict
  env:
    WANDB_API_KEY: ${{ secrets.WANDB_API_KEY }}
  run: python train.py
Enter fullscreen mode Exit fullscreen mode

Record the score

After checking your score on the SIGNATE leaderboard:

signate-wandb-sync score https://wandb.ai/your-entity/your-project/runs/abc123 \
    --score 0.85 --rank 3
Enter fullscreen mode Exit fullscreen mode

Output:

Updated run: my-run-name (your-entity/your-project/abc123)
  submitted = True
  signate_score = 0.85
  signate_rank = 3
Enter fullscreen mode Exit fullscreen mode

Options

Option Description
--score SIGNATE submission score (float)
--rank Leaderboard rank (int)
--metric KEY=VALUE Additional metric, repeatable
--project entity/project Required when using bare run ID 
# Log extra metrics alongside score
signate-wandb-sync score <W&B URL> \
    --score 0.85 --rank 3 \
    -m fbeta=0.85 -m recall=0.91
Enter fullscreen mode Exit fullscreen mode

run_id formats 

# Full URL (recommended — copy from Actions log)
signate-wandb-sync score https://wandb.ai/entity/project/runs/abc123 --score 0.85

# Path format
signate-wandb-sync score entity/project/abc123 --score 0.85

# Bare ID (requires --project)
signate-wandb-sync score abc123 --project entity/project --score 0.85
Enter fullscreen mode Exit fullscreen mode

Windows note 

PYTHONUTF8=1 signate-wandb-sync score <W&B URL> --score 0.85
Enter fullscreen mode Exit fullscreen mode

Summary 

pip install signate-deploy signate-wandb-sync
Enter fullscreen mode Exit fullscreen mode
  • signate-deploy: Automate data download, training, and submission on GitHub Actions
  • signate-wandb-sync: Record submission scores back to W&B

Together, they close the loop on experiment tracking for SIGNATE competitions.

PyPI: https://pypi.org/project/signate-wandb-sync/
GitHub: https://github.com/yasumorishima/signate-wandb-sync

Top comments (0)