What a run records¶

Everything below is a method on the Run that visin.init returns. Every method returns immediately, and the report is sent in the background, in the order you made it.

Epochs¶

run.log_epoch(
    12,
    train={"loss": 0.41},
    val={"loss": 0.47, "mean_iou": 0.58, "car": {"iou": 0.71}, "pedestrian": {"iou": 0.42}},
    learning_rate=1e-4,
    epoch_time=812.5,  # seconds; Visin adds these up, and prices them if the project has rates
)

results is yours to shape, and anything you send is stored. train= and val= are shorthand for results["train"] and results["val"]. Visin's charts read these names:

Chart	Reads
Loss	`train.loss`, `val.loss`
Mean IoU	`train.mean_iou`, `val.mean_iou`
Pixel accuracy, mean accuracy, Dice	`pixel_accuracy`, `mean_accuracy`, `dice_score`
Per class	any key under `train` or `val` whose value holds `iou`, `precision`, `recall`, `f1` or `ap`

Classes need no setup. The first epoch that reports car adds a car line.

log_epoch returns the epoch's UUID straight away. It is derived from the run and the epoch number, so the same epoch always has the same UUID, even from another process. That is what makes a retry, or a restarted job that logs the epoch again, harmless: the server keeps the first copy.

NumPy, PyTorch and NaN

Metrics can be NumPy scalars, zero-dimensional tensors or Decimals, and they are converted for you. NaN and infinity have no JSON form, so they are sent as null, which Visin charts as a gap. A warning names them the first time.

The System tab¶

visin.init(..., system_metrics=True), or log_epoch(..., system=True) for a single epoch, adds a snapshot of memory and GPU use to each epoch. That fills the run's System tab:

memory used, and the process's peak
per GPU: memory used, reserved and peak, temperature, power and fan speed

GPU figures come from PyTorch if your script has already imported it, and from nvidia-smi otherwise. pip install 'visin[system]' adds psutil for CPU load and the process's own use.

Benchmarks record what the machine is, through visin.system_info(): core counts, memory, OS, Python version, the CPU model, and for GPUs the name, memory, driver and CUDA version. Call it yourself to see exactly what is sent; nothing else about the machine leaves it. Pass your own system_info to log_benchmark to override any of it.

Test results¶

Scores on held-out data, attached to an epoch, usually the checkpoint you would ship:

run.log_test_results(
    12,
    {
        "day": {"overall": {"pixel_accuracy": 0.93}, "car": {"iou": 0.74}, "pedestrian": {"iou": 0.47}},
        "night": {"overall": {"pixel_accuracy": 0.86}, "car": {"iou": 0.61}, "pedestrian": {"iou": 0.29}},
    },
)

The top-level keys are conditions (one per test set), the keys inside them are classes, and the numbers inside those are metrics. overall holds metrics for a whole condition.

Benchmarks¶

How fast the model runs and how big it is, on one or more devices:

run.log_benchmark(
    [
        {
            "device": "cuda",
            "device_type": "RTX 4090",
            "batch_size": 1,
            "mean_time_ms": 8.4,
            "fps": 119.0,
            "total_parameters_m": 27.4,
            "flops_giga": 62.1,
        },
        {"device": "cpu", "batch_size": 1, "mean_time_ms": 142.0, "fps": 7.0},
    ],
    epoch=12,
)

Visin requires a description of the machine with each benchmark. Any part of it you leave out is filled in by visin.system_info(): CPU cores, memory, GPU name, memory and driver.

Prediction frames¶

run.upload_visualization(12, "renders/overlay_0001.png", kind="overlay")

PNG, JPEG, GIF, WebP, MP4, MOV and PDF files are accepted. The file is copied when you call, so you can render the next epoch's frame over it straight away. kind is your own name for what the image shows, and the Visualizations tab filters by it. Render the same inputs under the same file names each epoch, and you can follow one image through training.

An image you hold in memory needs no file. Pass the array, tensor, PIL image or Matplotlib figure, and give it a name:

overlay = (0.6 * image + 0.4 * mask_colour).clip(0, 255).astype("uint8")  # H x W x 3
run.upload_visualization(12, overlay, kind="overlay", name="sample_0001")
fig = plot_confusion(...)
run.upload_visualization(12, fig, kind="confusion")  # stored as confusion.png

It is stored as PNG. Arrays and tensors may be H x W, H x W x 3, H x W x 4, or channels first (3 x H x W). A float array is read as 0 to 1, or as 0 to 255 when it holds a larger value. No extra package is needed for arrays and tensors; PIL images and figures are saved by their own libraries. The name defaults to <kind>.png, so give each frame its own when you log several per epoch.

Checkpoints on Hugging Face¶

run.log_model("checkpoints/best.pt", "acme/clftv2-zod", epoch=12)

Uploads the file (or folder) to the Hub model repo and links it to the run. Visin keeps only a pointer to the commit that upload made, shown on the run as acme/clftv2-zod @ 3f2a1c9, so the run keeps naming exactly those weights. The repo is created if needed, private unless you pass private=False.

It also writes the repo's README from the run: the dataset and commit, epochs reported, headline results, per-condition test scores and speed, with the Hub's model-index as front matter, so the scores show on the model page. A README is added only when the repo has none, so one you wrote is never replaced; pass card=False to skip it. No config values go in it. It is written from what Visin has when you call, so log the epoch and test results first.

safetensors=True also uploads a PyTorch checkpoint that is a plain state dict as model.safetensors, beside the original (it needs torch installed). The format holds tensors only, so anything saved next to the weights, like a pipeline's model_info, stays in the original file, which is why neither replaces the other.

Needs pip install 'visin[hf]' and your own Hub token (HF_TOKEN or huggingface-cli login). Visin never sees it. The project must keep its storage on Hugging Face (project settings, Storage): a project set to Visin refuses the link, and the checkpoint is then on the Hub but not shown on the run. A failure is logged, never raised into training; the call returns the commit hash, or None when nothing was uploaded.

Checkpoints kept on Visin¶

digest = run.save_model(
    "checkpoints/best.pth",
    epoch=12,
    licence="cc-by-4.0",
    credit="Acme Robotics",
    model_info={"backbone": "clftv2", "transforms": {"resize": 384}},
)
run.save_model("export/best.onnx", epoch=12, licence="cc-by-4.0", source_sha256=digest)

Transient retries retain the upload reservation and resume the same file. A lost claim response repeats only that claim after upload completion. If finish() times out during a large model upload, its op and staged copy remain on disk for visin sync, even if the sender finishes later. A download checks the whole-file Repr-Digest when present, including on a ranged response, against the recorded model digest.

Where log_model links a repo on the Hub and keeps no bytes, save_model uploads the file to your Visin, so the run offers it for download, in the models list and over the API. It takes a PyTorch .pth or an .onnx export, and returns the file's SHA-256, which is how a checkpoint is named: give a .pth's digest as source_sha256 with the .onnx made from it, and the two are one model with two formats to evaluations and leaderboards.

A model is served only when two things hold. Visin has read the stored file and found it to be the file you sent (the run shows it as being verified until then; a file that does not match the digest is refused and deleted), and you have stated the licence it is shared under, an id such as "cc-by-4.0" or a mapping like {"id": "other", "name": "Acme terms", "url": "https://..."}. Without a licence the model is stored but not offered: say so deliberately. credit is the attribution shown with it, model_info is what the weights record about the network and its input (kept as given, so a consumer knows the resize and normalisation), and parity is an export's agreement with its weights.

The file goes up in pieces, which gets a large checkpoint past a proxy's request-size limit and lets it resume after a dropped connection. With keep_copy=True (the default) it is first copied beside the run's reports, so it can still be sent if training has since overwritten or deleted it; that costs the disk, and keep_copy=False sends the original instead. One file is kept per format and epoch: sending a different file for the same slot is refused until the first is unlinked. A failure is logged, never raised into training.

Config¶

run.log_config(args)  # argparse.Namespace, dict, dataclass, pydantic model or Hydra config

The config is stored and linked to the run, and shown on its page. Values JSON cannot hold, such as a loss module, are kept as their repr. visin.init(..., config=cfg) does the same in one call.

Changing the run¶

run.update(tags=["best"], metadata={"best_epoch": 12, "best_val_iou": 0.61})

metadata replaces what the run had rather than merging with it.

From a separate test or benchmark script¶

Research code often trains in one script and tests, renders and benchmarks in others, run afterwards as separate processes. Each later script reports into the training's run with Run.attach(training_uuid, mark_status=False):

run = visin.Run.attach(training_uuid, mark_status=False)
with run:
    run.log_test_results(epoch, results, epoch_uuid=epoch_uuid)

mark_status=False leaves the run's status to the training. Without it, the test script's finish, or its crash, would mark the training completed or failed.
epoch_uuid= names the checkpoint's epoch directly. The training script can put it in the checkpoint's file name: log_epoch returns it, and visin.epoch_uuid_for(training_uuid, epoch) gives the same UUID anywhere. log_test_results, log_benchmark and upload_visualization all take it.

Logging without passing the run around¶

A helper deep in a training codebase can log without being handed the run. init and Run.attach remember the run they made, and these functions use it:

# train.py
visin.init("unet baseline", project="road-seg")

# losses.py, anywhere in the same process
import visin

visin.log_epoch(epoch, train={"loss": loss})

visin.get_run() returns that run, for anything the shortcuts do not cover. log_epoch, log_test_results, log_benchmark, log_config, log_model, save_model, upload_visualization, update and finish all take what the Run methods take. With no run started they do nothing, like a disabled run. A run made with Run.create directly is yours to hold and is not made current. With two runs in one process, call the methods on the one you mean.

Finding the run in Visin¶

run.url is the run's page in the web app, once the server has answered init:

with visin.init("unet baseline") as run:
    print(run.url)  # https://app.visin.eu/trainings/…

It is None for an offline or disabled run, and for a self-hosted deployment until you set VISIN_APP_URL. The same link is in the log line printed when the run is created; see visin.enable_console_logging() to see it.

Resuming¶

A job that restarts with the same training_uuid reports into its existing run, which run.resumed tells you. The run is marked running again. Epochs it logs again keep their first recorded values, and visin warns once when that happens. A job whose epoch numbering starts over should start a new run instead.

Finishing¶

with visin.init(...) as run: finishes the run for you: completed when the block ends normally, failed when it raises. Without with:

run.finish()  # or run.finish("failed"), or run.fail(exc)

A script that ends without calling finish is finished at exit: completed if it ended normally, failed if it crashed or was terminated. SIGTERM, which is what a scheduler sends before it kills a job, is turned into an ordinary exit so this still happens. If your framework already handles SIGTERM, as Lightning does on SLURM, its handler is left alone.

When things go wrong¶

Two things raise, both at startup, where you are there to read them: init raises ApiError when Visin refuses the run outright (a bad token, a project the key does not cover), and Run.attach raises ConfigurationError when it has no run to attach to. Visin being unreachable does not raise; reports wait on disk.

After that nothing raises into your loop. A report that fails is logged as a warning, and finish prints a summary. It reads like this:

visin: run 5f0c… completed: 118 reports sent, 2 failed, 0 dropped, 0 waiting on disk

Pass strict=True to init to get exceptions instead, which is useful in tests.