Api¶
Api ¶
Api(url: str | None = None, token: str | None = None, *, verify_ssl: bool | None = None, client: HttpClient | None = None)
A read client for one Visin instance.
Everything it returns is a model from :mod:visin.models. Anything a
method takes as a run accepts a :class:~visin.models.Training or a run's
UUID.
projects ¶
projects(search: str | None = None) -> list[Project]
The projects this credential can see.
trainings ¶
trainings(*, project: str | None = None, status: str | None = None, tags: Iterable[str] | str | None = None, search: str | None = None, dataset: str | None = None, sort_by: str = 'updated_at', order: str = 'desc', limit: int | None = None, page_size: int = 100) -> Iterator[Training]
Runs, newest first, fetched a page at a time as you iterate.
sort_by is one of updated_at, created_at, start_time,
end_time, name or status. limit stops after that many;
without it, every matching run.
training ¶
training(ref: str | Training) -> Training
One run, by its UUID, or a run you already have.
find ¶
find(name: str | None = None, *, project: str | None = None, tags: Iterable[str] | str | None = None, status: str | None = None) -> Training | None
The newest run called exactly name that matches the rest, or None.
Without name, the newest run that matches the filters. For a script
that must pick up where an earlier one left off without having kept
its UUID.
comparisons ¶
comparisons(*, project: str | None = None, type: str | None = None, search: str | None = None, limit: int | None = None, page_size: int = MAX_PAGE_SIZE) -> Iterator[Comparison]
Saved comparisons, a page at a time, optionally of one type (trainings, tests, ...).
findings ¶
findings(*, project: str | None = None, run: str | Training | None = None, limit: int | None = None) -> Iterator[Finding]
Recorded findings, newest first: a project's, or those about one run (as its subject or cited).
summary ¶
summary(ref: str | Training) -> Summary
How a run did, in one call: its state, models and provenance, and per result the best epoch.
summary.metric("val.mean_iou") gives the best value, the epoch it was reached at and the last
value, for the direction the project says is better. A run's last epoch is not its result, so both
are there. Read direction_from on a metric: default means Visin guessed the direction.
config ¶
config(ref: str | Training) -> Configuration | None
What the run was launched with, or None if it logged no config.
visualizations ¶
visualizations(ref: str | Training, kind: str | None = None) -> list[Visualization]
The frames a run stored, newest first, each with a signed link to its file.
kind keeps one kind, as given to upload_visualization.
download_visualization ¶
download_visualization(visualization: Visualization, directory: str | PathLike[str]) -> Path
Save a frame under directory, by its file name, and return the path.
models ¶
models(ref: str | Training) -> list[StoredModel]
The checkpoints a run kept on Visin, one per format (.pth, .onnx).
Each says whether it can be downloaded (:attr:~visin.models.StoredModel.downloadable) and, if not,
why. A model the run only links on the Hugging Face Hub is not here: it is in Training.models.
download_model ¶
download_model(ref: str | Training, model: str | StoredModel | None = None, directory: str | PathLike[str] = '.', *, file_format: str | None = None, outputs: bool = False, progress: Any = None) -> Path
Save a stored model's file under directory and return its path.
model is a :class:~visin.models.StoredModel or its id; without one the run must keep exactly
one (narrow with file_format="onnx"). The file is checked against the SHA-256 Visin recorded, an
interrupted download resumes where it stopped, and a file that does not match is deleted and refused.
With outputs=True this saves instead the zip of what the model produced on the frames its run
was evaluated on: predictions, confidence, the checkpoint's results and a manifest. That zip is built
as it is sent, so it is fetched whole and not checked against a digest.
test_results ¶
test_results(ref: str | Training) -> list[TestResult]
What the run's checkpoints scored, newest first: its test results, with or without a suite.
benchmarks ¶
benchmarks(ref: str | Training | None = None, *, project: str | None = None) -> list[Benchmark]
Benchmarks of a run, or of a whole project, newest first.
suites ¶
suites(*, slug: str | None = None, project: str | None = None, include_archived: bool = False) -> list[Suite]
The suites this credential can read, newest version first within a name.
A suite is readable by whoever can read its project, and by anyone when it is public. Archived suites
(retired: no new results) are left out unless include_archived is true.
suite ¶
suite(ref: str, version: int | str | None = None) -> Suite
One suite, by "slug@1", by slug and version, or by slug alone for the latest.
evaluations ¶
evaluations(*, project: str | None = None, suite: str | None = None, state: str | None = None, status: str | None = None, checkpoint_key: str | None = None) -> list[Evaluation]
Evaluations this credential can read, newest first, without their results and provenance.
suite is "slug@version". state keeps one verdict (eligible, incomplete,
incompatible, exploratory, legacy-unverified); open an evaluation with
:meth:evaluation for its results.
evaluation ¶
evaluation(ref: str | Evaluation, *, project: str | None = None) -> Evaluation
One evaluation, with its results and provenance, by id or (with project) by your own uuid.
Use the uuid form to recover after a lost response: it finds what an earlier attempt stored.
check_protocol ¶
check_protocol(source: str | Path | Mapping[str, Any]) -> str
The digest Visin gives a protocol file or dict, without publishing it.
leaderboard ¶
leaderboard(ref: str, version: int | str | None = None, *, page: int | None = None, limit: int | None = None, unranked_page: int | None = None, all_pages: bool = False, observed: bool = False) -> Leaderboard
The ranking of a suite version over the evaluations this credential can read.
One entry per checkpoint, from its latest ranked attempt, never its best. candidates says how many
evaluations were in the pool: a rank is a position in that pool, and ranks are global, so page 2 does
not restart at 1. pagination.total is how many checkpoints were selected. What has attempts but
none ranked is in unranked, with the reasons, and pages apart from the ranking
(unranked_page).
page and limit (at most 100) choose one page. all_pages=True reads every page of both
lists and returns them together with complete set; it cannot be combined with page.
observed=True ranks only the results whose evaluator sent complete evidence that matched; the
pool, its counts and the ranks are then those of that subset, and reported and promoted results
are left out.
public_leaderboards ¶
public_leaderboards(*, page: int | None = None, limit: int | None = None) -> list[dict[str, Any]]
One page of the leaderboards with something published, anonymously: the same list whoever asks.
The server sends 100 at most per page; iter_public_leaderboards reads them all.
iter_public_leaderboards ¶
iter_public_leaderboards(*, page_size: int = LEADERBOARD_PAGE) -> Iterator[dict[str, Any]]
Every public leaderboard, anonymously, reading page after page.
public_leaderboard ¶
public_leaderboard(ref: str, version: int | str | None = None, *, page: int | None = None, limit: int | None = None, all_pages: bool = False, observed: bool = False) -> Leaderboard
One public leaderboard, anonymously: only published results, and a version number is required.
Paged and global-ranked like leaderboard; there is no unranked list on the public view.
observed=True ranks only the results whose evaluator sent complete evidence that matched.
epochs_frame ¶
epochs_frame(ref: str | Training, *, sep: str = '.') -> Any
A run's epochs as a pandas DataFrame, one row per epoch.
Results are flattened into columns (val.loss, val.car.iou), and
learning_rate and epoch_time sit beside them. A metric an epoch
did not report is missing (NaN), as it is in Visin's charts.
compare_frame ¶
compare_frame(refs: Iterable[str | Training], metric: str, *, sep: str = '.') -> Any
One metric of several runs side by side: a row per epoch, a column per run.
metric is a flattened name such as val.mean_iou. Columns are run
names; two runs with the same name are told apart by the start of their
UUID. A run that never reported the metric is a column of NaN.
flatten ¶
flatten(results: Mapping[str, Any], sep: str = '.', prefix: str = '') -> dict[str, Any]
Nested results as one level: {"val": {"loss": 1}} → {"val.loss": 1}.
Lists are kept whole, as values: they are rarely metrics, and exploding them into numbered columns makes a frame nobody can read.
models ¶
What Visin returns, as objects with Python names.
The server spells fields its own way (_id, projectId, training_uuid).
These classes are the translation, so a script reads run.project_id and a
server rename changes this file, not every script. raw keeps the server's
JSON for anything not modelled here.
Project
dataclass
¶
Project(id: str, name: str, slug: str | None = None, description: str | None = None, visibility: str | None = None, taxonomy: dict[str, Any] = dict(), created_at: str | None = None, updated_at: str | None = None, raw: dict[str, Any] = dict())
A project, as Visin lists it.
direction ¶
direction(metric: str) -> str | None
higher or lower: which way is better for a result, as the project says, else None.
metric is a path such as val.mean_iou; the project names a result by that path or by its
last part. None means the project has not said, so any direction you assume is a guess.
from_json
classmethod
¶
from_json(data: Mapping[str, Any]) -> Project
The Project the server described in data.
Finding
dataclass
¶
Finding(id: str, project_id: str, title: str, body: str, recommendations: str | None = None, training_id: str | None = None, training_ids: tuple[str, ...] = (), author_kind: str | None = None, author_label: str | None = None, created_at: str | None = None, raw: dict[str, Any] = dict())
A conclusion someone (or an assistant) recorded about a project's runs.
from_json
classmethod
¶
from_json(data: Mapping[str, Any]) -> Finding
The Finding the server described in data.
Comparison
dataclass
¶
Comparison(id: str, uuid: str, name: str, type: str | None = None, description: str | None = None, item_ids: tuple[str, ...] = (), project_id: str | None = None, created_at: str | None = None, updated_at: str | None = None, raw: dict[str, Any] = dict())
A saved comparison of runs, test results, benchmarks or epochs.
from_json
classmethod
¶
from_json(data: Mapping[str, Any]) -> Comparison
The Comparison the server described in data.
StoredModel
dataclass
¶
StoredModel(id: str, training_id: str, format: str, filename: str | None = None, size: int | None = None, sha256: str | None = None, source_sha256: str | None = None, epoch: int | None = None, status: str | None = None, verify_error: str | None = None, license: dict[str, Any] = dict(), credit: str | None = None, model_info: dict[str, Any] = dict(), parity: dict[str, Any] = dict(), added_at: str | None = None, raw: dict[str, Any] = dict())
A checkpoint a run kept on Visin, in one format.
A .pth and the .onnx exported from it are two of these with the same source_sha256 (or,
for the
.pth, the same sha256): one model in two files. Visin serves a model only when it is ready
(its file has been read and matched) and has a license; :attr:downloadable says which.
Training
dataclass
¶
Training(id: str, uuid: str, name: str, status: str | None = None, description: str | None = None, project_id: str | None = None, dataset_id: str | None = None, config_id: str | None = None, tags: tuple[str, ...] = (), metadata: dict[str, Any] = dict(), start_time: str | None = None, end_time: str | None = None, created_at: str | None = None, updated_at: str | None = None, metrics: dict[str, Any] = dict(), models: tuple[dict[str, Any], ...] = (), provenance: dict[str, Any] = dict(), notes: str | None = None, raw: dict[str, Any] = dict())
A run. uuid is the one you chose; id is the database's.
MetricSummary
dataclass
¶
MetricSummary(path: str, direction: str, direction_from: str, best_value: float, best_epoch: int, last_value: float, last_epoch: int)
One result of a run: its path in an epoch, which way is better, and the best epoch beside the last.
direction_from is taxonomy when the project said which way is better, and default when Visin
guessed from the name (a loss or a latency is lower-is-better, the rest higher): check it before trusting
a guess.
from_json
classmethod
¶
from_json(data: Mapping[str, Any]) -> MetricSummary
The MetricSummary the server described in data.
Summary
dataclass
¶
Summary(training: Training, epoch_count: int, last_epoch: int | None, metrics: tuple[MetricSummary, ...] = (), models: tuple[dict[str, Any], ...] = (), provenance: dict[str, Any] = dict(), raw: dict[str, Any] = dict())
How a run did, in one call: the run, its models and provenance, and the best epoch of every result.
Epoch
dataclass
¶
Epoch(epoch: int | float, uuid: str | None = None, training_uuid: str | None = None, results: dict[str, Any] = dict(), timestamp: str | None = None, learning_rate: float | None = None, epoch_time: float | None = None, metadata: dict[str, Any] = dict(), raw: dict[str, Any] = dict())
One epoch of a run, with everything it reported in results.
from_json
classmethod
¶
from_json(data: Mapping[str, Any]) -> Epoch
The Epoch the server described in data.
TestResult
dataclass
¶
TestResult(test_uuid: str | None = None, epoch_uuid: str | None = None, training_uuid: str | None = None, results: dict[str, Any] = dict(), created_at: str | None = None, id: str | None = None, raw: dict[str, Any] = dict())
Scores on held-out data, recorded against an epoch: an evaluation a run reported.
from_json
classmethod
¶
from_json(data: Mapping[str, Any]) -> TestResult
The TestResult the server described in data: an evaluation, or a test result in older words.
Benchmark
dataclass
¶
Benchmark(id: str, results: Any = None, system_info: dict[str, Any] = dict(), training_uuid: str | None = None, epoch_uuid: str | None = None, created_at: str | None = None, raw: dict[str, Any] = dict())
How a model ran on a machine: results per setting, and the system_info it ran on.
from_json
classmethod
¶
from_json(data: Mapping[str, Any]) -> Benchmark
The Benchmark the server described in data.
Configuration
dataclass
¶
Configuration(id: str, uuid: str | None = None, name: str | None = None, training_id: str | None = None, config: dict[str, Any] = dict(), created_at: str | None = None, raw: dict[str, Any] = dict())
What a run was launched with. config is the dict that was logged.
from_json
classmethod
¶
from_json(data: Mapping[str, Any]) -> Configuration
The Configuration the server described in data.
Visualization
dataclass
¶
Visualization(uuid: str, epoch_uuid: str | None = None, training_uuid: str | None = None, kind: str | None = None, filename: str | None = None, url: str | None = None, created_at: str | None = None, raw: dict[str, Any] = dict())
A frame stored against an epoch. url is a signed link, present when it was asked for.
from_json
classmethod
¶
from_json(data: Mapping[str, Any]) -> Visualization
The Visualization the server described in data.
Dataset
dataclass
¶
Dataset(id: str, name: str, size: int | None = None, filename: str | None = None, raw: dict[str, Any] = dict())
A dataset on Visin. size is the zip's size in bytes, when it has one.
source
property
¶
source: dict[str, Any] | None
The Hugging Face repo and commit this dataset lives at, when it is kept there.
downloadable
property
¶
downloadable: bool
Whether there is something to download: a zip on Visin, or a Hub repo.
from_json
classmethod
¶
from_json(data: Mapping[str, Any]) -> Dataset
The Dataset the server described in data.
Reason
dataclass
¶
Reason(code: str, detail: str | None = None)
One thing the server found wrong with a result, or worth saying about it.
code is stable (missing-condition, protocol-mismatch) and each is explained, with how to fix
it, in Visin's documentation under Why is my result unranked?. detail says what it is about: a
condition such as night, or night/mIoU.
from_json
classmethod
¶
from_json(data: Mapping[str, Any]) -> Reason
The Reason the server described in data.
Verdict
dataclass
¶
Verdict(state: str, reasons: tuple[Reason, ...] = (), warnings: tuple[Reason, ...] = (), scores: dict[str, Any] | None = None, evidence: str | None = None)
Whether a result can be ranked on its suite, and why not.
state is eligible (it can be ranked), incomplete (something the suite requires is missing),
incompatible (it measured something other than the suite measures), exploratory (no suite was
named) or legacy-unverified. scores holds the numbers the ranking uses, for an eligible result.
evidence labels the result, and never decides whether it is ranked: observed (the evaluator sent
the data, protocol and evaluator it ran, and each matched), reported (ranked on the submitter's word,
with less than that), attested (a manager vouched for a promoted result) or none (no suite). It
is always the submitter's word.
Evaluation
dataclass
¶
Evaluation(id: str | None = None, uuid: str | None = None, project_id: str | None = None, checkpoint: dict[str, Any] = dict(), checkpoint_key: str | None = None, suite: str | None = None, suite_digest: str | None = None, status: str | None = None, verdict: Verdict | None = None, sample_counts: dict[str, int] = dict(), received_at: str | None = None, published_at: str | None = None, pending_approval: bool = False, hidden: dict[str, Any] | None = None, superseded_by: str | None = None, results: dict[str, Any] = dict(), stored: bool = True, queued: bool = False, raw: dict[str, Any] = dict())
One checkpoint scored on one suite version, and the server's verdict on it.
stored is false for a dry run, which has a verdict but no id. queued is true when the result
could not be sent and waits on disk for visin sync: it has no verdict yet.
ranked
property
¶
ranked: bool
Whether it can be ranked. False while it is queued, since nothing has judged it yet.
public
property
¶
public: bool
Whether anyone can see it now: published, not waiting for approval and not hidden.
from_json
classmethod
¶
from_json(data: Mapping[str, Any], *, stored: bool = True) -> Evaluation
The Evaluation the server described in data.
Suite
dataclass
¶
Suite(id: str, slug: str, version: int, name: str, description: str | None = None, project_id: str | None = None, visibility: str | None = None, digest: str | None = None, archived_at: str | None = None, protocol: dict[str, Any] = dict(), raw: dict[str, Any] = dict())
A published scoring protocol: results on one suite version can be compared.
A published version never changes. digest is the hash of protocol, so two suites with one digest
score identically.
Pagination
dataclass
¶
Pagination(page: int, limit: int, total: int, pages: int)
Where a page sits among the rows the server selected: total counts rows, pages counts pages.
from_json
classmethod
¶
from_json(data: object) -> Pagination | None
The pagination the server described in data, or None when it sent none.
LeaderboardEntry
dataclass
¶
LeaderboardEntry(rank: int, evaluation_id: str, checkpoint: dict[str, Any], headline: float, worst_condition: str, worst_value: float, gap: float, attempts: int, project: str | None = None, raw: dict[str, Any] = dict())
One ranked checkpoint: its rank, headline score, where it is weakest, and the evidence.
Leaderboard
dataclass
¶
Leaderboard(suite: str, headline_key: str, direction: str, candidates: int, entries: tuple[LeaderboardEntry, ...] = (), unranked: tuple[dict[str, Any], ...] = (), public: bool = False, pagination: Pagination | None = None, unranked_pagination: Pagination | None = None, complete: bool = False, raw: dict[str, Any] = dict())
The ranking of one suite version.
One entry per checkpoint, from its latest ranked attempt and never its best. candidates is how many
evaluations were in the pool a rank is a position in (attempts of any outcome, not everything ever
run), while pagination.total is how many checkpoints were selected from it: the two differ whenever
a checkpoint has more than one attempt. Ranks are global, so page 2 starts at the rank the pool gives
it, never at 1.
unranked holds what has attempts but none ranked, with the reasons, paged apart from the ranking in
unranked_pagination: only the signed-in view has it, never the public one. complete says
entries and unranked cover every page rather than the one that was asked for.
from_json
classmethod
¶
from_json(data: Mapping[str, Any], *, public: bool = False) -> Leaderboard
The Leaderboard the server described in data.