Evaluation and suites¶
evaluate ¶
evaluate(results: Mapping[str, Any], *, suite: str | None, checkpoint: Mapping[str, Any] | None, project: str | None = None, sample_counts: Mapping[str, int] | None = None, run: Any = None, epoch: int | None = None, epoch_uuid: str | None = None, evaluator: Mapping[str, Any] | None = None, provenance: Mapping[str, Any] | None = None, evidence: Mapping[str, Any] | None = None, data: Mapping[str, Any] | None = None, protocol: str | Path | Mapping[str, Any] | None = None, protocol_digest: str | None = None, classes: Mapping[str, Any] | None = None, uuid: str | None = None, supersedes: Evaluation | str | None = None, executed_at: str | datetime | None = None, status: str = 'completed', dry_run: bool = False, url: str | None = None, token: str | None = None, directory: str | Path | None = None) -> Evaluation
Record one checkpoint's results on a suite version, and return Visin's verdict.
results is the condition → class → metric blob your evaluator produced; the suite names which
metrics count, and everything else is kept. suite is "slug@version" (a concrete version:
latest moves). checkpoint comes from :func:local_checkpoint or :func:hub_checkpoint.
sample_counts says how many samples each condition scored: a suite pins those, so a result that does
not say, or says a different number, is not ranked. project defaults to VISIN_PROJECT, then to
the project a pipeline key is limited to.
run (a run, or its UUID), epoch and epoch_uuid say where the checkpoint came from. That is
provenance only; it does not say which bytes were scored, which checkpoint does. evaluator is
what produced the numbers, such as {"package": "visin-fusion", "version": "1.4.2", "commit":
"9d1c2ab"}, and provenance is anything else worth keeping. Visin refuses credentials in it.
Send your own uuid to make a repeat harmless: the same result again returns the stored one, and a
different result under that uuid raises. supersedes names the evaluation this one corrects.
You need none of the following to be ranked. A result can also say what it actually ran, and Visin
compares each part with the suite: with the data, the protocol and the evaluator all given (and matching)
the row is labelled observed, with less it is reported. data is the identity of the data your
evaluator read, in the shape the suite pins it:
{"kind": "external", "manifestSha256": ...} (see :func:~visin.manifest_digest),
{"kind": "hf", "repo": ..., "commit": ...} or {"kind": "visin", "archiveSha256": ...}.
protocol is the protocol file (or dict) your evaluator loaded: its digest is asked of Visin, and
waits until visin sync when Visin cannot be reached. protocol_digest is that digest when you
already have it (see :func:~visin.check_protocol). The evaluator's package and version come from
evaluator. classes is {"scored": [...], "ignored": [...]}, compared when given. evidence
is the whole object for anything else; the arguments above are merged over it. What you give must match
the suite, or the result is incompatible with a reason that names the part. None of it is verified:
it is what your evaluator reported.
Results are not lost to a bad network. With VISIN_MODE=offline, or when Visin cannot be reached, the
result waits on disk and visin sync sends it; the returned evaluation then has queued=True and no
verdict. A refusal (an unknown suite, a bad body, no access) raises :class:~visin.errors.ApiError.
With nothing configured the call does nothing and returns an evaluation that was not stored.
dry_run=True stores nothing and returns the verdict only (stored=False, no id). It needs a server.
local_checkpoint ¶
local_checkpoint(path: str | Path, label: str | None = None) -> dict[str, Any]
A checkpoint held on this machine, named by the digest of the weights that will be loaded.
Only the digest and the label are sent, never the path or the file. Two evaluations of the same weights are the same model whatever the file is called or wherever it moves. Hash the file you actually load.
hub_checkpoint ¶
hub_checkpoint(repo: str, commit: str, path: str | None = None) -> dict[str, Any]
A checkpoint on the Hugging Face Hub, pinned to a full commit.
commit is the 40-character hash, never a branch or a tag: a branch moves, and a result must keep
naming the bytes it scored. path is a file or folder inside the repo when the model is not all of
it. The project's storage must be Hugging Face, or Visin refuses a Hub checkpoint.
sha256_of ¶
sha256_of(path: str | Path) -> str
The SHA-256 of a file, read in blocks so a large checkpoint never has to fit in memory.
promote ¶
promote(evaluation: Evaluation | TestResult | str, *, suite: str, checkpoint: Mapping[str, Any], sample_counts: Mapping[str, int], url: str | None = None, token: str | None = None) -> Evaluation
Rank a result recorded without a suite by copying it onto one, without running anything again.
evaluation is the evaluation to copy, or a test result (which is one), or its id. It is not changed. A
result recorded without a suite never said which checkpoint it was or how many samples each condition
scored, so you say both here, and what you say is yours to vouch for. Needs manage access to the result's
project. Promoting the same result onto the same suite twice returns the first.
publish ¶
publish(evaluation: Evaluation | str, *, url: str | None = None, token: str | None = None) -> Evaluation
Put one ranked evaluation on its suite's public leaderboard.
Needs manage access to the project. The evaluation must be ranked, in a public project, on a public suite that takes your project's results, and the checkpoint's latest ranked result on it, or Visin refuses and says which: a board shows each model by its latest result, and a newer ranked result withdraws the older one. On a suite that approves submissions, another project's result waits for a manager of the suite before it is shown. What becomes public is a fixed set of fields: scores, checkpoint, sample counts and the evaluator, never commands, hosts or configuration. Safe to repeat.
withdraw ¶
withdraw(evaluation: Evaluation | str, *, url: str | None = None, token: str | None = None) -> Evaluation
Take an evaluation off the public leaderboard at once. Safe to repeat.
manifest_digest ¶
manifest_digest(conditions: Mapping[str, Iterable[str]]) -> str
The identity of the samples a suite scores: the SHA-256 of one canonical manifest.
conditions maps each condition's name to the samples scored in it (file names, frame ids: whatever
names a sample on your side). The manifest is those lists sorted, under sorted condition names, as compact
JSON, so the same data gives the same digest in any order and on any machine, and a changed, added or
removed sample changes it. Put it in the suite as data.manifestSha256; visin suites manifest
prints it from split files.
read_split ¶
read_split(path: str | Path) -> list[str]
The samples a split file lists: one per line, blank lines and surrounding spaces left out.
push_suite ¶
push_suite(source: str | Path | Mapping[str, Any], *, project: str | None = None, visibility: str | None = None, url: str | None = None, token: str | None = None) -> Suite
Publish a suite version from a file or a dict, and return it.
project (an id or slug) is the project the suite belongs to and whose access rules apply; it
defaults to the file's own project, then VISIN_PROJECT, then the project a pipeline key is
limited to. visibility is private (the project's readers) or public (anyone may read the
protocol, which grants nothing in the project); left out, it is the file's, then the previous version's,
then private. Needs contribute access.
load_suite ¶
load_suite(path: str | Path) -> dict[str, Any]
Read a suite file: JSON, or YAML (.yml or .yaml, needs PyYAML).
check_protocol ¶
check_protocol(source: str | Path | Mapping[str, Any], *, url: str | None = None, token: str | None = None) -> str
The digest Visin gives a protocol, without publishing it.
An evaluator sends this as protocol_digest to say which protocol it ran. The digest is computed by
the server, which fills in defaults and sorts keys, so it is never reimplemented here and always equals
the digest of the suite once published. Needs a server and a login, and stores nothing.
protocol_of ¶
protocol_of(source: str | Path | Mapping[str, Any]) -> dict[str, Any]
The protocol a suite file, a suite dict or a bare protocol dict holds.
A path is read as a suite file (see :func:load_suite). Whatever is given, the result is the protocol
itself: the protocol of a suite, or the dict as it came when it has no such key.
diff_evaluations ¶
diff_evaluations(baseline: Evaluation, candidate: Evaluation, suite: Suite, *, max_drop: float = 0.0, all_metrics: bool = False) -> Diff
Diff the candidate against the baseline on suite, score by score.
The headline metric is compared for the overall figure and every condition; all_metrics=True compares
every metric the suite names. A score that fell by more than max_drop (in the metric's own units) is
a regression. Raises ValueError when the two evaluations are not on this suite version, or not on
the same protocol, which cannot be compared.
Diff
dataclass
¶
Diff(suite: str, baseline: str | None, candidate: str | None, max_drop: float, changes: tuple[Change, ...], problems: tuple[str, ...] = ())
Every compared score, and what the diff decides.
regressions are scores that fell by more than max_drop; missing are scores one side lacks,
including every score of an evaluation that could not be ranked. passed needs neither.
Change
dataclass
¶
Change(scope: str, metric: str, direction: str, baseline: float | None, candidate: float | None, improvement: float | None, status: str)
One score in both evaluations: scope is a condition name or overall.
improvement is positive when the candidate is better in the metric's direction, negative when it is
worse, and None when either evaluation has no value (status is then missing).