diff --git a/AGENTS.md b/AGENTS.md index b7b63b3e..97c449d7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -132,6 +132,37 @@ Conventions that matter for correctness: supports both `hp.get("lr")` and `hp["lr"]` (subscript == `.get`), and stays live — reads reflect in-place updates and re-registration. +Recording values (pick by what the value is *about*, not by convenience): + +| Value describes | Verb | Example | +|---|---|---| +| one sample | `wl.save_signals(signals={...}, batch_ids=ids, ...)` | an image's loss | +| one annotation | `wl.save_instance_signals(...)` | one box's IoU | +| a group of samples | `wl.save_group_signals(signals={...}, group_ids=[...])` | a pair's contrastive loss | +| one training **step** | `wl.save_model_signals(signals={...})` | a gradient norm | + +The first three write dataframe rows (sortable/filterable in the grid); the +fourth only plots a curve. Never fake a step-level value by broadcasting it +across `batch_ids` — that writes a number onto samples it was never about. + +- **Training-dynamics signals** come free with + `wl.watch_or_edit(model, flag="model", track_model_signals=True, + model_signals_every_n_steps=N)`. That emits, per step and with no call in the + training loop: `metrics/global/{grad_norm,weights_norm}` and + `metrics/layer//{grad_norm,weights_norm,activation_mean, + activation_std,activation_max,activation_min}`. `` is the same id + architecture ops (freeze/reset) address, so a bad curve names the layer to act + on. Only collects inside `guard_training_context`, so eval never contaminates + it. Implementation: `weightslab/weightslab/components/model_signals.py`; + example: `examples/Usecases/wl-fashion-mnist-signals`. + - Diagnosing from these: early-layer `grad_norm` → 0 with healthy late layers + = vanishing gradient; `grad_norm` spiking orders of magnitude = exploding; + `activation_std` → 0 on a layer = that layer went constant (dead + ReLU/saturated BN); `weights_norm` growing while loss flattens = needs decay. + - Per-layer curves need per-layer modules: an `nn.Sequential` block resolves + to ONE layer id, so a model built out of Sequentials gets one curve for the + whole block. + --- ## 4. Configuration (environment variables) diff --git a/docs/_static/examples-gallery.js b/docs/_static/examples-gallery.js index d610919b..7a4c4697 100644 --- a/docs/_static/examples-gallery.js +++ b/docs/_static/examples-gallery.js @@ -82,6 +82,13 @@ tags: ['loss analysis', 'signal', 'categorical tag', 'per-sample', 'trajectory'], url: 'examples/usecases/loss_shape_classification.html', colab: COLAB + 'Usecases/wl-segmentation-loss-shapes-classification.ipynb' + }, + { + badge: 'Usecase', color: 'usecase', + title: 'Model Signals — Fashion-MNIST', + desc: 'Per-step training dynamics: global and per-layer gradient norms, weight norms and activation statistics, from one argument on the model wrap.', + tags: ['model signals', 'gradient norm', 'activations', 'per-layer', 'training dynamics'], + url: 'examples/usecases/model_signals.html' } ]; diff --git a/docs/examples/usecases/index.rst b/docs/examples/usecases/index.rst index 282cfe84..8155053d 100644 --- a/docs/examples/usecases/index.rst +++ b/docs/examples/usecases/index.rst @@ -4,7 +4,7 @@ Specific User Usecases ====================== Task-specific integrations that go beyond the standard loop: point-cloud -inputs and per-sample loss trajectory analysis. +inputs, per-sample loss trajectory analysis, and per-layer training dynamics. .. raw:: html @@ -18,3 +18,4 @@ inputs and per-sample loss trajectory analysis. lidar_detection loss_shape_classification + model_signals diff --git a/docs/examples/usecases/model_signals.rst b/docs/examples/usecases/model_signals.rst new file mode 100644 index 00000000..fb5cc2be --- /dev/null +++ b/docs/examples/usecases/model_signals.rst @@ -0,0 +1,162 @@ +Model Signals on Fashion-MNIST +=============================== + +.. raw:: html + +
+ Usecase + model signals + gradient norm + activations + per-layer + training dynamics +
+ +**Example:** ``weightslab/examples/Usecases/wl-fashion-mnist-signals`` + +This use case trains a small CNN on Fashion-MNIST and adds one thing on top of +the plain per-sample logging: the run plots **its own training dynamics**. A +loss curve tells you whether the model is learning; these curves tell you +*where* in the model something went wrong. + +Everything below comes from one argument. + +The integration +--------------- + +.. code-block:: python + + model = wl.watch_or_edit( + FashionCNN(), + flag="model", + device=device, + track_model_signals=True, # <- the whole feature + model_signals_every_n_steps=1, + ) + +No hooks to write, and **no call anywhere in the training loop** — the loop is +byte-for-byte the same as ``wl-classification``'s. Pass a list instead of +``True`` to narrow the set, e.g. ``track_model_signals=["grad_norm", +"activation_std"]``. + +What gets plotted +----------------- + +.. code-block:: text + + metrics/global/grad_norm whole-model gradient L2 norm + metrics/global/weights_norm whole-model parameter L2 norm + metrics/layer//grad_norm per-layer parameter gradients + metrics/layer//weights_norm per-layer parameters + metrics/layer//activation_mean + metrics/layer//activation_std + metrics/layer//activation_max + metrics/layer//activation_min + +Layers with parameters get all eight; parameter-free layers (``ReLU``, +``MaxPool2d``) get the four activation curves only. Containers and shape-only +ops (``Sequential``, ``Flatten``, ``Identity``, ``Dropout``) are skipped, since +their output statistics duplicate the layer before them. + +For the model in this example — three conv blocks and a two-layer head — that +is 74 curves: 14 layers × 4 activation stats, 8 parameterized layers × 2 norms, +and the 2 global norms. + +The layer legend +---------------- + +``metrics/layer/7/grad_norm`` says nothing on its own, so the example prints the +mapping at startup: + +.. code-block:: text + + layer_id module shape + 1 Conv2d (16, 1, 3, 3) + 2 BatchNorm2d (16,) + 3 ReLU - + 4 MaxPool2d - + 5 Conv2d (32, 16, 3, 3) + 6 BatchNorm2d (32,) + 7 ReLU - + 8 MaxPool2d - + 9 Conv2d (64, 32, 3, 3) + 10 BatchNorm2d (64,) + 11 ReLU - + 12 Flatten - + 13 Linear (128, 3136) + 14 ReLU - + 15 Linear (10, 128) + +These are the same ids the model panel and every architecture op (freeze / +reset / operate) use — so a curve that looks wrong names the layer you then act +on, whether from the UI, the CLI, or the agent. + +Note that every module in this example's model is a **named attribute** rather +than a member of an ``nn.Sequential``. That is deliberate: a Sequential block +resolves to one layer id, and therefore one curve, which defeats the purpose of +per-layer signals. + +Reading the curves +------------------ + +Fashion-MNIST is small enough to make each failure mode legible: + +.. list-table:: + :header-rows: 1 + :widths: 34 66 + + * - What you see + - What it means + * - ``grad_norm`` collapsing toward 0 in the **early** layers while the late + ones stay healthy + - Vanishing gradient. The run keeps "training" and stops learning. Act + from the layer where it dies. + * - ``grad_norm`` spiking by orders of magnitude + - Exploding gradient. Compare against the loss curve to see which moved + first. + * - ``activation_std`` → 0 on a layer + - That layer has gone constant (dead ReLUs, saturated BatchNorm). Still + consuming compute, contributing nothing. + * - ``activation_min`` pinned at exactly 0.0 across a whole ReLU + - The same story from the other side — nothing is getting through. + * - ``weights_norm`` climbing without bound while the loss flattens + - The model is growing weights instead of learning structure. Add decay. + +Cost, and how it is kept low +---------------------------- + +Three things keep the per-step overhead small enough to leave on by default: + +- **Activations are reduced on-device** into 0-d tensors and held there. The + whole step costs *one* host↔device sync no matter how many layers are + tracked. +- **Gradients are captured by post-accumulate hooks**, so nothing walks the + parameter list a second time — and nothing depends on where your loop calls + ``optimizer.zero_grad()``. +- **``model_signals_every_n_steps``** samples every Nth step. On a large model, + 10–50 makes the cost negligible while the curves stay just as readable. Reach + for this before dropping metrics. + +Collection only happens inside ``guard_training_context``, so the evaluation +pass contributes nothing — a gradient or activation curve never contains values +the optimizer did not see. This holds even for eval loops that skip +``model.eval()`` or ``torch.no_grad()``. + +Custom dynamics values +---------------------- + +``track_model_signals`` is a collector over ``wl.save_model_signals``, which is +the step-keyed write path in its own right. Use it directly for anything the +collector does not compute: + +.. code-block:: python + + # gradient-to-weight ratio: how big a step is this, relative to the weights? + wl.save_model_signals({ + "metrics/global/update_ratio": grad_norm / (weight_norm + 1e-12), + "metrics/global/lr": optimizer.param_groups[0]["lr"], + }) + +See :ref:`save_model_signals ` for the full reference, and +:doc:`../../model_interaction` for how these fit alongside the rest of the +model surface. diff --git a/docs/index.rst b/docs/index.rst index d01c8fa5..85f53b71 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -129,7 +129,7 @@ Weightslab is a Python SDK to inspect, monitor, and edit training behavior for c .. toctree:: :maxdepth: 2 - :caption: Getting Started + :caption: GETTING STARTED :hidden: quickstart @@ -137,7 +137,7 @@ Weightslab is a Python SDK to inspect, monitor, and edit training behavior for c .. toctree:: :maxdepth: 2 - :caption: Usage + :caption: USAGE :hidden: usage/good_practice @@ -145,7 +145,7 @@ Weightslab is a Python SDK to inspect, monitor, and edit training behavior for c .. toctree:: :maxdepth: 3 - :caption: Examples + :caption: EXAMPLES :hidden: examples/index @@ -153,7 +153,7 @@ Weightslab is a Python SDK to inspect, monitor, and edit training behavior for c .. toctree:: :maxdepth: 2 - :caption: Core Concepts + :caption: CORE CONCEPTS :hidden: four_way_approach @@ -171,7 +171,7 @@ Weightslab is a Python SDK to inspect, monitor, and edit training behavior for c .. toctree:: :maxdepth: 2 - :caption: External Library Integration + :caption: INTEGRATIONS :hidden: pytorch_lightning @@ -180,7 +180,7 @@ Weightslab is a Python SDK to inspect, monitor, and edit training behavior for c .. toctree:: :maxdepth: 1 - :caption: Configuration + :caption: CONFIGURATION :hidden: configuration @@ -188,9 +188,20 @@ Weightslab is a Python SDK to inspect, monitor, and edit training behavior for c .. toctree:: :maxdepth: 2 - :caption: Reference + :caption: REFERENCE :hidden: user_functions user_commands grpc/index + + +.. toctree:: + :maxdepth: 2 + :caption: MIGRATION + :hidden: + + From Weights & Biases + From Voxel 51 + From Tensorboard + \ No newline at end of file diff --git a/docs/logger.rst b/docs/logger.rst index d9e07f5b..94c05241 100644 --- a/docs/logger.rst +++ b/docs/logger.rst @@ -8,8 +8,68 @@ What gets logged - Scalar signals (losses, metrics) - Per-sample signal vectors +- Per-step **model** signals (gradient/weight norms, activation statistics) - Optional predictions/targets for deeper analysis +Two kinds of signal +------------------- + +Signals divide by what a value is *about*, and that decides which verb records +it: + +.. list-table:: + :header-rows: 1 + :widths: 22 30 48 + + * - Keyed by + - Verb + - Example + * - Sample + - ``wl.save_signals`` + - The classification loss of one image. + * - Annotation + - ``wl.save_instance_signals`` + - The IoU of one bounding box. + * - Group + - ``wl.save_group_signals`` + - A contrastive loss over an image pair. + * - **Step** + - ``wl.save_model_signals`` + - The gradient norm of layer 5 at step 900. + +The first three write onto dataframe rows; the sample grid can then be sorted +and filtered by them. The fourth does not — a gradient norm belongs to the +optimization step that produced it, not to any of the samples in the batch, so +it is plotted as a curve and nothing else. Recording it with ``save_signals`` +would mean broadcasting one number across a whole batch of ids and polluting +every one of those samples' history with a value that was never about them. + +Default plot order +------------------ + +The plots board groups curves by signal-name prefix, in this order: + +1. **Your experiment's signals** — losses, metrics, and the whole-model + ``metrics/global/*`` norms. These are what the board is for, so they stay at + the top. +2. **Per-layer model signals** — everything under ``metrics/layer/`` + (see :ref:`track_model_signals `). +3. **Resource monitors** — everything under ``resource/`` (CPU, memory, disk, + network, GPU and process telemetry). + +The grouping exists because arrival order stops being usable once model signals +are on: ``track_model_signals`` can emit dozens of ``metrics/layer/*`` curves in +a single step (74 for the Fashion-MNIST example) and resource monitoring is +enabled by default, so an unordered board buries the loss curve under +telemetry. Note that ``metrics/global/*`` deliberately sits in the *first* +group — a whole-model gradient norm is read next to the loss, not scrolled past +70 per-layer curves. + +This is only a default. Dragging a card puts it exactly where you drop it and +that arrangement is remembered, per browser; signals that appear later (a +``metrics/layer/*`` curve showing up once training starts) are filed into their +group without disturbing anything you have already arranged. + Start services -------------- diff --git a/docs/model_interaction.rst b/docs/model_interaction.rst index ab1fc609..338e832f 100644 --- a/docs/model_interaction.rst +++ b/docs/model_interaction.rst @@ -14,6 +14,8 @@ Why it matters -------------- - Observe training signals at batch/sample granularity. +- Watch the model's own training dynamics — gradients, weights, activations — + per layer and per step (see `Training-dynamics signals`_). - Keep a stable ledger/proxy handle across runtime updates. - Enable dynamic controls without rewriting your loop architecture. @@ -36,9 +38,71 @@ Minimal example log=True, ) +Training-dynamics signals +------------------------- + +A loss curve tells you *whether* the model is learning. It does not tell you +**where** in the model something went wrong. Wrapping the model with +``track_model_signals=True`` adds that second view — one curve per layer, per +step, for the three quantities that explain most training failures: + +.. code-block:: python + + model = wl.watch_or_edit( + my_model, + flag="model", + device="cuda", + track_model_signals=True, # or a list, e.g. ["grad_norm"] + model_signals_every_n_steps=1, # raise to 10-50 on a large model + ) + +That is the whole integration. Nothing is added to the training loop: gradients +are captured by post-accumulate hooks the moment they are final, activations by +forward hooks, and the set is flushed once per step just before +``optimizer.step()`` consumes it. + +.. code-block:: text + + metrics/global/grad_norm whole-model gradient L2 norm + metrics/global/weights_norm whole-model parameter L2 norm + metrics/layer//grad_norm per-layer parameter gradients + metrics/layer//weights_norm per-layer parameters + metrics/layer//activation_{mean,std,max,min} + +```` is the same module id the model panel and every architecture op +(freeze / reset / operate) use, so a curve that looks wrong names the layer you +then act on. + +What each one catches: + +- ``grad_norm`` collapsing toward 0 in the **early** layers while the late ones + stay healthy is a vanishing gradient — the run keeps "training" and stops + learning. Freeze or reinitialize from the layer where it dies. +- ``grad_norm`` spiking by orders of magnitude is the exploding case; compare + against the loss curve to see which moved first. +- ``activation_std`` → 0 on a layer means that layer has gone constant (dead + ReLUs, saturated BatchNorm). It is still consuming compute and contributing + nothing. +- ``weights_norm`` climbing without bound while the loss flattens is the model + growing weights instead of learning structure — time to add decay. + +Collection only happens inside ``guard_training_context``, so an evaluation +pass can never contaminate these curves with values the optimizer never saw. +For a dynamics value of your own (a gradient-to-weight ratio, a custom norm), +write it directly with ``wl.save_model_signals({...})``. + +Full reference: :ref:`track_model_signals `. Runnable example: +``examples/Usecases/wl-fashion-mnist-signals``. + Best practices -------------- - Use explicit names for losses/metrics to keep logs readable. - Prefer ``per_sample=True`` for losses when you need hard-example analysis. - Keep model/device arguments explicit to avoid ambiguity in multi-device setups. +- Give each layer its own attribute (rather than burying it in an + ``nn.Sequential``) if you want per-layer curves — a Sequential block resolves + to a single layer id, and therefore a single curve. +- Raise ``model_signals_every_n_steps`` before dropping metrics: the activation + forward hooks are the only per-step cost worth thinking about, and sampling + every 10th step keeps the curves just as readable. diff --git a/docs/user_functions.rst b/docs/user_functions.rst index d55a4559..dd7d63b2 100644 --- a/docs/user_functions.rst +++ b/docs/user_functions.rst @@ -18,6 +18,8 @@ Public API surface - ``wl.save_signals`` - ``wl.save_instance_signals`` *(per-instance / per-annotation signals)* - ``wl.save_group_signals`` *(group-level signals, e.g. pair/contrastive losses)* +- ``wl.save_model_signals`` *(per-step model signals, e.g. gradient norms)* +- ``wl.track_model_signals`` *(collects the above automatically via hooks)* - ``wl.tag_samples`` - ``wl.register_categorical_tag`` *(multi-value tags)* - ``wl.set_categorical_tag`` *(multi-value tags)* @@ -77,6 +79,23 @@ Register or wrap models, data loaders, optimizers, loggers, losses/metrics, and optimizer = wl.watch_or_edit(optim.Adam(model.parameters(), lr=1e-3), flag="optimizer") train_loss = wl.watch_or_edit(nn.CrossEntropyLoss(reduction="none"), flag="loss", signal_name="train-loss") +**Model kwargs for training-dynamics signals** + +- ``track_model_signals`` — ``True`` for every model signal, or a list to + narrow the set (e.g. ``["grad_norm", "activation_std"]``). Installs the hooks + that plot gradient norms, weight norms and activation statistics per layer; + see :ref:`track_model_signals `. +- ``model_signals_every_n_steps`` *(int, default 1)* — sample those signals + every Nth step. +- ``model_signals_layer_ids`` *(iterable, optional)* — restrict them to + specific layer ids. + +.. code-block:: python + + model = wl.watch_or_edit(my_model, flag="model", device="cuda", + track_model_signals=True, + model_signals_every_n_steps=10) + Hyperparameters via YAML path ----------------------------- @@ -704,6 +723,175 @@ If any member of a group is discarded, the group's signal update for that group is skipped for that call (per-sample signals are unaffected — only the group-level write is suppressed). +.. _model-signals: + +save_model_signals +------------------- + +**Signature** + +.. code-block:: python + + wl.save_model_signals(signals, step=None) + +**Purpose** + +Persist **per-step** scalars that describe the *model*, not any sample — the +step-keyed sibling of the three verbs above. ``save_signals`` (per sample), +``save_instance_signals`` (per annotation) and ``save_group_signals`` (per +group) all write onto dataframe rows, because every value they record belongs +to something in the dataset. A gradient norm does not: it belongs to the +optimization step that produced it, and the batch behind it is incidental. + +Nothing here touches the dataframe. Each value becomes one point on its own +signal curve, plotted exactly like a watched loss. + +Use it for training-dynamics values: gradient norms, weight norms, activation +statistics, learning rate, gradient-to-weight ratios. Reaching for +``save_signals`` instead means broadcasting one number across a whole batch of +``batch_ids``, which pollutes every one of those samples' history with a value +that was never about them. + +**Arguments** + +- ``signals`` *(dict)* — ``{name: value}``. Values may be Python numbers, or + 0-d / reducible tensors and arrays (mean-reduced to one scalar). Non-finite + values (NaN/inf) are dropped rather than plotted, so a diverging run breaks + the curve instead of rescaling the axis and hiding every healthy point + before it. +- ``step`` *(int, optional)* — training step; defaults to the current model + age, same as every other ``save_*`` verb. + +**Naming** + +``/`` is a path separator in the plots board, so the name is what groups the +curves. The convention the shipped examples use: + +.. code-block:: text + + metrics/global/ whole-model values + metrics/layer// per-layer values + +```` is the module id WeightsLab already assigns for architecture +ops (``get_module_id()`` / ``NetworkWithOps.get_layer_by_id``), so a layer's +curve and that same layer's freeze/reset controls name the same thing. + +**Typical usage** + +.. code-block:: python + + # straight after backward(), before zero_grad() + total = sum(p.grad.pow(2).sum() for p in model.parameters() if p.grad is not None) + wl.save_model_signals({"metrics/global/grad_norm": total.sqrt()}) + +In practice you rarely write that loop — see ``track_model_signals`` below. + +track_model_signals +-------------------- + +**Signature** + +.. code-block:: python + + wl.track_model_signals(model=None, metrics=METRICS, every_n_steps=1, + layer_ids=None, include_global=True) + + # or, equivalently, on the wrap itself: + wl.watch_or_edit(net, flag="model", track_model_signals=True, + model_signals_every_n_steps=1) + +**Purpose** + +Instrument a watched model so its training dynamics log themselves through +``save_model_signals``. One argument, no hooks to write, and **no call anywhere +in the training loop**. + +**Signals emitted** + +.. list-table:: + :header-rows: 1 + :widths: 34 66 + + * - Signal + - Meaning + * - ``metrics/global/grad_norm`` + - Whole-model gradient L2 norm. + * - ``metrics/global/weights_norm`` + - Whole-model parameter L2 norm. + * - ``metrics/layer//grad_norm`` + - That layer's parameter gradients, L2. + * - ``metrics/layer//weights_norm`` + - That layer's parameters, L2. + * - ``metrics/layer//activation_mean`` + - Mean of that layer's output. + * - ``metrics/layer//activation_std`` + - Standard deviation of that layer's output. + * - ``metrics/layer//activation_max`` + - Maximum of that layer's output. + * - ``metrics/layer//activation_min`` + - Minimum of that layer's output. + +Global norms combine correctly across layers (an L2 over the whole parameter +vector, not a sum of per-layer norms). Layers without parameters get +activation curves only; containers and shape-only ops (``Sequential``, +``Flatten``, ``Identity``, ``Dropout``) are skipped, since their output +statistics just duplicate the layer before them. + +**Arguments** + +- ``model`` — the watched model (what ``watch_or_edit(..., flag="model")`` + returned). Resolved from the ledger when omitted. +- ``metrics`` *(iterable of str)* — which signals to emit; defaults to all of + them. Narrow it with e.g. ``["grad_norm", "activation_std"]``. +- ``every_n_steps`` *(int, default 1)* — sample every Nth step. The activation + forward hooks are the only per-step cost worth thinking about; on a large + model raise this to 10–50 and the overhead becomes negligible while the + curves stay just as readable. +- ``layer_ids`` *(iterable, optional)* — restrict to these layer ids. + ``None`` tracks every layer. +- ``include_global`` *(bool, default ``True``)* — also emit the two + ``metrics/global/*`` curves. + +**Returns** a ``ModelSignalTracker``. Keep it if you want ``.flush()`` or +``.remove()``; ignoring it is fine, the hooks are already installed. + +**When each value is collected** + +- **Weights** are read off ``p.data`` at flush time — they are always there. +- **Gradients** come from ``Tensor.register_post_accumulate_grad_hook`` + (torch ≥ 2.1), which fires the instant a parameter's ``.grad`` is final + during backward. They are deliberately *not* read at flush time: a training + loop is free to call ``optimizer.zero_grad()`` before anything WeightsLab + controls runs again. +- **Activations** come from forward hooks, reduced on-device into 0-d tensors + and held there. The whole step costs **one** host↔device sync no matter how + many layers are tracked. +- The flush itself piggybacks on ``optimizer.step()`` — the one point in a step + where gradients are guaranteed present and the step is guaranteed finished. + The optimizer is resolved from the ledger lazily, on the first forward, since + a script watches its model *before* building the optimizer from + ``model.parameters()``. A custom loop with no watched optimizer can call + ``tracker.flush()`` itself. + +Collection only happens inside ``guard_training_context``, so an evaluation +pass can never contaminate a gradient or activation curve with values the +optimizer never saw — this holds even for eval loops that skip +``model.eval()`` or ``torch.no_grad()``. + +**Reading the curves** + +- ``grad_norm`` collapsing toward 0 in the *early* layers while late ones stay + healthy is a vanishing gradient: the run keeps "training" and stops learning. +- ``grad_norm`` spiking by orders of magnitude is the exploding case — pair it + with the loss curve to see which moved first. +- ``activation_std`` → 0 on a layer is that layer going constant (dead ReLUs, + saturated BatchNorm): still consuming compute, contributing nothing. +- ``weights_norm`` climbing without bound while the loss flattens is the model + growing weights instead of learning structure — time to add decay. + +See ``examples/Usecases/wl-fashion-mnist-signals`` for a complete runnable +example, including a startup legend that maps each layer id to its module. + .. _per-instance-signals: Per-sample vs per-instance watched signals diff --git a/tests/backend/test_signal_outliers.py b/tests/backend/test_signal_outliers.py new file mode 100644 index 00000000..5c6de945 --- /dev/null +++ b/tests/backend/test_signal_outliers.py @@ -0,0 +1,574 @@ +"""Tests for per-step signal outlier detection. + +Covers: +- _TrendTracker warm-up, band width, two-sided detection, top-N cap +- add_scalars attaching outliers to the averaged point (both aggregation modes) +- persistence + read-back through DuckDB (get_signal_history) +- get_step_outlier_sample_ids (backs the UI's "Highlight step samples") +- schema migration of a DB file written before the outlier columns existed +- the service layer's downsample never dropping an outlier-bearing point +""" + +import json +import os +import unittest +from unittest.mock import patch + +import duckdb + +from weightslab.backend.logger import LoggerQueue, _TrendTracker + + +def _lg() -> LoggerQueue: + """Unregistered LoggerQueue with no checkpoint manager (exp_hash = None).""" + lg = LoggerQueue(register=False) + lg.chkpt_manager = None + return lg + + +def _warm_and_spike(lg, signal="train/loss", calm_steps=40, spike_step=30, + spike=None, calm_value=0.40, batch=8): + """Log a calm curve, optionally injecting a spiking sample at one step. + + Returns the step the spike was logged at. A trailing step is logged so the + step-change flush emits the spike step's point. + """ + for step in range(calm_steps): + per_sample = {str(1000 + i): calm_value for i in range(batch)} + if spike is not None and step == spike_step: + per_sample["8123"] = spike + lg.add_scalars(signal, {}, step, per_sample, aggregate_by_step=True) + lg.add_scalars(signal, {}, calm_steps, {"1": calm_value}, aggregate_by_step=True) + return spike_step + + +def _entries(lg, signal="train/loss"): + """Flatten get_signal_history for one signal into {step: entry}.""" + history = lg.get_signal_history().get(signal, {}) + flat = {} + for steps in history.values(): + for step, entries in steps.items(): + for entry in entries: + flat[step] = entry + return flat + + +class TrendTrackerTest(unittest.TestCase): + def test_no_flagging_during_warmup(self): + """A fresh curve's steep early drop must not read as one long anomaly.""" + tracker = _TrendTracker() + tracker.min_steps = 10 + for value in [15.0, 12.0, 10.0, 8.0]: + tracker.observe(value) + self.assertIsNone(tracker.margin()) + top, total = tracker.find_outliers([("a", 100.0)]) + self.assertEqual((top, total), ([], 0)) + + def test_flags_sample_far_from_trend(self): + tracker = _TrendTracker() + tracker.min_steps = 5 + for _ in range(20): + tracker.observe(0.4) + top, total = tracker.find_outliers([("calm", 0.41), ("spike", 5.0)]) + self.assertEqual(total, 1) + self.assertEqual([item["sample_id"] for item in top], ["spike"]) + self.assertAlmostEqual(top[0]["value"], 5.0) + + def test_relative_margin_prevents_flagging_ordinary_jitter(self): + """On a flat curve the rolling std collapses; the relative floor must + still keep small noise from clearing the band.""" + tracker = _TrendTracker() + tracker.min_steps = 5 + for _ in range(50): + tracker.observe(1.0) + self.assertEqual(tracker.find_outliers([("noise", 1.05)]), ([], 0)) + # Half the EMA is the default floor, so 2x the trend does clear it. + top, total = tracker.find_outliers([("real", 2.0)]) + self.assertEqual(total, 1) + self.assertEqual(top[0]["sample_id"], "real") + + def test_detection_is_two_sided(self): + """Works for signals where 'bad' means low (e.g. accuracy), not only loss.""" + tracker = _TrendTracker() + tracker.min_steps = 5 + for _ in range(30): + tracker.observe(0.9) + top, total = tracker.find_outliers([("collapsed", 0.01)]) + self.assertEqual(total, 1) + self.assertEqual(top[0]["sample_id"], "collapsed") + + def test_top_n_cap_reports_full_total(self): + """The list is capped but the count must reflect every flagged sample, + so the UI can tell one spike from a batch-wide problem.""" + tracker = _TrendTracker() + tracker.min_steps = 5 + for _ in range(30): + tracker.observe(0.4) + with patch.dict(os.environ, {"WL_SIGNAL_OUTLIER_TOP_N": "3"}): + samples = [(f"s{i}", 10.0 + i) for i in range(9)] + top, total = tracker.find_outliers(samples) + self.assertEqual(total, 9) + self.assertEqual(len(top), 3) + # Strongest deviation first. + self.assertEqual([item["sample_id"] for item in top], ["s8", "s7", "s6"]) + + +class AddScalarsOutlierTest(unittest.TestCase): + def test_spike_step_carries_outliers_and_others_do_not(self): + lg = _lg() + step = _warm_and_spike(lg, spike=5.0) + entries = _entries(lg) + + self.assertIn("outliers", entries[step]) + self.assertEqual( + [o["sample_id"] for o in entries[step]["outliers"]], ["8123"]) + self.assertAlmostEqual(entries[step]["outliers"][0]["value"], 5.0) + self.assertEqual(entries[step]["outlier_count"], 1) + self.assertEqual(entries[step]["sample_count"], 9) + + flagged_steps = [s for s, e in entries.items() if e.get("outliers")] + self.assertEqual(flagged_steps, [step], "only the spike step should flag") + + def test_calm_run_flags_nothing(self): + lg = _lg() + _warm_and_spike(lg, spike=None) + self.assertEqual([e for e in _entries(lg).values() if e.get("outliers")], []) + + def test_batch_wide_step_reports_whole_batch(self): + """The 'high number of samples in the batch are outliers' case.""" + lg = _lg() + for step in range(40): + lg.add_scalars("train/loss", {}, step, + {str(i): 0.4 for i in range(6)}, aggregate_by_step=True) + lg.add_scalars("train/loss", {}, 40, + {"1": 9.9, "2": 9.8, "3": 9.7}, aggregate_by_step=True) + lg.add_scalars("train/loss", {}, 41, {"1": 0.4}, aggregate_by_step=True) + + entry = _entries(lg)[40] + self.assertEqual(entry["outlier_count"], 3) + self.assertEqual(entry["sample_count"], 3) + + def test_immediate_mode_also_detects(self): + """aggregate_by_step=False emits per call; outliers must still attach.""" + lg = _lg() + for step in range(40): + lg.add_scalars("m", {"m": 0.5}, step, + {str(i): 0.5 for i in range(4)}, aggregate_by_step=False) + lg.add_scalars("m", {"m": 0.5}, 40, + {"a": 0.5, "bad": 20.0}, aggregate_by_step=False) + + entry = _entries(lg, "m")[40] + self.assertEqual([o["sample_id"] for o in entry["outliers"]], ["bad"]) + + def test_disabled_by_env(self): + lg = _lg() + with patch.dict(os.environ, {"WL_SIGNAL_OUTLIER_ENABLED": "0"}): + _warm_and_spike(lg, spike=50.0) + self.assertEqual([e for e in _entries(lg).values() if e.get("outliers")], []) + + def test_signal_without_per_sample_data_is_unaffected(self): + """Signals that log only an aggregate have no ids to attribute, and must + keep working rather than erroring.""" + lg = _lg() + for step in range(20): + lg.add_scalars("lr", {"lr": 0.001}, step, {}, aggregate_by_step=False) + entries = _entries(lg, "lr") + self.assertEqual(len(entries), 20) + self.assertTrue(all(not e.get("outliers") for e in entries.values())) + + +class TrendBandTest(unittest.TestCase): + """The band the UI draws is the SAME band detection judges against. + + Drawing it is what makes an anomaly legible: unlike smoothing (which averages + a spike toward its neighbours and hides it), the band leaves values alone and + a value outside it reads as off-trend. + """ + + def test_no_band_during_warmup(self): + lg = _lg() + _warm_and_spike(lg, spike=None, calm_steps=40) + entries = _entries(lg) + self.assertNotIn("trend_value", entries[0]) + self.assertNotIn("trend_margin", entries[0]) + + def test_band_present_once_warm(self): + lg = _lg() + _warm_and_spike(lg, spike=None, calm_steps=40) + entry = _entries(lg)[20] + self.assertIn("trend_value", entry) + self.assertGreater(entry["trend_margin"], 0.0) + self.assertAlmostEqual(entry["trend_value"], 0.40, places=2) + + def test_outlier_falls_outside_its_own_band(self): + """The dot the UI draws must land outside the region it draws, or the + visual wouldn't mean anything.""" + lg = _lg() + step = _warm_and_spike(lg, spike=5.0) + entry = _entries(lg)[step] + + low = entry["trend_value"] - entry["trend_margin"] + high = entry["trend_value"] + entry["trend_margin"] + outlier_value = entry["outliers"][0]["value"] + self.assertFalse(low <= outlier_value <= high) + # The step MEAN is dragged outside too, so the curve visibly leaves the + # band — the Neptune-style cue, not just a lone marker. + self.assertFalse(low <= entry["metric_value"] <= high) + + def test_band_is_the_one_detection_used(self): + """Band and flag must be consistent: a step with a band but no outlier + should have all its samples inside that band.""" + lg = _lg() + _warm_and_spike(lg, spike=None, calm_steps=40, calm_value=0.40) + entry = _entries(lg)[25] + low = entry["trend_value"] - entry["trend_margin"] + high = entry["trend_value"] + entry["trend_margin"] + self.assertNotIn("outliers", entry) + self.assertTrue(low <= 0.40 <= high) + + def test_signals_without_per_sample_data_still_get_a_band(self): + """A band describes the CURVE, so an aggregate-only signal gets one even + though it has no samples to attribute an outlier to.""" + lg = _lg() + for step in range(40): + lg.add_scalars("lr", {"lr": 0.001}, step, {}, aggregate_by_step=False) + entry = _entries(lg, "lr")[30] + self.assertIn("trend_value", entry) + self.assertNotIn("outliers", entry) + + def test_eval_markers_do_not_carry_a_band(self): + """Eval markers are separate points under their own hash; folding them + into the training curve's trend would corrupt it.""" + lg = _lg() + _warm_and_spike(lg, spike=None, calm_steps=40) + lg.start_evaluation_mode("evalhash_1", "test_loader", []) + lg.add_scalars("train/loss", {}, 41, {"a": 0.4, "b": 0.5}, aggregate_by_step=True) + results = lg.stop_evaluation_mode(41) + self.assertTrue(results) + + marker = None + for steps in lg.get_signal_history()["train/loss"].values(): + for entries in steps.values(): + for entry in entries: + if entry.get("is_evaluation_marker"): + marker = entry + self.assertIsNotNone(marker) + self.assertNotIn("trend_value", marker) + + def test_band_disabled_with_outliers(self): + lg = _lg() + with patch.dict(os.environ, {"WL_SIGNAL_OUTLIER_ENABLED": "0"}): + _warm_and_spike(lg, spike=None, calm_steps=40) + self.assertNotIn("trend_value", _entries(lg)[20]) + + def test_band_survives_the_db_round_trip(self): + import tempfile + db_path = os.path.join(tempfile.mkdtemp(), "band.duckdb") + lg = _lg() + lg.set_db_path(db_path) + _warm_and_spike(lg, spike=None, calm_steps=40) + lg._flush_stage() + entry = _entries(lg)[20] + self.assertIn("trend_value", entry) + self.assertIn("trend_margin", entry) + + def test_load_signal_history_round_trips_the_band(self): + lg = _lg() + lg.load_signal_history({ + "m": {"h": {3: [{ + "metric_value": 1.0, "timestamp": 0, + "trend_value": 0.9, "trend_margin": 0.25, + }]}} + }) + entry = _entries(lg, "m")[3] + self.assertAlmostEqual(entry["trend_value"], 0.9) + self.assertAlmostEqual(entry["trend_margin"], 0.25) + + def test_service_forwards_the_band_with_a_presence_flag(self): + from weightslab.trainer.services.experiment_service import _logger_point_pb + + with_band = _logger_point_pb("m", { + "model_age": 1, "metric_value": 1.0, + "trend_value": 0.0, "trend_margin": 0.0, + }) + # A genuine zero-centred, zero-width band must not read as absent — + # proto3 scalars have no presence, hence the explicit flag. + self.assertTrue(with_band.has_trend_band) + + without = _logger_point_pb("m", {"model_age": 1, "metric_value": 1.0}) + self.assertFalse(without.has_trend_band) + + real = _logger_point_pb("m", { + "model_age": 1, "trend_value": 0.41, "trend_margin": 0.05, + }) + self.assertTrue(real.has_trend_band) + self.assertAlmostEqual(real.trend_value, 0.41, places=5) + self.assertAlmostEqual(real.trend_margin, 0.05, places=5) + + +class StepOutlierIdsTest(unittest.TestCase): + def test_returns_ids_for_the_step(self): + lg = _lg() + step = _warm_and_spike(lg, spike=5.0) + self.assertEqual(lg.get_step_outlier_sample_ids("train/loss", None, step), ["8123"]) + + def test_empty_for_clean_step(self): + lg = _lg() + _warm_and_spike(lg, spike=5.0) + self.assertEqual(lg.get_step_outlier_sample_ids("train/loss", None, 5), []) + + def test_hash_filter_isolates_runs(self): + lg = _lg() + _warm_and_spike(lg, spike=5.0) + # Points were written with exp_hash None; a different hash must not match. + self.assertEqual( + lg.get_step_outlier_sample_ids("train/loss", "other-hash", 30), []) + + def test_decode_tolerates_corrupt_payload(self): + self.assertEqual(LoggerQueue._decode_outliers("not json"), []) + self.assertEqual(LoggerQueue._decode_outliers(""), []) + self.assertEqual(LoggerQueue._decode_outliers(None), []) + self.assertEqual(LoggerQueue._decode_outliers('{"a":1}'), []) + self.assertEqual( + LoggerQueue._decode_outliers('[{"sample_id": 7, "value": "2.5"}]'), + [{"sample_id": "7", "value": 2.5}]) + + +class SchemaMigrationTest(unittest.TestCase): + def test_adopts_db_written_before_outlier_columns(self): + """A DB file from an older weightslab lacks the outlier columns. + CREATE TABLE IF NOT EXISTS won't add them, so opening it must ALTER them + in and keep the existing rows.""" + import tempfile + db_path = os.path.join(tempfile.mkdtemp(), "legacy.duckdb") + conn = duckdb.connect(db_path) + conn.execute( + """ + CREATE TABLE signals ( + metric_name VARCHAR, experiment_hash VARCHAR, step INTEGER, + metric_value DOUBLE, timestamp BIGINT, audit_mode BOOLEAN, + is_evaluation_marker BOOLEAN, split_name VARCHAR, + evaluation_tags VARCHAR, point_note VARCHAR, seq BIGINT + ) + """ + ) + conn.execute( + "INSERT INTO signals VALUES " + "('train/loss','abc',1,0.5,0,false,false,'','[]','',0)") + conn.close() + + lg = _lg() + lg.set_db_path(db_path) + + columns = { + row[0] for row in lg._conn.execute( + "SELECT column_name FROM information_schema.columns " + "WHERE table_name = 'signals'").fetchall() + } + self.assertIn("outliers", columns) + self.assertIn("outlier_count", columns) + self.assertIn("sample_count", columns) + + # Legacy row survived, and new writes still land. + entries = _entries(lg) + self.assertIn(1, entries) + step = _warm_and_spike(lg, spike=6.0) + self.assertTrue(_entries(lg)[step].get("outliers")) + + def test_load_signal_history_round_trips_outliers(self): + """Checkpoint restore must not silently drop the outlier payload.""" + lg = _lg() + lg.load_signal_history({ + "train/loss": { + "abc": { + 7: [{ + "metric_value": 0.4, + "timestamp": 0, + "outliers": [{"sample_id": "42", "value": 9.0}], + "outlier_count": 3, + "sample_count": 16, + }] + } + } + }) + entry = _entries(lg)[7] + self.assertEqual(entry["outliers"], [{"sample_id": "42", "value": 9.0}]) + self.assertEqual(entry["outlier_count"], 3) + self.assertEqual(entry["sample_count"], 16) + + +class ServiceDownsampleTest(unittest.TestCase): + def test_downsample_keeps_every_outlier_point(self): + """Striding a long curve must not discard the anomalies the feature + exists to surface — an outlier is a single step, so a stride of N would + drop most of them.""" + from weightslab.trainer.services.experiment_service import ( + _downsample_preserving_outliers, + ) + + history = [{"model_age": i} for i in range(1000)] + for spike in (3, 17, 998): + history[spike]["outliers"] = [{"sample_id": "x", "value": 9.0}] + + kept = _downsample_preserving_outliers(history, 100) + kept_steps = {entry["model_age"] for entry in kept} + for spike in (3, 17, 998): + self.assertIn(spike, kept_steps, f"outlier at step {spike} was dropped") + self.assertLess(len(kept), 200, "should still be a downsample") + + def test_downsample_noop_below_cap(self): + from weightslab.trainer.services.experiment_service import ( + _downsample_preserving_outliers, + ) + history = [{"model_age": i} for i in range(10)] + self.assertIs(_downsample_preserving_outliers(history, 100), history) + + def test_logger_point_pb_carries_outliers(self): + from weightslab.trainer.services.experiment_service import _logger_point_pb + + point = _logger_point_pb("train/loss", { + "model_age": 5, + "metric_value": 0.4, + "experiment_hash": "abc", + "timestamp": 0, + "outliers": [{"sample_id": "9", "value": 3.5}], + "outlier_count": 4, + "sample_count": 32, + }) + self.assertEqual(point.outlier_count, 4) + self.assertEqual(point.sample_count, 32) + self.assertEqual(point.outliers[0].sample_id, "9") + self.assertAlmostEqual(point.outliers[0].value, 3.5, places=5) + + def test_logger_point_pb_skips_malformed_outliers(self): + from weightslab.trainer.services.experiment_service import _logger_point_pb + + point = _logger_point_pb("m", { + "model_age": 1, + "outliers": ["nonsense", {"value": 1.0}, {"sample_id": "ok", "value": 2.0}], + }) + self.assertEqual([o.sample_id for o in point.outliers], ["ok"]) + + +if __name__ == "__main__": + unittest.main() + + +class ValueRangeTest(unittest.TestCase): + """The error band is drawn from the batch's ABSOLUTE extremes. + + That choice is what makes an outlier visible: min/max reach the outlier's own + value, so the band spikes out to it, whereas a standard deviation would + average it down toward the batch mean. + """ + + def test_range_recorded_per_step(self): + lg = _lg() + for step in range(5): + lg.add_scalars("m", {}, step, {"a": 0.2, "b": 0.8}, aggregate_by_step=True) + lg.add_scalars("m", {}, 5, {"a": 0.5}, aggregate_by_step=True) + + entry = _entries(lg, "m")[2] + self.assertAlmostEqual(entry["value_min"], 0.2) + self.assertAlmostEqual(entry["value_max"], 0.8) + # The curve itself stays the mean. + self.assertAlmostEqual(entry["metric_value"], 0.5) + + def test_range_spikes_to_the_outlier(self): + lg = _lg() + step = _warm_and_spike(lg, spike=5.0) + entry = _entries(lg)[step] + self.assertAlmostEqual(entry["value_max"], 5.0) + # A std-based band would have stayed near the mean; min/max reaches it. + self.assertGreater(entry["value_max"], entry["metric_value"]) + + def test_no_range_without_per_sample_data(self): + lg = _lg() + for step in range(5): + lg.add_scalars("lr", {"lr": 0.001}, step, {}, aggregate_by_step=False) + self.assertNotIn("value_min", _entries(lg, "lr")[2]) + + def test_range_survives_db_round_trip_and_restore(self): + import tempfile + db_path = os.path.join(tempfile.mkdtemp(), "range.duckdb") + lg = _lg() + lg.set_db_path(db_path) + lg.add_scalars("m", {}, 0, {"a": 1.0, "b": 3.0}, aggregate_by_step=True) + lg.add_scalars("m", {}, 1, {"a": 2.0}, aggregate_by_step=True) + lg._flush_stage() + entry = _entries(lg, "m")[0] + self.assertAlmostEqual(entry["value_min"], 1.0) + self.assertAlmostEqual(entry["value_max"], 3.0) + + restored = _lg() + restored.load_signal_history({ + "m": {"h": {9: [{"metric_value": 2.0, "timestamp": 0, + "value_min": 1.0, "value_max": 3.0}]}} + }) + self.assertAlmostEqual(_entries(restored, "m")[9]["value_max"], 3.0) + + def test_service_forwards_the_range_with_a_presence_flag(self): + from weightslab.trainer.services.experiment_service import _logger_point_pb + + point = _logger_point_pb("m", { + "model_age": 1, "metric_value": 0.9, "value_min": 0.39, "value_max": 5.0, + }) + self.assertTrue(point.has_value_range) + self.assertAlmostEqual(point.value_min, 0.39, places=5) + self.assertAlmostEqual(point.value_max, 5.0, places=5) + + # A genuine all-zero range must not read as absent. + zeroed = _logger_point_pb("m", {"model_age": 1, "value_min": 0.0, "value_max": 0.0}) + self.assertTrue(zeroed.has_value_range) + + absent = _logger_point_pb("m", {"model_age": 1}) + self.assertFalse(absent.has_value_range) + + +class StepSampleIdsTest(unittest.TestCase): + """Backs "Highlight step samples", which shows the WHOLE batch behind a point.""" + + def _logged(self): + lg = _lg() + for step in range(3): + lg.add_scalars("m", {}, step, + {str(i): 0.5 for i in range(4)}, aggregate_by_step=True) + lg.add_scalars("m", {}, 3, {"0": 0.5}, aggregate_by_step=True) + return lg + + def test_returns_every_sample_not_just_outliers(self): + lg = self._logged() + ids, total = lg.get_step_sample_ids("m", None, 1) + self.assertEqual(ids, ["0", "1", "2", "3"]) + self.assertEqual(total, 4) + + def test_empty_for_a_step_with_no_per_sample_rows(self): + lg = self._logged() + ids, total = lg.get_step_sample_ids("m", None, 99) + self.assertEqual((ids, total), ([], 0)) + + def test_hash_filter(self): + lg = self._logged() + self.assertEqual(lg.get_step_sample_ids("m", "nope", 1), ([], 0)) + + def test_cap_reports_the_true_total(self): + lg = self._logged() + ids, total = lg.get_step_sample_ids("m", None, 1, max_samples=2) + self.assertEqual(len(ids), 2) + self.assertEqual(total, 4, "total must be the pre-cap count") + + def test_numeric_aware_ordering(self): + """'9' must precede '10' — string ordering would invert them.""" + lg = _lg() + lg.add_scalars("m", {}, 0, {"9": 0.1, "10": 0.2, "2": 0.3}, + aggregate_by_step=True) + lg.add_scalars("m", {}, 1, {"9": 0.1}, aggregate_by_step=True) + ids, _ = lg.get_step_sample_ids("m", None, 0) + self.assertEqual(ids, ["2", "9", "10"]) + + def test_non_numeric_ids_fall_back_to_text_order(self): + lg = _lg() + lg.add_scalars("m", {}, 0, {"img_b": 0.1, "img_a": 0.2}, aggregate_by_step=True) + lg.add_scalars("m", {}, 1, {"img_a": 0.1}, aggregate_by_step=True) + ids, _ = lg.get_step_sample_ids("m", None, 0) + self.assertEqual(ids, ["img_a", "img_b"]) diff --git a/tests/general/test_model_signals.py b/tests/general/test_model_signals.py new file mode 100644 index 00000000..47fb7d68 --- /dev/null +++ b/tests/general/test_model_signals.py @@ -0,0 +1,230 @@ +"""Per-step model signals: `wl.save_model_signals` + `wl.track_model_signals`. + +Split into two halves that need very different setups: + + TestSaveModelSignals the write path alone, against a mock logger. Pins the + contract that makes these curves work at all -- + `aggregate_by_step=False` with no per-sample map, which + is what appends one point per step instead of bucketing. + TestTrackModelSignals the collector, against a real 3-layer model and a real + training step. Asserts on values, not just on calls: + the global norm has an arithmetic answer, and getting + it right is the whole point of combining layers in + quadrature rather than summing their norms. +""" + +import unittest +from unittest.mock import MagicMock, patch + +import torch +import torch.nn as nn + +import weightslab as wl +from weightslab.components.model_signals import ( + METRICS, + ModelSignalTracker, + _iter_layers, +) +from weightslab.components.tracking import TrackingMode + + +class TestSaveModelSignals(unittest.TestCase): + def _capture(self, signals, step=7): + """Run save_model_signals against a mock logger; return its calls.""" + mock_logger = MagicMock() + with patch("weightslab.src.get_logger", return_value=mock_logger), \ + patch("weightslab.src._get_step", side_effect=lambda step=None: step): + wl.save_model_signals(signals, step=step) + return mock_logger.add_scalars.call_args_list + + def test_emits_one_point_per_signal_in_immediate_mode(self): + calls = self._capture({ + "metrics/global/grad_norm": 1.5, + "metrics/layer/3/weights_norm": torch.tensor(2.5), + }) + self.assertEqual(len(calls), 2) + for call in calls: + name = call.args[0] + self.assertEqual(call.args[1], {name: unittest.mock.ANY}) + self.assertEqual(call.kwargs["global_step"], 7) + # The contract that makes a step-keyed curve behave: no per-sample + # map, and no per-step aggregation bucket to be averaged into. + self.assertIsNone(call.kwargs["signal_per_sample"]) + self.assertFalse(call.kwargs["aggregate_by_step"]) + + by_name = {c.args[0]: c.args[1][c.args[0]] for c in calls} + self.assertAlmostEqual(by_name["metrics/global/grad_norm"], 1.5, places=6) + self.assertAlmostEqual(by_name["metrics/layer/3/weights_norm"], 2.5, places=6) + + def test_reduces_a_tensor_to_one_scalar(self): + calls = self._capture({"metrics/global/x": torch.tensor([1.0, 2.0, 3.0])}) + self.assertEqual(len(calls), 1) + self.assertAlmostEqual(calls[0].args[1]["metrics/global/x"], 2.0, places=6) + + def test_drops_non_finite_points_without_raising(self): + """A diverged run should break the curve, not rescale the whole axis.""" + calls = self._capture({ + "metrics/global/exploded": float("inf"), + "metrics/global/nan": float("nan"), + "metrics/global/fine": 0.25, + }) + self.assertEqual([c.args[0] for c in calls], ["metrics/global/fine"]) + + def test_empty_and_non_numeric_are_no_ops(self): + self.assertEqual(self._capture({}), []) + self.assertEqual(self._capture({"metrics/global/text": "not a number"}), []) + + +class _ThreeLayer(nn.Module): + """Two parameterized layers with a ReLU between them. + + Small enough that the expected global gradient norm can be written out by + hand, which is what test_global_norm_combines_layers_in_quadrature needs. + """ + + def __init__(self): + super().__init__() + self.input_shape = (1, 4) + self.fc1 = nn.Linear(4, 3) + self.act = nn.ReLU() + self.fc2 = nn.Linear(3, 2) + + def forward(self, x): + return self.fc2(self.act(self.fc1(x))) + + +class TestTrackModelSignals(unittest.TestCase): + def setUp(self): + self.model = _ThreeLayer() + # The tracker reads tracking_mode to tell a training step from an eval + # pass; a bare nn.Module has no such attribute, so stand in for what + # guard_training_context would set. + self.model.tracking_mode = TrackingMode.TRAIN + self.model.get_age = lambda: 0 + self.trackers = [] + + def tearDown(self): + for tracker in self.trackers: + tracker.remove() + + def _tracker(self, **kwargs): + # `_ensure_flush_hooked` looks up a watched optimizer to wrap; there is + # no ledger in this test, so short it out and drive flush() by hand. + tracker = ModelSignalTracker(self.model, **kwargs) + tracker._flush_hooked = True + self.trackers.append(tracker) + return tracker + + def _train_step(self, tracker, flush=True): + """One real forward/backward, then flush. Returns the emitted map.""" + emitted = {} + with patch("weightslab.src.save_model_signals", + side_effect=lambda signals, step=None: emitted.update(signals)): + self.model.zero_grad() + out = self.model(torch.randn(5, 4)) + out.pow(2).mean().backward() + if flush: + tracker.flush() + return emitted + + def test_layer_ids_come_from_the_module_id_the_rest_of_wl_uses(self): + rows = _iter_layers(self.model) + self.assertEqual([type(m).__name__ for _, m in rows], + ["Linear", "ReLU", "Linear"]) + + def test_emits_every_metric_for_every_eligible_layer(self): + emitted = self._train_step(self._tracker()) + + # Parameterized layers get norms; the ReLU gets activations only. + for metric in ("grad_norm", "weights_norm"): + names = [n for n in emitted if n.endswith("/" + metric) + and n.startswith("metrics/layer/")] + self.assertEqual(len(names), 2, f"{metric}: {names}") + for metric in ("activation_mean", "activation_std", + "activation_max", "activation_min"): + names = [n for n in emitted if n.endswith("/" + metric)] + self.assertEqual(len(names), 3, f"{metric}: {names}") + + self.assertIn("metrics/global/grad_norm", emitted) + self.assertIn("metrics/global/weights_norm", emitted) + + def test_global_norm_combines_layers_in_quadrature(self): + """sqrt(sum of squares), not a sum of per-layer norms. + + The wrong version (summing norms) is an L1 over L2s and always reads + high, so it silently misreports every run rather than failing loudly. + """ + emitted = self._train_step(self._tracker(metrics=["grad_norm"])) + + per_layer = [v for k, v in emitted.items() + if k.startswith("metrics/layer/") and k.endswith("/grad_norm")] + expected = sum(v ** 2 for v in per_layer) ** 0.5 + self.assertAlmostEqual(emitted["metrics/global/grad_norm"], expected, places=5) + # And it is genuinely below the naive sum, so this asserts something. + self.assertLess(emitted["metrics/global/grad_norm"], sum(per_layer)) + + def test_relu_activation_min_is_exactly_zero(self): + """A cheap end-to-end sanity check on the activation values themselves.""" + emitted = self._train_step(self._tracker(metrics=["activation_min"])) + relu_id = next(lid for lid, m in _iter_layers(self.model) + if isinstance(m, nn.ReLU)) + self.assertEqual(emitted[f"metrics/layer/{relu_id}/activation_min"], 0.0) + + def test_collects_nothing_outside_a_training_context(self): + tracker = self._tracker() + self.model.tracking_mode = TrackingMode.EVAL + self.assertEqual(self._train_step(tracker), {}) + + def test_every_n_steps_skips_unsampled_steps(self): + tracker = self._tracker(every_n_steps=10) + + self.model.get_age = lambda: 3 + self.assertEqual(self._train_step(tracker), {}) + + self.model.get_age = lambda: 20 + self.assertTrue(self._train_step(tracker)) + + def test_layer_ids_filter_restricts_what_is_tracked(self): + first_id = _iter_layers(self.model)[0][0] + emitted = self._train_step(self._tracker(layer_ids=[first_id])) + layer_names = {n for n in emitted if n.startswith("metrics/layer/")} + self.assertTrue(layer_names) + self.assertEqual({n.split("/")[2] for n in layer_names}, {first_id}) + + def test_include_global_false_drops_only_the_global_curves(self): + emitted = self._train_step(self._tracker(include_global=False)) + self.assertFalse([n for n in emitted if n.startswith("metrics/global/")]) + self.assertTrue([n for n in emitted if n.startswith("metrics/layer/")]) + + def test_unknown_metric_is_rejected_at_construction(self): + with self.assertRaises(ValueError) as ctx: + ModelSignalTracker(self.model, metrics=["grad_norm", "nope"]) + self.assertIn("nope", str(ctx.exception)) + + def test_flush_resets_so_a_skipped_step_emits_nothing(self): + tracker = self._tracker() + self.assertTrue(self._train_step(tracker)) + # No new forward/backward: everything was consumed by the first flush, + # so only the always-available weight norms remain. + emitted = {} + with patch("weightslab.src.save_model_signals", + side_effect=lambda signals, step=None: emitted.update(signals)): + tracker.flush() + self.assertFalse([n for n in emitted if "grad_norm" in n]) + self.assertFalse([n for n in emitted if "activation" in n]) + + def test_remove_is_idempotent_and_stops_collection(self): + tracker = self._tracker() + tracker.remove() + tracker.remove() + self.assertEqual(self._train_step(tracker), {}) + + def test_metrics_constant_matches_what_the_tracker_accepts(self): + """Guards the docs: METRICS is the published list of signal names.""" + emitted = self._train_step(self._tracker(metrics=METRICS)) + suffixes = {n.rsplit("/", 1)[1] for n in emitted} + self.assertEqual(suffixes, set(METRICS)) + + +if __name__ == "__main__": + unittest.main() diff --git a/weightslab/__init__.py b/weightslab/__init__.py index 48d8f3e8..e0aea1c0 100644 --- a/weightslab/__init__.py +++ b/weightslab/__init__.py @@ -36,7 +36,8 @@ # Everything re-exported straight from .src (attribute name == export name). for _name in ( "watch_or_edit", "start_training", "serve", "keep_serving", "save_signals", - "save_instance_signals", "save_group_signals", "tag_samples", + "save_instance_signals", "save_group_signals", "save_model_signals", + "track_model_signals", "tag_samples", "register_categorical_tag", "set_categorical_tag", "discard_samples", "get_samples_by_tag", "get_discarded_samples", "signal", "eval_fn", "compute_signals", "SignalContext", "BatchSignalContext", "StaleSignalError", @@ -191,6 +192,8 @@ def _clean(v: str) -> str: "save_signals", "save_instance_signals", "save_group_signals", + "save_model_signals", + "track_model_signals", "signal", "compute_signals", "set_log_directory", diff --git a/weightslab/backend/logger.py b/weightslab/backend/logger.py index d7206930..c86a21e9 100644 --- a/weightslab/backend/logger.py +++ b/weightslab/backend/logger.py @@ -50,7 +50,8 @@ _SIGNAL_COLS = [ "metric_name", "experiment_hash", "step", "metric_value", "timestamp", "audit_mode", "is_evaluation_marker", "split_name", "evaluation_tags", - "point_note", "seq", + "point_note", "outliers", "outlier_count", "sample_count", + "trend_value", "trend_margin", "value_min", "value_max", "seq", ] _SAMPLE_COLS = ["metric_name", "experiment_hash", "sample_id", "step", "value", "seq"] _INSTANCE_COLS = [ @@ -69,12 +70,126 @@ def _default_flush_interval_seconds() -> float: return 2.0 +def _env_float(name: str, default: float) -> float: + try: + return float(os.environ.get(name, default)) + except (TypeError, ValueError): + return default + + +def _env_int(name: str, default: int) -> int: + try: + return int(os.environ.get(name, default)) + except (TypeError, ValueError): + return default + + +def _outliers_enabled() -> bool: + return os.environ.get("WL_SIGNAL_OUTLIER_ENABLED", "1").strip().lower() not in ( + "0", "false", "no", "off", + ) + + +# Hard cap on how many (sample_id, value) pairs one buffered step retains for +# outlier detection. Batches are small, but add_scalars can be called many times +# per step; this bounds memory on pathological loggers. +_MAX_BUFFERED_SAMPLES_PER_STEP = 4096 + + +class _TrendTracker: + """Rolling trend of one signal's averaged curve, for outlier detection. + + Keeps an EMA of the per-step average plus an EMA of squared deviation (a + rolling variance). A sample is "off-trend" when it sits further from the EMA + than ``k`` rolling standard deviations. + + Two guards keep this from firing constantly: + + * ``min_steps`` — no flagging until the curve has enough history, so the + steep warm-up of a fresh loss curve isn't one long outlier run. + * ``rel_margin`` — the band never narrows below a fraction of |EMA|. On an + almost-flat curve the rolling std collapses toward zero, and without this + floor ordinary jitter would clear a 3-sigma test. + + Deviation is measured two-sided (by magnitude) so this works for signals + where "bad" means low, e.g. accuracy, as well as loss-shaped ones. + """ + + __slots__ = ("ema", "ema_var", "steps", "alpha", "k", "min_steps", "rel_margin") + + def __init__(self) -> None: + self.ema = None + self.ema_var = 0.0 + self.steps = 0 + self.alpha = _env_float("WL_SIGNAL_OUTLIER_EMA_ALPHA", 0.05) + self.k = _env_float("WL_SIGNAL_OUTLIER_K", 3.0) + self.min_steps = _env_int("WL_SIGNAL_OUTLIER_MIN_STEPS", 10) + self.rel_margin = _env_float("WL_SIGNAL_OUTLIER_REL_MARGIN", 0.5) + + def margin(self): + """Half-width of the on-trend band, or ``None`` while still warming up.""" + if self.ema is None or self.steps < self.min_steps: + return None + std = self.ema_var ** 0.5 + return max(self.k * std, self.rel_margin * abs(self.ema)) + + def observe(self, average: float) -> None: + """Fold this step's average into the trend. Call once per emitted point.""" + if self.ema is None: + self.ema = average + self.steps = 1 + return + deviation = average - self.ema + self.ema += self.alpha * deviation + self.ema_var = (1.0 - self.alpha) * self.ema_var + self.alpha * (deviation ** 2) + self.steps += 1 + + def find_outliers(self, samples): + """Flag the off-trend members of one step's batch. + + Args: + samples: Iterable of ``(sample_id, value)`` for this step. + + Returns: + ``(top, total)`` where *top* is a list of ``{"sample_id", "value"}`` + dicts sorted by deviation (strongest first) and truncated to + ``WL_SIGNAL_OUTLIER_TOP_N``, and *total* is how many samples were + flagged before truncation. ``([], 0)`` while warming up. + """ + margin = self.margin() + if margin is None or not samples: + return [], 0 + + flagged = [] + for sample_id, value in samples: + deviation = abs(value - self.ema) + if deviation > margin: + flagged.append((deviation, sample_id, value)) + + if not flagged: + return [], 0 + + flagged.sort(key=lambda row: row[0], reverse=True) + top_n = max(1, _env_int("WL_SIGNAL_OUTLIER_TOP_N", 5)) + top = [ + {"sample_id": str(sample_id), "value": float(value)} + for _, sample_id, value in flagged[:top_n] + ] + return top, len(flagged) + + class LoggerQueue: def __init__(self, register: bool = True, db_path: str = ":memory:") -> None: self.graph_names = set() self._current_step_buffer = {} self._last_step = None + # Rolling trend per (graph_name, exp_hash), used to flag the samples in a + # step's batch that sit off the curve (see _TrendTracker). In-memory only: + # a resumed run re-warms from its first min_steps points rather than + # inheriting a stale band. + self._trend_trackers: dict = defaultdict(_TrendTracker) + # Live-streaming queue of new points waiting to be sent to WeightsStudio. self._pending_queue = [] self._buffered_step = None @@ -293,6 +408,13 @@ def _schema_ddl(prefix: str = "") -> str: split_name VARCHAR, evaluation_tags VARCHAR, point_note VARCHAR, + outliers VARCHAR, + outlier_count INTEGER, + sample_count INTEGER, + trend_value DOUBLE, + trend_margin DOUBLE, + value_min DOUBLE, + value_max DOUBLE, seq BIGINT ); CREATE TABLE IF NOT EXISTS {prefix}per_sample ( @@ -314,9 +436,52 @@ def _schema_ddl(prefix: str = "") -> str: ); """ + # Columns added to `signals` after the table's first release, as + # (name, DDL type, default). A DB file written by an older weightslab + # predates them, and CREATE TABLE IF NOT EXISTS won't retrofit them, so + # _ensure_tables ALTERs them in on open. Appended columns land at the end of + # the table, which is why every INSERT names its columns explicitly instead + # of relying on staging-buffer order (see _flush_stage). + _SIGNAL_MIGRATIONS = ( + ("outliers", "VARCHAR", "''"), + ("outlier_count", "INTEGER", "0"), + ("sample_count", "INTEGER", "0"), + # NULL (not 0) so "no band recorded" stays distinguishable from a real + # band centred on zero. + ("trend_value", "DOUBLE", "NULL"), + ("trend_margin", "DOUBLE", "NULL"), + ("value_min", "DOUBLE", "NULL"), + ("value_max", "DOUBLE", "NULL"), + ) + def _ensure_tables(self) -> None: with self._lock: self._conn.execute(self._schema_ddl()) + self._migrate_signal_columns() + + def _migrate_signal_columns(self) -> None: + """Add any post-release `signals` columns missing from an older DB file.""" + try: + existing = { + row[0] for row in self._conn.execute( + "SELECT column_name FROM information_schema.columns " + "WHERE table_name = 'signals'" + ).fetchall() + } + except Exception as exc: # pragma: no cover - defensive + logger.debug("Could not introspect signals columns: %s", exc) + return + + for name, ddl_type, default in self._SIGNAL_MIGRATIONS: + if name in existing: + continue + try: + self._conn.execute( + f"ALTER TABLE signals ADD COLUMN {name} {ddl_type} DEFAULT {default}" + ) + logger.info("Migrated signal history: added signals.%s", name) + except Exception as exc: + logger.warning("Failed to add signals.%s: %s", name, exc) _HISTORY_TABLES = ("signals", "per_sample", "per_instance") @@ -492,7 +657,12 @@ def _flush_stage(self) -> None: if self._stage_signals: df = pd.DataFrame(self._stage_signals, columns=_SIGNAL_COLS) self._conn.register("_stg_sig", df) - self._conn.execute("INSERT INTO signals SELECT * FROM _stg_sig") + # Column-explicit: migrated columns sit at the end of an older + # table, so positional INSERT ... SELECT * would misalign. + cols = ", ".join(_SIGNAL_COLS) + self._conn.execute( + f"INSERT INTO signals ({cols}) SELECT {cols} FROM _stg_sig" + ) self._conn.unregister("_stg_sig") self._stage_signals = [] if self._stage_sample: @@ -509,11 +679,21 @@ def _flush_stage(self) -> None: self._stage_instance = [] def _stage_signal_row(self, graph_name, exp_hash, step, metric_value, timestamp, - audit_mode, is_marker, split_name, eval_tags, point_note): + audit_mode, is_marker, split_name, eval_tags, point_note, + outliers=None, outlier_count=0, sample_count=0, + trend_value=None, trend_margin=None, + value_min=None, value_max=None): self._stage_signals.append(( graph_name, exp_hash, int(step), float(metric_value), int(timestamp), bool(audit_mode), bool(is_marker), split_name or "", - json.dumps(list(eval_tags or [])), point_note or "", self._next_seq(), + json.dumps(list(eval_tags or [])), point_note or "", + json.dumps(list(outliers)) if outliers else "", + int(outlier_count), int(sample_count), + None if trend_value is None else float(trend_value), + None if trend_margin is None else float(trend_margin), + None if value_min is None else float(value_min), + None if value_max is None else float(value_max), + self._next_seq(), )) self._maybe_autoflush() @@ -611,8 +791,14 @@ def _get_audit_mode(self): def _append_history_entry(self, graph_name, exp_hash, global_step, metric_value, audit_mode=None, is_marker=False, split_name="", - evaluation_tags=None): - """Stage a signals row and return the live-queue entry dict.""" + evaluation_tags=None, batch_samples=None): + """Stage a signals row and return the live-queue entry dict. + + *batch_samples*, when given, is the ``(sample_id, value)`` batch this + point's average came from. It is compared against the signal's rolling + trend to flag off-trend samples, which ride along on the point so the UI + can mark the spike and jump to the samples behind it. + """ if audit_mode is None: audit_mode = self._get_audit_mode() @@ -630,11 +816,51 @@ def _append_history_entry(self, graph_name, exp_hash, global_step, metric_value, signal_entry["split_name"] = split_name signal_entry["evaluation_tags"] = list(evaluation_tags or []) + outliers, outlier_count = [], 0 + trend_value, trend_margin = None, None + value_min, value_max = None, None + sample_count = len(batch_samples) if batch_samples else 0 + + # Absolute extremes of the batch. These are the real lowest/highest + # sample values, so the band the UI draws from them spikes out to an + # outlier rather than averaging it down the way a std band would. + if batch_samples: + values = [value for _, value in batch_samples] + value_min, value_max = min(values), max(values) + + if _outliers_enabled() and not is_marker: + tracker = self._trend_trackers[(graph_name, exp_hash)] + # Snapshot the band BEFORE folding this point in, so a spike is + # measured against clean history instead of partly against itself. + # This is the same band find_outliers uses, which is what lets the UI + # draw the region a flagged sample fell outside of. + margin = tracker.margin() + if margin is not None: + trend_value, trend_margin = tracker.ema, margin + if batch_samples: + outliers, outlier_count = tracker.find_outliers(batch_samples) + tracker.observe(metric_value) + if outliers: + signal_entry["outliers"] = outliers + signal_entry["outlier_count"] = outlier_count + if sample_count: + signal_entry["sample_count"] = sample_count + if trend_value is not None: + signal_entry["trend_value"] = trend_value + signal_entry["trend_margin"] = trend_margin + if value_min is not None: + signal_entry["value_min"] = value_min + signal_entry["value_max"] = value_max + with self._lock: self._stage_signal_row( graph_name, exp_hash, global_step, metric_value, timestamp, bool(audit_mode), bool(is_marker), split_name, list(evaluation_tags or []), "", + outliers=outliers, outlier_count=outlier_count, + sample_count=sample_count, + trend_value=trend_value, trend_margin=trend_margin, + value_min=value_min, value_max=value_max, ) return signal_entry @@ -651,6 +877,7 @@ def _flush_current_step_buffer(self, add_to_queue: bool): exp_hash=exp_hash, global_step=self._buffered_step, metric_value=metric_value, + batch_samples=payload.get("samples"), ) if add_to_queue: self._pending_queue.append(signal_entry) @@ -841,10 +1068,20 @@ def add_scalars(self, graph_name, signal, global_step, signal_per_sample, aggreg for sid, value in signal_per_sample.items(): self._stage_sample_row(graph_name, exp_hash, sid, step_i, self._to_float(value)) + # (sample_id, value) for this call's batch, so _append_history_entry + # can attribute an off-trend point to the samples responsible. + # Derived from signal_per_sample independently of which branch below + # supplies metric_values: in immediate mode the emitted value comes + # from `signal`, but the batch behind it is still signal_per_sample, + # and outliers are judged against the curve's trend either way. + batch_samples = [ + (str(sid), self._to_float(value)) + for sid, value in signal_per_sample.items() + ] if isinstance(signal_per_sample, dict) and len(signal_per_sample) else [] + metric_values = [] if isinstance(signal_per_sample, dict) and aggregate_by_step and len(signal_per_sample): - for value in signal_per_sample.values(): - metric_values.append(self._to_float(value)) + metric_values = [value for _, value in batch_samples] else: for _, line_value in signal.items(): metric_values.append(self._to_float(line_value)) @@ -854,9 +1091,16 @@ def add_scalars(self, graph_name, signal, global_step, signal_per_sample, aggreg self._buffered_step = global_step buffer_key = (global_step, graph_name, exp_hash) if buffer_key not in self._current_step_buffer: - self._current_step_buffer[buffer_key] = {"sum": 0.0, "count": 0} - self._current_step_buffer[buffer_key]["sum"] += sum(metric_values) - self._current_step_buffer[buffer_key]["count"] += len(metric_values) + self._current_step_buffer[buffer_key] = { + "sum": 0.0, "count": 0, "samples": [], + } + payload = self._current_step_buffer[buffer_key] + payload["sum"] += sum(metric_values) + payload["count"] += len(metric_values) + if batch_samples: + headroom = _MAX_BUFFERED_SAMPLES_PER_STEP - len(payload["samples"]) + if headroom > 0: + payload["samples"].extend(batch_samples[:headroom]) return # Update averaged signal history immediately. Only emit when we have at @@ -869,6 +1113,7 @@ def add_scalars(self, graph_name, signal, global_step, signal_per_sample, aggreg exp_hash=exp_hash, global_step=global_step, metric_value=sum(metric_values) / len(metric_values) if len(metric_values) > 1 else metric_values[0], + batch_samples=batch_samples or None, ) if signal_entry is not None: @@ -960,6 +1205,103 @@ def list_instance_signal_names(self) -> list: rows = self._conn.execute("SELECT DISTINCT metric_name FROM per_instance").fetchall() return [r[0] for r in rows] + @staticmethod + def _decode_outliers(raw): + """Parse a stored ``outliers`` JSON blob into a list of dicts. + + Rows written before the column existed read back as NULL/'' and legacy + files could in principle hold junk, so a parse failure degrades to "no + outliers" rather than breaking the whole history read. + """ + if not raw: + return [] + try: + parsed = json.loads(raw) + except (TypeError, ValueError): + return [] + if not isinstance(parsed, list): + return [] + out = [] + for item in parsed: + if not isinstance(item, dict): + continue + sample_id = str(item.get("sample_id", "")) + if not sample_id: + continue + try: + value = float(item.get("value", 0.0) or 0.0) + except (TypeError, ValueError): + continue + out.append({"sample_id": sample_id, "value": value}) + return out + + def get_step_outlier_sample_ids(self, metric_name: str, experiment_hash: str, + model_age: int) -> list: + """Sample ids flagged as off-trend for one point of one curve. + + Backs the plot's "Highlight step samples" action: the UI hands back the + metric/hash/step it was right-clicked on and gets the ids to filter the + data grid down to. + """ + with self._lock: + self._flush_stage() + params = [metric_name, int(model_age)] + sql = ("SELECT outliers FROM signals " + "WHERE metric_name = ? AND step = ?") + if experiment_hash: + sql += " AND experiment_hash = ?" + params.append(experiment_hash) + sql += " ORDER BY seq" + rows = self._conn.execute(sql, params).fetchall() + + seen, ids = set(), [] + for (raw,) in rows: + for item in self._decode_outliers(raw): + sample_id = item["sample_id"] + if sample_id and sample_id not in seen: + seen.add(sample_id) + ids.append(sample_id) + return ids + + def get_step_sample_ids(self, metric_name: str, experiment_hash: str, + model_age: int, max_samples: int = 0) -> tuple: + """Every sample id that contributed to one step of one signal. + + Backs the plot's "Highlight step samples" action, which shows the WHOLE + batch behind a point rather than only the off-trend members of it. + + Args: + metric_name: Signal name. + experiment_hash: Restrict to one run; ``""``/``None`` means any. + model_age: The step. + max_samples: Cap on returned ids (0 = no cap). + + Returns: + ``(ids, total_available)`` — *total_available* is the count before + the cap, so a caller can say "showing 200 of 4096". + """ + with self._lock: + self._flush_stage() + params = [metric_name, int(model_age)] + sql = ("SELECT DISTINCT sample_id FROM per_sample " + "WHERE metric_name = ? AND step = ?") + if experiment_hash: + sql += " AND experiment_hash = ?" + params.append(experiment_hash) + rows = self._conn.execute(sql, params).fetchall() + + ids = [str(row[0]) for row in rows if row[0] is not None] + # Numeric-aware ordering so "9" precedes "10"; falls back to plain text + # for non-numeric ids. + try: + ids.sort(key=lambda value: (0, int(value))) + except ValueError: + ids.sort() + total = len(ids) + if max_samples and max_samples > 0: + ids = ids[:max_samples] + return ids, total + def get_signal_history(self): """Reconstruct aggregated history as ``{metric: {hash: {step: [entry, ...]}}}``.""" with self._lock: @@ -967,13 +1309,17 @@ def get_signal_history(self): rows = self._conn.execute( """ SELECT metric_name, experiment_hash, step, metric_value, timestamp, - audit_mode, is_evaluation_marker, split_name, evaluation_tags, point_note + audit_mode, is_evaluation_marker, split_name, evaluation_tags, point_note, + outliers, outlier_count, sample_count, trend_value, trend_margin, + value_min, value_max FROM signals ORDER BY seq """ ).fetchall() result: dict = {} - for (metric, h, step, val, ts, audit, marker, split, tags, note) in rows: + for (metric, h, step, val, ts, audit, marker, split, tags, note, + outliers, outlier_count, sample_count, trend_value, trend_margin, + value_min, value_max) in rows: entry = { "model_age": step, "metric_name": metric, @@ -987,6 +1333,18 @@ def get_signal_history(self): } if note: entry["point_note"] = note + parsed_outliers = self._decode_outliers(outliers) + if parsed_outliers: + entry["outliers"] = parsed_outliers + entry["outlier_count"] = int(outlier_count or 0) + if sample_count: + entry["sample_count"] = int(sample_count) + if trend_value is not None and trend_margin is not None: + entry["trend_value"] = float(trend_value) + entry["trend_margin"] = float(trend_margin) + if value_min is not None and value_max is not None: + entry["value_min"] = float(value_min) + entry["value_max"] = float(value_max) result.setdefault(metric, {}).setdefault(h, {}).setdefault(step, []).append(entry) return result @@ -1527,6 +1885,13 @@ def _stage_entry(metric_name, exp_hash, step, entry): entry.get("split_name", ""), entry.get("evaluation_tags", []), entry.get("point_note", "") or "", + outliers=entry.get("outliers") or None, + outlier_count=int(entry.get("outlier_count", 0) or 0), + sample_count=int(entry.get("sample_count", 0) or 0), + trend_value=entry.get("trend_value"), + trend_margin=entry.get("trend_margin"), + value_min=entry.get("value_min"), + value_max=entry.get("value_max"), ) if isinstance(signals, dict): diff --git a/weightslab/components/model_signals.py b/weightslab/components/model_signals.py new file mode 100644 index 00000000..64c1a03e --- /dev/null +++ b/weightslab/components/model_signals.py @@ -0,0 +1,492 @@ +"""Per-step model signals: global gradient norm, per-layer weight/gradient/activation stats. + +Everything WeightsLab's public signal API records is keyed by something in the +dataset -- ``save_signals`` by sample, ``save_instance_signals`` by annotation, +``save_group_signals`` by group. Model health is not: a gradient norm belongs to +the optimization STEP that produced it, and the batch behind it is incidental. +``src.save_model_signals`` is the step-keyed write path; this module is what +produces the values so a training script never writes the collection loop by +hand. + +Signals emitted (all opt-in, see ``METRICS``): + + metrics/global/grad_norm whole-model gradient L2 norm + metrics/global/weights_norm whole-model parameter L2 norm + metrics/layer//grad_norm that layer's parameter gradients + metrics/layer//weights_norm that layer's parameters + metrics/layer//activation_{mean,std,max,min} + +```` is the module id WeightsLab already assigns for architecture ops +(``NetworkWithOps.get_layer_by_id`` / ``module.get_module_id()``), so a layer's +curve here and the same layer in the model panel / a freeze request name the +same thing. Layers without an id fall back to their position in +``model.layers``. + +Collection points, and why each one is where it is: + + weights read off ``p.data`` at flush time. Weights are always there; + no hook needed. + gradients ``Tensor.register_post_accumulate_grad_hook`` (torch >= 2.1), + which fires the instant a parameter's ``.grad`` is final + during backward. Deliberately NOT read at flush time: a + training loop is free to call ``optimizer.zero_grad()`` + before anything we control runs again, and by then the + gradients are gone. Reading them from the next forward hook + instead would report every step's gradients one step late. + activations forward hooks on the tracked layers, reduced on-device into + 0-d tensors and held there. Nothing is copied to the host + until the step flushes, which keeps the per-step cost to one + device sync no matter how many layers are tracked. + +The flush piggybacks on ``optimizer.step()``: the one point in a step where +gradients are guaranteed present AND the step is guaranteed finished. The +optimizer is resolved from the ledger LAZILY, on the first forward rather than +at construction, because a script watches its model BEFORE it builds the +optimizer from ``model.parameters()`` -- see ``_ensure_flush_hooked``. A script +that never registers an optimizer (or a custom loop that steps by hand) can +call ``tracker.flush()`` itself. +""" + +from __future__ import annotations + +import logging +from typing import Iterable, Optional + +import torch + +from weightslab.components.tracking import TrackingMode + +_LOGGER = logging.getLogger(__name__) + +# Every metric this module can emit. Also the default set: a first run should +# show the whole picture, and dropping metrics you don't want is easier than +# discovering ones you didn't know existed. Pass `metrics=[...]` to narrow. +METRICS = ( + "grad_norm", + "weights_norm", + "activation_mean", + "activation_std", + "activation_max", + "activation_min", +) + +_ACTIVATION_METRICS = frozenset(m for m in METRICS if m.startswith("activation_")) +_PARAM_METRICS = frozenset({"grad_norm", "weights_norm"}) + +# Module types whose output is not worth an activation curve: containers (their +# output is just their last child's) and shape-only ops (identical statistics to +# their input, so the curve duplicates the layer before it). +_UNINTERESTING_ACTIVATIONS = ( + torch.nn.Sequential, + torch.nn.ModuleList, + torch.nn.ModuleDict, + torch.nn.Flatten, + torch.nn.Identity, + torch.nn.Dropout, +) + + +def _layer_id(module: torch.nn.Module, position: int) -> str: + """The id this layer is known by elsewhere in WeightsLab, or its position. + + ``get_module_id`` is what the dependency manager assigns and what every + architecture op (freeze/reset/operate) and the model panel address a layer + by, so using it here is what makes a layer's signal curve and a layer's + controls refer to the same layer. A model whose dependencies were never + computed (``compute_dependencies=False``) has no ids at all, hence the + positional fallback. + """ + getter = getattr(module, "get_module_id", None) + if callable(getter): + try: + mid = getter() + if mid is not None: + return str(mid) + except Exception: + pass + return str(position) + + +def _iter_layers(model) -> list: + """``(layer_id, module)`` for each tracked layer, most specific source first. + + ``model.layers`` is WeightsLab's own linearized view (the same list the + model panel and architecture ops walk), so it is preferred -- it excludes + containers and is ordered the way the UI shows them. A plain ``nn.Module`` + that was never wrapped has no such attribute, so fall back to + ``named_modules()`` minus containers. + """ + layers = None + try: + candidate = getattr(model, "layers", None) + if candidate: + layers = list(candidate) + except Exception: + layers = None + + if layers: + return [(_layer_id(m, i), m) for i, m in enumerate(layers)] + + inner = getattr(model, "model", model) + out = [] + for i, (_, module) in enumerate(inner.named_modules()): + if module is inner or isinstance(module, (torch.nn.Sequential, torch.nn.ModuleList, torch.nn.ModuleDict)): + continue + out.append((_layer_id(module, i), module)) + return out + + +class ModelSignalTracker: + """Hooks on one model; emits its signals once per training step. + + One instance per model. Constructing it installs the hooks; ``remove()`` + takes them all off again (and is idempotent, so it is safe in a ``finally``). + """ + + def __init__( + self, + model, + metrics: Iterable[str] = METRICS, + every_n_steps: int = 1, + layer_ids: Optional[Iterable] = None, + include_global: bool = True, + ): + unknown = sorted(set(metrics) - set(METRICS)) + if unknown: + raise ValueError( + f"Unknown model signal(s) {unknown}. Available: {list(METRICS)}" + ) + + self.model = model + self.metrics = frozenset(metrics) + self.every_n_steps = max(1, int(every_n_steps)) + self.include_global = include_global + self._wanted = {str(l) for l in layer_ids} if layer_ids is not None else None + + # {layer_id: {metric: 0-d tensor}} for the step being collected. Values + # stay on-device until flush() -- see the module docstring. + self._activations: dict = {} + # {layer_id: [0-d tensor of grad**2, ...]}, one entry per parameter of + # that layer. Squared, so a layer norm is sqrt(sum(...)) and the global + # norm is sqrt(sum over every layer) -- summing the norms themselves + # would be wrong (that is an L1 over L2s, not an L2). + self._grad_sq: dict = {} + self._handles: list = [] + self._flush_hooked = False + self._removed = False + self._layers = _iter_layers(model) + if self._wanted is not None: + self._layers = [(lid, m) for lid, m in self._layers if lid in self._wanted] + + self._install() + + # -- setup ------------------------------------------------------------ # + + def _install(self) -> None: + want_activations = bool(self.metrics & _ACTIVATION_METRICS) + want_grads = "grad_norm" in self.metrics + + for layer_id, module in self._layers: + if want_activations and not isinstance(module, _UNINTERESTING_ACTIVATIONS): + self._handles.append( + module.register_forward_hook(self._make_activation_hook(layer_id)) + ) + if want_grads: + for param in module.parameters(recurse=False): + if not param.requires_grad: + continue + try: + self._handles.append( + param.register_post_accumulate_grad_hook( + self._make_grad_hook(layer_id) + ) + ) + except AttributeError: + # torch < 2.1. Gradients then simply aren't collected; + # weights and activations still are, and the run is not + # worth failing over a missing curve. + _LOGGER.warning( + "torch %s has no register_post_accumulate_grad_hook; " + "gradient-norm signals are unavailable", + torch.__version__, + ) + want_grads = False + break + + if not self._layers: + _LOGGER.warning("track_model_signals: no layers matched; nothing will be logged") + return + + # Driving _ensure_flush_hooked() from a pre-hook on the first tracked + # layer, rather than from the metric hooks themselves, covers the case + # where the requested metric set installs no per-step hooks at all + # (`metrics=["weights_norm"]` reads weights at flush time and hooks + # nothing) -- without this, that configuration would collect correctly + # and then never flush. + first_module = self._layers[0][1] + self._handles.append( + first_module.register_forward_pre_hook(lambda *_: self._ensure_flush_hooked()) + ) + + def _ensure_flush_hooked(self) -> None: + """Wrap the watched optimizer's ``step`` so flush() runs once per step. + + Lazy on purpose: a script calls ``watch_or_edit(model, flag="model")`` + before it builds the optimizer out of ``model.parameters()``, so at + construction time there is nothing to wrap yet. By the first forward + pass there always is. + """ + if self._flush_hooked: + return + self._flush_hooked = True # set first: one attempt, success or not + + from weightslab.backend.ledgers import get_optimizer + + try: + optimizer = get_optimizer() + except Exception: + optimizer = None + step_fn = getattr(optimizer, "step", None) + if optimizer is None or not callable(step_fn): + _LOGGER.info( + "track_model_signals: no watched optimizer found; call " + "tracker.flush() yourself after backward()" + ) + return + + tracker = self + + def step_and_flush(*args, **kwargs): + # Flush BEFORE stepping: these signals describe the gradients and + # the weights that this step is about to consume, which is the + # pairing that makes them readable together ("this gradient norm + # was applied to those weights"). Flushing after would report the + # post-update weights against the pre-update gradients. + try: + tracker.flush() + except Exception: + _LOGGER.debug("model signal flush failed", exc_info=True) + return step_fn(*args, **kwargs) + + try: + optimizer.step = step_and_flush + except Exception: + _LOGGER.info("track_model_signals: could not wrap optimizer.step; call flush() yourself") + + # -- hooks ------------------------------------------------------------ # + + def _should_collect(self) -> bool: + """Only inside a training step, and only on a sampled step. + + ``tracking_mode`` is set by ``guard_training_context`` / + ``guard_testing_context``, which is a stronger gate than + ``model.training`` or ``torch.is_grad_enabled()``: several shipped + examples run their eval pass without calling ``model.eval()`` and + without ``torch.no_grad()``, so both of those would happily report + eval-pass activations as training signals. + """ + if self._removed: + return False + if getattr(self.model, "tracking_mode", None) is not TrackingMode.TRAIN: + return False + if self.every_n_steps == 1: + return True + age = self._age() + return age is not None and age % self.every_n_steps == 0 + + def _age(self) -> Optional[int]: + getter = getattr(self.model, "get_age", None) + if callable(getter): + try: + value = getter() + return None if value is None else int(value) + except Exception: + return None + return None + + def _make_activation_hook(self, layer_id: str): + def hook(_module, _inputs, output): + if not self._should_collect(): + return + tensor = output if isinstance(output, torch.Tensor) else None + if tensor is None and isinstance(output, (tuple, list)) and output: + tensor = output[0] if isinstance(output[0], torch.Tensor) else None + if tensor is None or tensor.numel() == 0: + return + # Reduce on-device and keep it there; float() casts half/bf16 so a + # mixed-precision run doesn't overflow the sum inside std/mean. + values = tensor.detach().float() + stats = {} + if "activation_mean" in self.metrics: + stats["activation_mean"] = values.mean() + if "activation_std" in self.metrics: + # std() of a single element is NaN by definition (zero degrees + # of freedom); report 0 spread instead of poisoning the curve. + stats["activation_std"] = ( + values.std() if values.numel() > 1 else torch.zeros((), device=values.device) + ) + if "activation_max" in self.metrics: + stats["activation_max"] = values.max() + if "activation_min" in self.metrics: + stats["activation_min"] = values.min() + # A layer called twice in one forward (weight sharing, a recurrent + # block) keeps its LAST call, matching how the rest of WeightsLab + # treats a repeated write within a step. + self._activations[layer_id] = stats + + return hook + + def _make_grad_hook(self, layer_id: str): + def hook(param): + if not self._should_collect(): + return + grad = param.grad + if grad is None: + return + self._grad_sq.setdefault(layer_id, []).append( + grad.detach().float().pow(2).sum() + ) + + return hook + + # -- emit -------------------------------------------------------------- # + + def flush(self, step: Optional[int] = None) -> dict: + """Emit everything collected for the current step and reset. + + Called automatically from the wrapped ``optimizer.step()``; public + because a loop that steps by hand (or has no watched optimizer) needs + to drive it itself. Returns the ``{name: value}`` map it emitted, which + is also what makes it straightforward to assert on in a test. + """ + activations, grad_sq = self._activations, self._grad_sq + self._activations, self._grad_sq = {}, {} + + if self._removed: + return {} + + want_weights = "weights_norm" in self.metrics + if not activations and not grad_sq and not want_weights: + return {} + # Nothing was collected this step (an unsampled step, or an eval pass): + # don't compute weight norms either, or they would be the one metric + # logged at a different cadence than everything else. + if want_weights and not activations and not grad_sq and self.metrics & (_ACTIVATION_METRICS | {"grad_norm"}): + if not self._should_collect(): + return {} + + # Build the whole batch as on-device 0-d tensors, then convert ONCE. + # A per-metric .item() here would be one host<->device sync per layer + # per step, which is exactly the kind of cost that makes people turn + # instrumentation off. + names: list = [] + tensors: list = [] + + def add(name: str, value: torch.Tensor) -> None: + names.append(name) + tensors.append(value.reshape(())) + + for layer_id, stats in activations.items(): + for metric, value in stats.items(): + add(f"metrics/layer/{layer_id}/{metric}", value) + + global_grad_sq = [] + for layer_id, squares in grad_sq.items(): + total = squares[0] if len(squares) == 1 else torch.stack(squares).sum() + global_grad_sq.append(total) + add(f"metrics/layer/{layer_id}/grad_norm", total.sqrt()) + + global_weight_sq = [] + if want_weights: + for layer_id, module in self._layers: + squares = [ + p.detach().float().pow(2).sum() + for p in module.parameters(recurse=False) + ] + if not squares: + continue + total = squares[0] if len(squares) == 1 else torch.stack(squares).sum() + global_weight_sq.append(total) + add(f"metrics/layer/{layer_id}/weights_norm", total.sqrt()) + + if self.include_global: + if global_grad_sq: + add("metrics/global/grad_norm", torch.stack(global_grad_sq).sum().sqrt()) + if global_weight_sq: + add("metrics/global/weights_norm", torch.stack(global_weight_sq).sum().sqrt()) + + if not names: + return {} + + # The single sync for the whole step. + values = torch.stack(tensors).cpu().tolist() + signals = dict(zip(names, values)) + + from weightslab.src import save_model_signals + + save_model_signals(signals, step=step) + return signals + + # -- teardown ---------------------------------------------------------- # + + def remove(self) -> None: + """Take every hook back off. Idempotent.""" + self._removed = True + for handle in self._handles: + try: + handle.remove() + except Exception: + pass + self._handles.clear() + self._activations.clear() + self._grad_sq.clear() + + +def track_model_signals( + model=None, + metrics: Iterable[str] = METRICS, + every_n_steps: int = 1, + layer_ids: Optional[Iterable] = None, + include_global: bool = True, +) -> ModelSignalTracker: + """Instrument a watched model so its training dynamics log themselves. + + Args: + model: The watched model (what ``watch_or_edit(..., flag="model")`` + returned). Resolved from the ledger when omitted. + metrics: Which of :data:`METRICS` to emit. Defaults to all of them. + every_n_steps: Sample every Nth step instead of every step. Activation + hooks are the only per-step cost worth thinking about on a large + model; raising this is how you make that cost negligible. + layer_ids: Restrict to these layer ids (as reported by the model panel + / ``get_module_id()``). ``None`` tracks every layer. + include_global: Also emit ``metrics/global/{grad_norm,weights_norm}``, + the whole-model L2 norms. + + Returns: + ModelSignalTracker: keep it if you want ``.flush()`` or ``.remove()``; + ignoring it is fine, the hooks are already installed. + + Example: + ``model = wl.watch_or_edit(net, flag="model", track_model_signals=True)`` + does this for you. Called directly:: + + model = wl.watch_or_edit(net, flag="model", device=device) + wl.track_model_signals(model, every_n_steps=10) + """ + if model is None: + from weightslab.backend.ledgers import get_model + + model = get_model() + if model is None: + raise ValueError( + "No model to track. Call wl.watch_or_edit(model, flag='model') first, " + "or pass the model explicitly." + ) + return ModelSignalTracker( + model, + metrics=metrics, + every_n_steps=every_n_steps, + layer_ids=layer_ids, + include_global=include_global, + ) diff --git a/weightslab/examples/Notebooks/Local/wl-local-studio-quickstart.ipynb b/weightslab/examples/Notebooks/Local/wl-local-studio-quickstart.ipynb index 80027971..5894b4db 100644 --- a/weightslab/examples/Notebooks/Local/wl-local-studio-quickstart.ipynb +++ b/weightslab/examples/Notebooks/Local/wl-local-studio-quickstart.ipynb @@ -39,8 +39,6 @@ ] }, { -<<<<<<< HEAD -======= "cell_type": "code", "execution_count": null, "id": "661992d3", @@ -52,7 +50,6 @@ ] }, { ->>>>>>> origin/main "cell_type": "markdown", "metadata": {}, "source": [ @@ -65,9 +62,6 @@ "id": "db7ea2f5", "metadata": {}, "outputs": [], -<<<<<<< HEAD - "source": "import os\n\n# Imports weightslab\nimport weightslab as wl\n\n# Serve the weightslab app\nwl.serve()\n\n# Define root log dir for experiment\nroot_log_dir = os.environ.get(\"WEIGHTSLAB_ROOT_LOG_DIR\", None)" -======= "source": [ "import os\n", "\n", @@ -80,7 +74,6 @@ "# Define root log dir for experiment\n", "root_log_dir = os.environ.get(\"WEIGHTSLAB_ROOT_LOG_DIR\", None)" ] ->>>>>>> origin/main } ], "metadata": { @@ -95,8 +88,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -<<<<<<< HEAD } -======= -} ->>>>>>> origin/main diff --git a/weightslab/examples/Usecases/wl-fashion-mnist-signals/config.yaml b/weightslab/examples/Usecases/wl-fashion-mnist-signals/config.yaml new file mode 100644 index 00000000..454edc30 --- /dev/null +++ b/weightslab/examples/Usecases/wl-fashion-mnist-signals/config.yaml @@ -0,0 +1,43 @@ +# Fashion-MNIST classification with per-step model signals. +experiment_name: fashion_mnist_model_signals +device: auto +training_steps_to_do: 3000 # null = train until stopped from the UI +# root_log_dir: # defaults to $WEIGHTSLAB_ROOT_LOG_DIR, else a temp dir + +optimizer: + lr: 0.001 + +# How often the gradient/weight/activation curves are sampled. 1 = every step. +# The activation forward hooks are the only per-step cost worth thinking about: +# on a large model set this to 10-50 and the overhead becomes negligible while +# the curves stay just as readable. +model_signals_every_n_steps: 1 + +# Experiment cadence +eval_full_to_train_steps_ratio: 250 +experiment_dump_to_train_steps_ratio: 500 +write_export_ratio: 250 +skip_checkpoint_load: false +tqdm_display: true +is_training: false +compute_natural_sort: false + +# Global dataframe storage +ledger_enable_flushing_threads: true +ledger_enable_h5_persistence: true +ledger_flush_max_rows: 15000 +ledger_flush_interval: 30.0 + +serving_grpc: true + +# data_root: ./data # auto-downloaded from the Fashion-MNIST mirror +data: + train_loader: + shuffle: true + batch_size: 8 + max_samples: 8192 # raise (or drop) for the full 60k + test_loader: + shuffle: false + batch_size: 8 + max_samples: 2048 + drop_last: false diff --git a/weightslab/examples/Usecases/wl-fashion-mnist-signals/main.py b/weightslab/examples/Usecases/wl-fashion-mnist-signals/main.py new file mode 100644 index 00000000..11af1a21 --- /dev/null +++ b/weightslab/examples/Usecases/wl-fashion-mnist-signals/main.py @@ -0,0 +1,441 @@ +"""Fashion-MNIST classification with per-step MODEL signals. + +Same shape as ``examples/PyTorch/wl-classification`` (plain PyTorch loop, +watched model/optimizer/loaders/loss/metric, guard contexts) with one addition: +this run also plots its own training dynamics. + + metrics/global/grad_norm whole-model gradient L2 norm + metrics/global/weights_norm whole-model parameter L2 norm + metrics/layer//grad_norm per-layer parameter gradients + metrics/layer//weights_norm per-layer parameters + metrics/layer//activation_mean + metrics/layer//activation_std + metrics/layer//activation_max + metrics/layer//activation_min + +All of it comes from ONE argument -- ``track_model_signals=True`` on the model's +``watch_or_edit`` (see MODEL below). No hooks, no per-step bookkeeping, and no +call anywhere in the training loop: gradients are read by post-accumulate hooks +the moment they are final, activations by forward hooks, and the whole set is +flushed once per step just before ``optimizer.step()`` consumes it. + +Why these signals are NOT ``wl.save_signals``: everything that verb records is +keyed by a sample, and a gradient norm does not belong to any sample -- it +belongs to the step. ``wl.save_model_signals`` is the step-keyed write path (use +it directly for any dynamics value of your own, e.g. a gradient-to-weight +ratio); ``wl.track_model_signals`` is the collector that fills it in for you. + +What the curves are for -- Fashion-MNIST makes each failure legible: + + grad_norm collapsing toward 0 in the EARLY layers while the late ones stay + healthy is vanishing gradient; the run keeps "training" and stops learning. + + grad_norm spiking by orders of magnitude is the exploding case -- pair it + with the loss curve to see which came first. + + activation_std -> 0 on a layer is that layer going constant (dead ReLUs, + saturated BatchNorm): it is still consuming compute and contributing + nothing. activation_min stuck at exactly 0.0 across a whole ReLU is the + same story from the other side. + + weights_norm climbing without bound while the loss flattens is the model + growing weights instead of learning structure -- the moment to add decay. + +Run:: + + python main.py # reads config.yaml next to this file + WEIGHTSLAB_ROOT_LOG_DIR= python main.py +""" + +import itertools +import logging +import os +import ssl +import tempfile +import time + +# Windows SSL fix: some Windows cert stores contain malformed ASN1 certs that +# crash ssl.create_default_context(). Fall back to unverified only when broken. +try: + ssl.create_default_context() +except ssl.SSLError: + ssl._create_default_https_context = ssl._create_unverified_context + +import torch +import torch.nn as nn +import torch.optim as optim +import tqdm +import yaml +from torch.utils.data import Dataset +from torchmetrics.classification import Accuracy +from torchvision import datasets, transforms + +import weightslab as wl +from weightslab.components.global_monitoring import ( + guard_testing_context, + guard_training_context, +) + +logging.basicConfig(level=logging.ERROR) +logger = logging.getLogger(__name__) + +# Fashion-MNIST's own label order (torchvision docs). Carried as per-sample +# metadata so the grid shows "Pullover" instead of "2" -- which is the +# difference between spotting a shirt/coat/pullover confusion and not. +CLASS_NAMES = ( + "T-shirt/top", "Trouser", "Pullover", "Dress", "Coat", + "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot", +) + + +# ============================================================================= +# Dataset +# ============================================================================= +class FashionMNISTDataset(Dataset): + """Fashion-MNIST yielding ``(image, sample_id, label)``. + + ``sample_id`` is offset per split (``id_base``) so train and test ids never + collide in the shared ledger -- without it, test sample 0 would overwrite + train sample 0's signal history. + """ + + def __init__(self, root, train=True, download=True, transform=None, + max_samples=None, id_base=0): + try: + self.data = datasets.FashionMNIST(root=root, train=train, + download=download, transform=None) + except RuntimeError as exc: + logger.error(f"Error loading Fashion-MNIST: {exc}") + self.data = datasets.FashionMNIST(root=root, train=train, + download=True, transform=None) + self.transform = transform + self.train = train + self.max_samples = max_samples + self.id_base = id_base + + def __len__(self): + if self.max_samples is not None: + return min(len(self.data), self.max_samples) + return len(self.data) + + def __getitem__(self, idx): + image, label = self.data[idx] + if self.transform: + image = self.transform(image) + return image, self.id_base + idx, label + + def fast_get_label(self, idx): + """Lets the ledger read labels at init without decoding every image.""" + return int(self.data.targets[idx]) + + def get_metadata(self, idx): + """Per-sample metadata surfaced in the grid / metadata panel.""" + label = int(self.data.targets[idx]) + return { + "class_name": CLASS_NAMES[label], + "split": "train" if self.train else "test", + } + + +# ============================================================================= +# Model +# ============================================================================= +class FashionCNN(nn.Module): + """Three conv blocks + a two-layer head. + + Deliberately deeper than the task needs: per-layer signals only tell you + something once there are enough layers for the early ones to behave + differently from the late ones. Every module is a named attribute (no + ``nn.Sequential``) so each gets its own layer id and therefore its own + curve -- a Sequential block would collapse to one. + """ + + def __init__(self, num_classes=10): + super().__init__() + self.input_shape = (1, 1, 28, 28) + + self.conv1 = nn.Conv2d(1, 16, 3, padding=1) + self.bn1 = nn.BatchNorm2d(16) + self.relu1 = nn.ReLU() + self.pool1 = nn.MaxPool2d(2) # 28 -> 14 + + self.conv2 = nn.Conv2d(16, 32, 3, padding=1) + self.bn2 = nn.BatchNorm2d(32) + self.relu2 = nn.ReLU() + self.pool2 = nn.MaxPool2d(2) # 14 -> 7 + + self.conv3 = nn.Conv2d(32, 64, 3, padding=1) + self.bn3 = nn.BatchNorm2d(64) + self.relu3 = nn.ReLU() + + self.flatten = nn.Flatten() + self.fc1 = nn.Linear(64 * 7 * 7, 128) + self.relu4 = nn.ReLU() + self.fc2 = nn.Linear(128, num_classes) + + def forward(self, x): + x = self.pool1(self.relu1(self.bn1(self.conv1(x)))) + x = self.pool2(self.relu2(self.bn2(self.conv2(x)))) + x = self.relu3(self.bn3(self.conv3(x))) + # Logits, not softmax: the watched CrossEntropyLoss applies its own. + return self.fc2(self.relu4(self.fc1(self.flatten(x)))) + + +def print_layer_legend(model): + """Map each layer id to its module, once, at startup. + + ``metrics/layer/7/grad_norm`` says nothing on its own. These are the same + ids the model panel and every architecture op (freeze/reset) use, so this + legend is what lets you read a curve and act on the layer behind it. + """ + from weightslab.components.model_signals import _iter_layers + + rows = _iter_layers(model) + if not rows: + print(" (no layer ids resolved -- per-layer curves will be positional)") + return + print(f" {'layer_id':>9} {'module':<14} shape") + for layer_id, module in rows: + weight = getattr(module, "weight", None) + shape = tuple(weight.shape) if weight is not None else "-" + print(f" {layer_id:>9} {type(module).__name__:<14} {shape}") + + +# ----------------------------------------------------------------------------- +# Train / test +# ----------------------------------------------------------------------------- +def train(loader, model, optimizer, criterion, device): + """One training step. Nothing here logs model signals -- the hooks do.""" + with guard_training_context: + inputs, ids, labels = next(loader) + inputs = inputs.to(device) + labels = labels.to(device) + + optimizer.zero_grad() + logits = model(inputs) + preds = logits.argmax(dim=1, keepdim=True) + + loss_per_sample = criterion( + logits.float(), labels.long(), batch_ids=ids, preds=preds, + ) + total_loss = loss_per_sample.mean() + total_loss.backward() + optimizer.step() + + return total_loss.detach().cpu().item() + + +def test(loader, model, criterion, metric, device, num_batches): + """Full pass over the test split. + + No model signals come out of this: the tracker only collects inside + ``guard_training_context``, so an eval pass can't contaminate a gradient or + activation curve with values the optimizer never saw. + """ + losses = torch.tensor(0.0, device=device) + + for inputs, ids, labels in loader: + with guard_testing_context, torch.no_grad(): + inputs = inputs.to(device) + labels = labels.to(device) + + logits = model(inputs) + preds = logits.argmax(dim=1, keepdim=True) + + losses += criterion( + logits, labels, batch_ids=ids, preds=preds, + ).mean() + metric.update(logits, labels) + + correct = (preds.view(-1) == labels.view(-1)).float() + wl.save_signals( + signals={ + "test_metric/accuracy_per_sample": correct, + "test_metric/error_per_sample": 1.0 - correct, + }, + batch_ids=ids, + preds_raw=logits, + targets=labels, + preds=preds, + ) + + return (losses / num_batches).detach().cpu().item(), (metric.compute() * 100).detach().cpu().item() + + +# ----------------------------------------------------------------------------- +# Main +# ----------------------------------------------------------------------------- +if __name__ == "__main__": + start_time = time.time() + + parameters = {} + config_path = os.path.join(os.path.dirname(__file__), "config.yaml") + if os.path.exists(config_path): + with open(config_path, "r") as fh: + parameters = yaml.safe_load(fh) or {} + + parameters.setdefault("experiment_name", "fashion_mnist_model_signals") + parameters.setdefault("device", "auto") + parameters.setdefault("training_steps_to_do", 3000) + parameters.setdefault("eval_full_to_train_steps_ratio", 250) + parameters.setdefault("model_signals_every_n_steps", 1) + + # Hyperparameters first: everything below reads from the watched dict, so a + # value edited in the UI is picked up without restarting. + wl.watch_or_edit(parameters, flag="hyperparameters", poll_interval=1.0) + + if parameters.get("device", "auto") == "auto": + parameters["device"] = torch.device("cuda" if torch.cuda.is_available() else "cpu") + device = parameters["device"] + + # `weightslab start ` exports WEIGHTSLAB_ROOT_LOG_DIR -- honor it, so + # this run lands in the directory the dashboard is actually watching. A + # temp dir only when nothing said otherwise. + if not parameters.get("root_log_dir"): + parameters["root_log_dir"] = os.environ.get("WEIGHTSLAB_ROOT_LOG_DIR") or tempfile.mkdtemp() + os.makedirs(parameters["root_log_dir"], exist_ok=True) + log_dir = parameters["root_log_dir"] + + verbose = parameters.get("verbose", True) + tqdm_display = parameters.get("tqdm_display", True) + eval_ratio = parameters.get("eval_full_to_train_steps_ratio", 250) + enable_h5 = parameters.get("enable_h5_persistence", True) + steps_to_do = parameters.get("training_steps_to_do", 3000) + + # ---- MODEL ------------------------------------------------------------- + # `track_model_signals=True` is the whole feature: it installs the hooks + # that produce every metrics/* curve. Pass a list to narrow the set, e.g. + # track_model_signals=["grad_norm", "activation_std"]. + # + # `model_signals_every_n_steps` samples every Nth step. The activation + # hooks are the only per-step cost worth thinking about; on a big model + # raise this to 10-50 and the overhead disappears while the curves stay + # perfectly readable. + model = wl.watch_or_edit( + FashionCNN(num_classes=len(CLASS_NAMES)).to(device), + flag="model", + device=device, + track_model_signals=True, + model_signals_every_n_steps=parameters.get("model_signals_every_n_steps", 1), + ) + + # Build the optimizer from the WATCHED model's parameters, not the raw one. + lr = parameters.get("optimizer", {}).get("lr", 0.001) + optimizer = wl.watch_or_edit(optim.Adam(model.parameters(), lr=lr), flag="optimizer") + + # ---- DATA -------------------------------------------------------------- + if parameters.get("data_root"): + data_root = parameters["data_root"] + should_download = not os.path.exists(data_root) + else: + data_root = os.path.join(log_dir, "data") + should_download = True + os.makedirs(data_root, exist_ok=True) + + train_cfg = parameters.get("data", {}).get("train_loader", {}) + test_cfg = parameters.get("data", {}).get("test_loader", {}) + + to_tensor = transforms.Compose([transforms.ToTensor()]) + + train_dataset = FashionMNISTDataset( + root=data_root, train=True, download=should_download, transform=to_tensor, + max_samples=train_cfg.get("max_samples"), id_base=0, + ) + test_dataset = FashionMNISTDataset( + root=data_root, train=False, download=should_download, transform=to_tensor, + max_samples=test_cfg.get("max_samples"), id_base=1_000_000, + ) + + train_loader = wl.watch_or_edit( + train_dataset, flag="data", loader_name="train_loader", + batch_size=train_cfg.get("batch_size", 64), + shuffle=train_cfg.get("shuffle", True), + is_training=True, compute_hash=False, + preload_labels=True, preload_metadata=True, + enable_h5_persistence=enable_h5, + ) + test_loader = wl.watch_or_edit( + test_dataset, flag="data", loader_name="test_loader", + batch_size=test_cfg.get("batch_size", 256), + shuffle=test_cfg.get("shuffle", False), + is_training=False, compute_hash=False, + preload_labels=True, preload_metadata=True, + enable_h5_persistence=enable_h5, + ) + + # ---- LOSS / METRIC ----------------------------------------------------- + train_criterion = wl.watch_or_edit( + nn.CrossEntropyLoss(reduction="none"), + flag="loss", signal_name="train-loss-CE", log=True, per_sample=True) + test_criterion = wl.watch_or_edit( + nn.CrossEntropyLoss(reduction="none"), + flag="loss", signal_name="test-loss-CE", log=True, per_sample=True) + metric = wl.watch_or_edit( + Accuracy(task="multiclass", num_classes=len(CLASS_NAMES)).to(device), + flag="metric", signal_name="metric-ACC", log=True) + + wl.serve(serving_grpc=parameters.get("serving_grpc", True)) + + print("=" * 72) + print(" FASHION-MNIST + PER-STEP MODEL SIGNALS") + print(f" train={len(train_dataset)} test={len(test_dataset)} device={device}") + print(f" eval every {eval_ratio} steps | model signals every " + f"{parameters.get('model_signals_every_n_steps', 1)} step(s)") + print(f" logs -> {log_dir}") + print("-" * 72) + print(" LAYER LEGEND (these ids name the metrics/layer//* curves)") + print_layer_legend(model) + print("=" * 72 + "\n") + + if tqdm_display: + train_range = tqdm.tqdm( + range(steps_to_do) if steps_to_do is not None else itertools.count(), + desc="Training", + bar_format="{desc}: {n}/{total} [{elapsed}<{remaining}, {rate_fmt}] {bar} | {postfix}", + ncols=140, position=0, leave=True, + ) + else: + train_range = range(steps_to_do) if steps_to_do is not None else itertools.count() + + wl.start_training(timeout=3) + + train_loss = None + test_loss, test_metric = None, None + test_batches = len(test_loader) + + for train_step in train_range: + age = model.get_age() if hasattr(model, "get_age") else train_step + + train_loss = train(train_loader, model, optimizer, train_criterion, device) + + if age > 0 and age % eval_ratio == 0: + test_loss, test_metric = test( + test_loader, model, test_criterion, metric, device, test_batches) + + if tqdm_display: + parts = [f"train_loss={train_loss:.4f}"] + if test_loss is not None: + parts.append(f"test_loss={test_loss:.4f}") + if test_metric is not None: + parts.append(f"test_acc={test_metric:.1f}%") + train_range.set_postfix_str(" | ".join(parts)) + elif verbose: + import sys + msg = f"Step {train_step} (age {age}): loss={train_loss:.4f}" + if test_loss is not None: + msg += f" | test={test_loss:.4f} ({test_metric:.1f}%)" + sys.stdout.write(f"\r{msg:<100}") + sys.stdout.flush() + + print("\n" + "=" * 72) + print(f" Done in {time.time() - start_time:.1f}s | logs -> {log_dir}") + print("=" * 72) + + # Flush async signals before reading anything back, then dump both the + # step-keyed signal history (where every metrics/* curve lives) and the + # per-sample grid. + wl.drain_signals() + wl.write_history() + wl.write_dataframe() + + wl.keep_serving() diff --git a/weightslab/proto/experiment_service.proto b/weightslab/proto/experiment_service.proto index d48ddfb9..cc77eb0a 100644 --- a/weightslab/proto/experiment_service.proto +++ b/weightslab/proto/experiment_service.proto @@ -30,6 +30,10 @@ service ExperimentService { // downsampled to at most max_points and is only included when the sample has // at least 3 recorded points. rpc GetSignalTrajectory (GetSignalTrajectoryRequest) returns (GetSignalTrajectoryResponse); + // Every sample id that contributed to one step of one signal. Backs the plot's + // right-click "Highlight step samples": the whole batch behind a point, not + // just the samples that were flagged as off-trend. + rpc GetStepSamples (StepSamplesRequest) returns (StepSamplesResponse); // Raw point cloud of one sample (task_type "detection_pointcloud"), server-streamed // in binary chunks for the interactive 3D viewer. rpc GetPointCloud (PointCloudRequest) returns (stream PointCloudChunk); @@ -79,6 +83,14 @@ message GetLatestLoggerDataRequest { string graph_name = 5; // specific signal/graph name to filter (used when break_by_slices=true) } +// One sample whose signal value at a step sits far off the curve's own trend. +// Emitted alongside the averaged point it was averaged into, so the UI can show +// "this step's mean is 0.39, but sample 8123 contributed 2.71". +message SignalOutlier { + string sample_id = 1; + float value = 2; // that sample's raw signal value at this step +} + message LoggerDataPoint { string metric_name = 1; // The metric/signal name (e.g., "train/loss", "eval/accuracy") int32 model_age = 2; @@ -91,6 +103,31 @@ message LoggerDataPoint { repeated string evaluation_tags = 9; string point_note = 10; bool audit_mode = 11; // True if logged during audit mode (read-only inspection) + // Samples in this step's batch that deviated from the curve's rolling trend, + // strongest first and capped at WL_SIGNAL_OUTLIER_TOP_N. Empty for the common + // case where the whole batch tracked the trend. + repeated SignalOutlier outliers = 12; + // How many samples in the batch were flagged, before the top-N cap. Lets the + // UI tell "one sample spiked" apart from "the entire batch is off-trend". + int32 outlier_count = 13; + // Batch size the average was taken over, so outlier_count can be read as a + // fraction of the batch rather than an absolute. + int32 sample_count = 14; + // The on-trend band this point was judged against: centre (the rolling EMA of + // the curve) and half-width. Not drawn directly; it is what decides which + // samples count as off-trend. has_trend_band is false while the tracker is + // still warming up. + float trend_value = 15; + float trend_margin = 16; + bool has_trend_band = 17; + // ABSOLUTE extremes of this step's batch — the real lowest and highest sample + // values, not a standard deviation. This is what the UI draws as the error + // band around the curve, so a step containing an outlier makes the band spike + // out to that value instead of averaging it away. + // has_value_range is false for signals that log no per-sample data. + float value_min = 18; + float value_max = 19; + bool has_value_range = 20; } message GetLatestLoggerDataResponse { @@ -496,6 +533,21 @@ message GetMetaDataResponse { } // --- On-demand per-signal trajectory (right-click a signal -> plot) --- +// --- All samples behind one plotted step --- +message StepSamplesRequest { + string metric_name = 1; + string experiment_hash = 2; // "" = any run + int32 model_age = 3; + int32 max_samples = 4; // 0 => no cap +} + +message StepSamplesResponse { + bool success = 1; + string message = 2; + repeated string sample_ids = 3; + int32 total_available = 4; // count before max_samples truncation +} + message GetSignalTrajectoryRequest { string signal_name = 1; // signal to plot (UI spelling; resolved server-side) repeated string sample_ids = 2; // samples currently shown to plot for diff --git a/weightslab/proto/experiment_service_pb2.py b/weightslab/proto/experiment_service_pb2.py index dd2af12c..8544b930 100644 --- a/weightslab/proto/experiment_service_pb2.py +++ b/weightslab/proto/experiment_service_pb2.py @@ -24,7 +24,7 @@ -DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n)weightslab/proto/experiment_service.proto\"\x89\x01\n\x1aGetLatestLoggerDataRequest\x12\x1c\n\x14request_full_history\x18\x01 \x01(\x08\x12\x12\n\nmax_points\x18\x02 \x01(\x05\x12\x17\n\x0f\x62reak_by_slices\x18\x03 \x01(\x08\x12\x0c\n\x04tags\x18\x04 \x03(\t\x12\x12\n\ngraph_name\x18\x05 \x01(\t\"\x81\x02\n\x0fLoggerDataPoint\x12\x13\n\x0bmetric_name\x18\x01 \x01(\t\x12\x11\n\tmodel_age\x18\x02 \x01(\x05\x12\x14\n\x0cmetric_value\x18\x03 \x01(\x02\x12\x17\n\x0f\x65xperiment_hash\x18\x04 \x01(\t\x12\x11\n\ttimestamp\x18\x05 \x01(\x03\x12\x11\n\tsample_id\x18\x06 \x01(\t\x12\x1c\n\x14is_evaluation_marker\x18\x07 \x01(\x08\x12\x12\n\nsplit_name\x18\x08 \x01(\t\x12\x17\n\x0f\x65valuation_tags\x18\t \x03(\t\x12\x12\n\npoint_note\x18\n \x01(\t\x12\x12\n\naudit_mode\x18\x0b \x01(\x08\"[\n\x1bGetLatestLoggerDataResponse\x12 \n\x06points\x18\x01 \x03(\x0b\x32\x10.LoggerDataPoint\x12\x1a\n\x12weightslab_version\x18\x02 \x01(\t\"\x07\n\x05\x45mpty\"/\n\x08NeuronId\x12\x10\n\x08layer_id\x18\x01 \x01(\x05\x12\x11\n\tneuron_id\x18\x02 \x01(\x05\"\x91\x02\n\x0fWeightOperation\x12*\n\x07op_type\x18\x01 \x01(\x0e\x32\x14.WeightOperationTypeH\x00\x88\x01\x01\x12\x15\n\x08layer_id\x18\x02 \x01(\x05H\x01\x88\x01\x01\x12\x1d\n\nneuron_ids\x18\x03 \x03(\x0b\x32\t.NeuronId\x12\x16\n\x0eneurons_to_add\x18\t \x01(\x05\x12 \n\x18zerofy_from_incoming_ids\x18\x0b \x03(\x05\x12\x1c\n\x14zerofy_to_neuron_ids\x18\x0c \x03(\x05\x12+\n\x11zerofy_predicates\x18\r \x03(\x0e\x32\x10.ZerofyPredicateB\n\n\x08_op_typeB\x0b\n\t_layer_id\"_\n\x17WeightsOperationRequest\x12/\n\x10weight_operation\x18\x01 \x01(\x0b\x32\x10.WeightOperationH\x00\x88\x01\x01\x42\x13\n\x11_weight_operation\"<\n\x18WeightsOperationResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"\xc1\x05\n\x0fHyperParameters\x12\x1c\n\x0f\x65xperiment_name\x18\x01 \x01(\tH\x00\x88\x01\x01\x12!\n\x14training_steps_to_do\x18\x02 \x01(\x05H\x01\x88\x01\x01\x12\x1a\n\rlearning_rate\x18\x03 \x01(\x02H\x02\x88\x01\x01\x12\x17\n\nbatch_size\x18\x04 \x01(\x05H\x03\x88\x01\x01\x12 \n\x13\x66ull_eval_frequency\x18\x05 \x01(\x05H\x04\x88\x01\x01\x12 \n\x13\x63heckpont_frequency\x18\x06 \x01(\x05H\x05\x88\x01\x01\x12\x18\n\x0bis_training\x18\x07 \x01(\x08H\x06\x88\x01\x01\x12\x15\n\x08nb_steps\x18\x08 \x01(\x05H\x07\x88\x01\x01\x12\x19\n\x0c\x61uditor_mode\x18\t \x01(\x08H\x08\x88\x01\x01\x12\x1d\n\x10train_batch_size\x18\n \x01(\x05H\t\x88\x01\x01\x12\x1b\n\x0eval_batch_size\x18\x0b \x01(\x05H\n\x88\x01\x01\x12\x1c\n\x0ftest_batch_size\x18\x0c \x01(\x05H\x0b\x88\x01\x01\x12\x1c\n\x0f\x65valuation_mode\x18\r \x01(\x08H\x0c\x88\x01\x01\x12\x1e\n\x11\x65valuation_config\x18\x0e \x01(\tH\r\x88\x01\x01\x42\x12\n\x10_experiment_nameB\x17\n\x15_training_steps_to_doB\x10\n\x0e_learning_rateB\r\n\x0b_batch_sizeB\x16\n\x14_full_eval_frequencyB\x16\n\x14_checkpont_frequencyB\x0e\n\x0c_is_trainingB\x0b\n\t_nb_stepsB\x0f\n\r_auditor_modeB\x13\n\x11_train_batch_sizeB\x11\n\x0f_val_batch_sizeB\x12\n\x10_test_batch_sizeB\x12\n\x10_evaluation_modeB\x14\n\x12_evaluation_config\",\n\rMetricsStatus\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\x02\"~\n\rAnnotatStatus\x12\x0c\n\x04name\x18\x01 \x01(\t\x12.\n\x08metadata\x18\x02 \x03(\x0b\x32\x1c.AnnotatStatus.MetadataEntry\x1a/\n\rMetadataEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\x02:\x02\x38\x01\"\x90\x02\n\x10TrainingStatusEx\x12\x16\n\ttimestamp\x18\x01 \x01(\tH\x00\x88\x01\x01\x12\x1c\n\x0f\x65xperiment_name\x18\x02 \x01(\tH\x01\x88\x01\x01\x12\x16\n\tmodel_age\x18\x03 \x01(\x05H\x02\x88\x01\x01\x12+\n\x0emetrics_status\x18\x04 \x01(\x0b\x32\x0e.MetricsStatusH\x03\x88\x01\x01\x12+\n\x0e\x61nnotat_status\x18\x05 \x01(\x0b\x32\x0e.AnnotatStatusH\x04\x88\x01\x01\x42\x0c\n\n_timestampB\x12\n\x10_experiment_nameB\x0c\n\n_model_ageB\x11\n\x0f_metrics_statusB\x11\n\x0f_annotat_status\"]\n\x15HyperParameterCommand\x12/\n\x10hyper_parameters\x18\x01 \x01(\x0b\x32\x10.HyperParametersH\x00\x88\x01\x01\x42\x13\n\x11_hyper_parameters\">\n\x14\x44\x65nySamplesOperation\x12\x12\n\nsample_ids\x18\x01 \x03(\t\x12\x12\n\naccumulate\x18\x02 \x01(\x08\"0\n\x17LoadCheckpointOperation\x12\x15\n\rcheckpoint_id\x18\x01 \x01(\x05\"b\n\x11PlotNoteOperation\x12\x13\n\x0bmetric_name\x18\x01 \x01(\t\x12\x17\n\x0f\x65xperiment_hash\x18\x02 \x01(\t\x12\x11\n\tmodel_age\x18\x03 \x01(\x05\x12\x0c\n\x04note\x18\x04 \x01(\t\"L\n\x17SaveCheckpointOperation\x12\x19\n\x11save_architecture\x18\x01 \x01(\x08\x12\x16\n\x0esave_optimizer\x18\x02 \x01(\x08\"\x1a\n\x18RestartInstanceOperation\"\x8d\x08\n\x0eTrainerCommand\x12\x1c\n\x14get_hyper_parameters\x18\x04 \x01(\x08\x12\x1e\n\x16get_interactive_layers\x18\x05 \x01(\x08\x12\x1d\n\x10get_data_records\x18\x06 \x01(\tH\x00\x88\x01\x01\x12%\n\x18get_single_layer_info_id\x18\x08 \x01(\x05H\x01\x88\x01\x01\x12;\n\x16hyper_parameter_change\x18\x01 \x01(\x0b\x32\x16.HyperParameterCommandH\x02\x88\x01\x01\x12:\n\x16\x64\x65ny_samples_operation\x18\x07 \x01(\x0b\x32\x15.DenySamplesOperationH\x03\x88\x01\x01\x12?\n\x1b\x64\x65ny_eval_samples_operation\x18\n \x01(\x0b\x32\x15.DenySamplesOperationH\x04\x88\x01\x01\x12@\n\x19load_checkpoint_operation\x18\t \x01(\x0b\x32\x18.LoadCheckpointOperationH\x05\x88\x01\x01\x12\x42\n\x1eremove_from_denylist_operation\x18\x0b \x01(\x0b\x32\x15.DenySamplesOperationH\x06\x88\x01\x01\x12G\n#remove_eval_from_denylist_operation\x18\x0c \x01(\x0b\x32\x15.DenySamplesOperationH\x07\x88\x01\x01\x12\x34\n\x13plot_note_operation\x18\r \x01(\x0b\x32\x12.PlotNoteOperationH\x08\x88\x01\x01\x12@\n\x19save_checkpoint_operation\x18\x0e \x01(\x0b\x32\x18.SaveCheckpointOperationH\t\x88\x01\x01\x12\x39\n\x11restart_operation\x18\x0f \x01(\x0b\x32\x19.RestartInstanceOperationH\n\x88\x01\x01\x42\x13\n\x11_get_data_recordsB\x1b\n\x19_get_single_layer_info_idB\x19\n\x17_hyper_parameter_changeB\x19\n\x17_deny_samples_operationB\x1e\n\x1c_deny_eval_samples_operationB\x1c\n\x1a_load_checkpoint_operationB!\n\x1f_remove_from_denylist_operationB&\n$_remove_eval_from_denylist_operationB\x16\n\x14_plot_note_operationB\x1c\n\x1a_save_checkpoint_operationB\x14\n\x12_restart_operation\"\x9d\x01\n\x12HyperParameterDesc\x12\r\n\x05label\x18\x01 \x01(\t\x12\x0c\n\x04name\x18\x02 \x01(\t\x12\x0c\n\x04type\x18\x03 \x01(\t\x12\x1c\n\x0fnumerical_value\x18\x04 \x01(\x02H\x00\x88\x01\x01\x12\x19\n\x0cstring_value\x18\x05 \x01(\tH\x01\x88\x01\x01\x42\x12\n\x10_numerical_valueB\x0f\n\r_string_value\"\xf2\x02\n\x10NeuronStatistics\x12!\n\tneuron_id\x18\x01 \x01(\x0b\x32\t.NeuronIdH\x00\x88\x01\x01\x12\x17\n\nneuron_age\x18\x02 \x01(\x05H\x01\x88\x01\x01\x12\x1f\n\x12train_trigger_rate\x18\x03 \x01(\x02H\x02\x88\x01\x01\x12\x1e\n\x11\x65val_trigger_rate\x18\x04 \x01(\x02H\x03\x88\x01\x01\x12\x1a\n\rlearning_rate\x18\x07 \x01(\x02H\x04\x88\x01\x01\x12\x36\n\x0bincoming_lr\x18\x08 \x03(\x0b\x32!.NeuronStatistics.IncomingLrEntry\x1a\x31\n\x0fIncomingLrEntry\x12\x0b\n\x03key\x18\x01 \x01(\x05\x12\r\n\x05value\x18\x02 \x01(\x02:\x02\x38\x01\x42\x0c\n\n_neuron_idB\r\n\x0b_neuron_ageB\x15\n\x13_train_trigger_rateB\x14\n\x12_eval_trigger_rateB\x10\n\x0e_learning_rate\"\xf0\x02\n\x13LayerRepresentation\x12\x15\n\x08layer_id\x18\x01 \x01(\x05H\x00\x88\x01\x01\x12\x17\n\nlayer_name\x18\x02 \x01(\tH\x01\x88\x01\x01\x12\x17\n\nlayer_type\x18\x03 \x01(\tH\x02\x88\x01\x01\x12\x1a\n\rneurons_count\x18\x04 \x01(\x05H\x03\x88\x01\x01\x12#\n\x16incoming_neurons_count\x18\x05 \x01(\x05H\x04\x88\x01\x01\x12\x18\n\x0bkernel_size\x18\x06 \x01(\x05H\x05\x88\x01\x01\x12\x13\n\x06stride\x18\x07 \x01(\x05H\x06\x88\x01\x01\x12-\n\x12neurons_statistics\x18\n \x03(\x0b\x32\x11.NeuronStatisticsB\x0b\n\t_layer_idB\r\n\x0b_layer_nameB\r\n\x0b_layer_typeB\x10\n\x0e_neurons_countB\x19\n\x17_incoming_neurons_countB\x0e\n\x0c_kernel_sizeB\t\n\x07_stride\"H\n\x11\x41\x63tivationRequest\x12\x10\n\x08layer_id\x18\x01 \x01(\x05\x12\x11\n\tsample_id\x18\x02 \x01(\t\x12\x0e\n\x06origin\x18\x03 \x01(\t\"H\n\rActivationMap\x12\x11\n\tneuron_id\x18\x01 \x01(\x05\x12\x0e\n\x06values\x18\x02 \x03(\x02\x12\t\n\x01H\x18\x03 \x01(\x05\x12\t\n\x01W\x18\x04 \x01(\x05\"d\n\x12\x41\x63tivationResponse\x12\x12\n\nlayer_type\x18\x01 \x01(\t\x12\x15\n\rneurons_count\x18\x02 \x01(\x05\x12#\n\x0b\x61\x63tivations\x18\x03 \x03(\x0b\x32\x0e.ActivationMap\"\x93\x01\n\tTaskField\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x15\n\x0b\x66loat_value\x18\x02 \x01(\x02H\x00\x12\x13\n\tint_value\x18\x03 \x01(\x05H\x00\x12\x16\n\x0cstring_value\x18\x04 \x01(\tH\x00\x12\x15\n\x0b\x62ytes_value\x18\x05 \x01(\x0cH\x00\x12\x14\n\nbool_value\x18\x06 \x01(\x08H\x00\x42\x07\n\x05value\"\x87\x03\n\x0eRecordMetadata\x12\x11\n\tsample_id\x18\x01 \x01(\t\x12\x14\n\x0csample_label\x18\x02 \x03(\x05\x12\x19\n\x11sample_prediction\x18\x03 \x03(\x05\x12=\n\x10sample_last_loss\x18\x04 \x03(\x0b\x32#.RecordMetadata.SampleLastLossEntry\x12\x19\n\x11sample_encounters\x18\x05 \x01(\x05\x12\x18\n\x10sample_discarded\x18\x06 \x01(\x08\x12 \n\x0c\x65xtra_fields\x18\x07 \x03(\x0b\x32\n.TaskField\x12\x16\n\x0eprediction_raw\x18\t \x01(\x0c\x12\x11\n\ttask_type\x18\n \x01(\t\x12\x19\n\x11sample_label_text\x18\x0b \x03(\t\x12\x1e\n\x16sample_prediction_text\x18\x0c \x03(\t\x1a\x35\n\x13SampleLastLossEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\x02:\x02\x38\x01\"\x93\x01\n\x10SampleStatistics\x12\x13\n\x06origin\x18\x06 \x01(\tH\x00\x88\x01\x01\x12\x19\n\x0csample_count\x18\x07 \x01(\x05H\x01\x88\x01\x01\x12\x11\n\ttask_type\x18\t \x01(\t\x12 \n\x07records\x18\x08 \x03(\x0b\x32\x0f.RecordMetadataB\t\n\x07_originB\x0f\n\r_sample_count\"\xe6\x01\n\x0f\x43ommandResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x33\n\x16hyper_parameters_descs\x18\x03 \x03(\x0b\x32\x13.HyperParameterDesc\x12\x33\n\x15layer_representations\x18\x04 \x03(\x0b\x32\x14.LayerRepresentation\x12\x31\n\x11sample_statistics\x18\x05 \x01(\x0b\x32\x11.SampleStatisticsH\x00\x88\x01\x01\x42\x14\n\x12_sample_statistics\"U\n\rSampleRequest\x12\x16\n\tsample_id\x18\x01 \x01(\tH\x00\x88\x01\x01\x12\x13\n\x06origin\x18\x02 \x01(\tH\x01\x88\x01\x01\x42\x0c\n\n_sample_idB\t\n\x07_origin\"\xad\x02\n\x15SampleRequestResponse\x12\x16\n\tsample_id\x18\x01 \x01(\tH\x00\x88\x01\x01\x12\x13\n\x06origin\x18\x02 \x01(\tH\x01\x88\x01\x01\x12\x12\n\x05label\x18\x03 \x01(\x05H\x02\x88\x01\x01\x12\x11\n\x04\x64\x61ta\x18\x04 \x01(\x0cH\x03\x88\x01\x01\x12\x1a\n\rerror_message\x18\x05 \x01(\tH\x04\x88\x01\x01\x12\x15\n\x08raw_data\x18\x06 \x01(\x0cH\x05\x88\x01\x01\x12\x11\n\x04mask\x18\x07 \x01(\x0cH\x06\x88\x01\x01\x12\x17\n\nprediction\x18\x08 \x01(\x0cH\x07\x88\x01\x01\x42\x0c\n\n_sample_idB\t\n\x07_originB\x08\n\x06_labelB\x07\n\x05_dataB\x10\n\x0e_error_messageB\x0b\n\t_raw_dataB\x07\n\x05_maskB\r\n\x0b_prediction\"\x92\x01\n\x12\x42\x61tchSampleRequest\x12\x12\n\nsample_ids\x18\x01 \x03(\t\x12\x0e\n\x06origin\x18\x02 \x01(\t\x12\x19\n\x0cresize_width\x18\x03 \x01(\x05H\x00\x88\x01\x01\x12\x1a\n\rresize_height\x18\x04 \x01(\x05H\x01\x88\x01\x01\x42\x0f\n\r_resize_widthB\x10\n\x0e_resize_height\">\n\x13\x42\x61tchSampleResponse\x12\'\n\x07samples\x18\x01 \x03(\x0b\x32\x16.SampleRequestResponse\".\n\x0eWeightsRequest\x12\x1c\n\tneuron_id\x18\x01 \x01(\x0b\x32\t.NeuronId\"\x9d\x02\n\x0fWeightsResponse\x12\x1c\n\tneuron_id\x18\x01 \x01(\x0b\x32\t.NeuronId\x12\x17\n\nlayer_name\x18\x02 \x01(\tH\x00\x88\x01\x01\x12\x17\n\nlayer_type\x18\x03 \x01(\tH\x01\x88\x01\x01\x12\x10\n\x08incoming\x18\x04 \x01(\x05\x12\x10\n\x08outgoing\x18\x05 \x01(\x05\x12\x18\n\x0bkernel_size\x18\x06 \x01(\x05H\x02\x88\x01\x01\x12\x0f\n\x07weights\x18\x07 \x03(\x02\x12\x0f\n\x07success\x18\x0b \x01(\x08\x12\x1a\n\rerror_message\x18\x0c \x01(\tH\x03\x88\x01\x01\x42\r\n\x0b_layer_nameB\r\n\x0b_layer_typeB\x0e\n\x0c_kernel_sizeB\x10\n\x0e_error_message\"R\n\x10\x44\x61taQueryRequest\x12\r\n\x05query\x18\x01 \x01(\t\x12\x12\n\naccumulate\x18\x02 \x01(\x08\x12\x1b\n\x13is_natural_language\x18\x03 \x01(\x08\"5\n\x11\x43\x61tegoricalTagDef\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x12\n\ncategories\x18\x02 \x03(\t\"\xa9\x02\n\x11\x44\x61taQueryResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x1d\n\x15number_of_all_samples\x18\x03 \x01(\x05\x12%\n\x1dnumber_of_samples_in_the_loop\x18\x04 \x01(\x05\x12#\n\x1bnumber_of_discarded_samples\x18\x05 \x01(\x05\x12\x13\n\x0bunique_tags\x18\x06 \x03(\t\x12+\n\x11\x61gent_intent_type\x18\x07 \x01(\x0e\x32\x10.AgentIntentType\x12\x17\n\x0f\x61nalysis_result\x18\x08 \x01(\t\x12,\n\x10\x63\x61tegorical_tags\x18\t \x03(\x0b\x32\x12.CategoricalTagDef\"\xc2\x01\n\x12\x44\x61taSamplesRequest\x12\x13\n\x0bstart_index\x18\x01 \x01(\x05\x12\x13\n\x0brecords_cnt\x18\x02 \x01(\x05\x12 \n\x18include_transformed_data\x18\x03 \x01(\x08\x12\x18\n\x10include_raw_data\x18\x04 \x01(\x08\x12\x19\n\x11stats_to_retrieve\x18\x05 \x03(\t\x12\x14\n\x0cresize_width\x18\x06 \x01(\x05\x12\x15\n\rresize_height\x18\x07 \x01(\x05\"m\n\x08\x44\x61taStat\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0c\n\x04type\x18\x02 \x01(\t\x12\r\n\x05shape\x18\x03 \x03(\x05\x12\r\n\x05value\x18\x04 \x03(\x02\x12\x14\n\x0cvalue_string\x18\x05 \x01(\t\x12\x11\n\tthumbnail\x18\x06 \x01(\x0c\">\n\nDataRecord\x12\x11\n\tsample_id\x18\x01 \x01(\t\x12\x1d\n\ndata_stats\x18\x02 \x03(\x0b\x32\t.DataStat\"Z\n\x13\x44\x61taSamplesResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12!\n\x0c\x64\x61ta_records\x18\x03 \x03(\x0b\x32\x0b.DataRecord\"C\n\x0fHistogramSubBar\x12\x0e\n\x06origin\x18\x01 \x01(\t\x12\x11\n\tdiscarded\x18\x02 \x01(\x08\x12\r\n\x05\x63ount\x18\x03 \x01(\x03\"h\n\x0cHistogramBin\x12\x0b\n\x03min\x18\x01 \x01(\x01\x12\x0b\n\x03max\x18\x02 \x01(\x01\x12\x0b\n\x03\x61vg\x18\x03 \x01(\x01\x12\r\n\x05\x63ount\x18\x04 \x01(\x03\x12\"\n\x08sub_bars\x18\x05 \x03(\x0b\x32\x10.HistogramSubBar\"[\n\x17\x43\x61tegoricalHistogramBar\x12\r\n\x05label\x18\x01 \x01(\t\x12\r\n\x05\x63ount\x18\x02 \x01(\x03\x12\"\n\x08sub_bars\x18\x03 \x03(\x0b\x32\x10.HistogramSubBar\"4\n\x10HistogramRequest\x12\x0e\n\x06\x63olumn\x18\x01 \x01(\t\x12\x10\n\x08max_bins\x18\x02 \x01(\x05\"\xb2\x01\n\x11HistogramResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x12\n\ntotal_rows\x18\x03 \x01(\x03\x12\x1b\n\x04\x62ins\x18\x04 \x03(\x0b\x32\r.HistogramBin\x12\x16\n\x0eis_categorical\x18\x05 \x01(\x08\x12\x32\n\x10\x63\x61tegorical_bars\x18\x06 \x03(\x0b\x32\x18.CategoricalHistogramBar\"W\n\x12GetMetaDataRequest\x12\x13\n\x0bstart_index\x18\x01 \x01(\x05\x12\x13\n\x0brecords_cnt\x18\x02 \x01(\x05\x12\x17\n\x0fmodal_sample_id\x18\x03 \x01(\t\"\x99\x01\n\x13GetMetaDataResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x1a\n\x12\x61ll_metadata_names\x18\x03 \x03(\t\x12!\n\x0cgrid_records\x18\x04 \x03(\x0b\x32\x0b.DataRecord\x12!\n\x0cmodal_record\x18\x05 \x01(\x0b\x32\x0b.DataRecord\"Y\n\x1aGetSignalTrajectoryRequest\x12\x13\n\x0bsignal_name\x18\x01 \x01(\t\x12\x12\n\nsample_ids\x18\x02 \x03(\t\x12\x12\n\nmax_points\x18\x03 \x01(\x05\"4\n\x10SignalTrajectory\x12\x11\n\tsample_id\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x03(\x02\"}\n\x1bGetSignalTrajectoryResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x13\n\x0bsignal_name\x18\x03 \x01(\t\x12\'\n\x0ctrajectories\x18\x04 \x03(\x0b\x32\x11.SignalTrajectory\"J\n\x11PointCloudRequest\x12\x11\n\tsample_id\x18\x01 \x01(\t\x12\x0e\n\x06origin\x18\x02 \x01(\t\x12\x12\n\nmax_points\x18\x03 \x01(\x05\"\xbf\x01\n\x0fPointCloudChunk\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x12\n\nnum_points\x18\x03 \x01(\x05\x12\x14\n\x0cnum_features\x18\x04 \x01(\x05\x12\x10\n\x08pc_range\x18\x05 \x03(\x02\x12\x0c\n\x04\x64\x61ta\x18\x06 \x01(\x0c\x12\x13\n\x0b\x63hunk_index\x18\x07 \x01(\x05\x12\x14\n\x0ctotal_chunks\x18\x08 \x01(\x05\x12\x15\n\rfeature_names\x18\t \x03(\t\"\xdc\x01\n\x10\x44\x61taEditsRequest\x12\x11\n\tstat_name\x18\x01 \x01(\t\x12\x13\n\x0b\x66loat_value\x18\x02 \x01(\x02\x12\x14\n\x0cstring_value\x18\x03 \x01(\t\x12\x12\n\nbool_value\x18\x04 \x01(\x08\x12\x1d\n\x04type\x18\x05 \x01(\x0e\x32\x0f.SampleEditType\x12\x13\n\x0bsamples_ids\x18\x06 \x03(\t\x12\x16\n\x0esample_origins\x18\x07 \x03(\t\x12\x16\n\x0eis_categorical\x18\x08 \x01(\x08\x12\x12\n\ncategories\x18\t \x03(\t\"5\n\x11\x44\x61taEditsResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\":\n\x12\x44\x61taSplitsResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x13\n\x0bsplit_names\x18\x02 \x03(\t\"9\n\x13\x41gentHealthResponse\x12\x11\n\tavailable\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"^\n\x16InitializeAgentRequest\x12\x0f\n\x07\x61pi_key\x18\x01 \x01(\t\x12$\n\x08provider\x18\x02 \x01(\x0e\x32\x12.AgentProviderType\x12\r\n\x05model\x18\x03 \x01(\t\";\n\x17InitializeAgentResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"(\n\x17\x43hangeAgentModelRequest\x12\r\n\x05model\x18\x01 \x01(\t\"<\n\x18\x43hangeAgentModelResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"\x17\n\x15GetAgentModelsRequest\"J\n\x16GetAgentModelsResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0e\n\x06models\x18\x02 \x03(\t\x12\x0f\n\x07message\x18\x03 \x01(\t\"6\n\x12ResetAgentResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"3\n\x18RestoreCheckpointRequest\x12\x17\n\x0f\x65xperiment_hash\x18\x01 \x01(\t\"=\n\x19RestoreCheckpointResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"R\n\x18TriggerEvaluationRequest\x12\x12\n\nsplit_name\x18\x01 \x01(\t\x12\x0c\n\x04tags\x18\x02 \x03(\t\x12\x14\n\x0cuse_full_set\x18\x03 \x01(\x08\"=\n\x19TriggerEvaluationResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"\x1c\n\x1aGetEvaluationStatusRequest\"\x81\x01\n\x1bGetEvaluationStatusResponse\x12\x0e\n\x06status\x18\x01 \x01(\t\x12\x0f\n\x07\x63urrent\x18\x02 \x01(\x05\x12\r\n\x05total\x18\x03 \x01(\x05\x12\x0f\n\x07message\x18\x04 \x01(\t\x12\r\n\x05\x65rror\x18\x05 \x01(\t\x12\x12\n\nsplit_name\x18\x06 \x01(\t\")\n\x17\x43\x61ncelEvaluationRequest\x12\x0e\n\x06reason\x18\x01 \x01(\t\"<\n\x18\x43\x61ncelEvaluationResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"7\n\x16RunNotebookCellRequest\x12\x0c\n\x04\x63ode\x18\x01 \x01(\t\x12\x0f\n\x07\x63\x65ll_id\x18\x02 \x01(\t\"2\n\x10NotebookCellDone\x12\x12\n\nexec_count\x18\x01 \x01(\x05\x12\n\n\x02ok\x18\x02 \x01(\x08\"\x1e\n\x1cInterruptNotebookCellRequest\":\n\x1dInterruptNotebookCellResponse\x12\n\n\x02ok\x18\x01 \x01(\x08\x12\r\n\x05\x65rror\x18\x02 \x01(\t\"\xbd\x01\n\x11NotebookCellChunk\x12\x0f\n\x07\x63\x65ll_id\x18\x01 \x01(\t\x12\x10\n\x06stdout\x18\x02 \x01(\tH\x00\x12\x10\n\x06stderr\x18\x03 \x01(\tH\x00\x12\x15\n\x0bresult_text\x18\x04 \x01(\tH\x00\x12\x13\n\timage_png\x18\x05 \x01(\x0cH\x00\x12\x19\n\x0f\x65rror_traceback\x18\x06 \x01(\tH\x00\x12!\n\x04\x64one\x18\x07 \x01(\x0b\x32\x11.NotebookCellDoneH\x00\x42\t\n\x07payload\"S\n\x10NotebookResponse\x12\x12\n\nipynb_json\x18\x01 \x01(\t\x12\x0f\n\x07\x65xisted\x18\x02 \x01(\x08\x12\x0c\n\x04path\x18\x03 \x01(\t\x12\x0c\n\x04name\x18\x04 \x01(\t\"7\n\x13SaveNotebookRequest\x12\x12\n\nipynb_json\x18\x01 \x01(\t\x12\x0c\n\x04name\x18\x02 \x01(\t\"M\n\x14SaveNotebookResponse\x12\n\n\x02ok\x18\x01 \x01(\x08\x12\x0c\n\x04path\x18\x02 \x01(\t\x12\r\n\x05\x65rror\x18\x03 \x01(\t\x12\x0c\n\x04name\x18\x04 \x01(\t\"C\n\x1bGenerateNotebookCodeRequest\x12\x0e\n\x06prompt\x18\x01 \x01(\t\x12\x14\n\x0c\x63ontext_code\x18\x02 \x01(\t\"\\\n\x1cGenerateNotebookCodeResponse\x12\x0c\n\x04\x63ode\x18\x01 \x01(\t\x12\x13\n\x0b\x65xplanation\x18\x02 \x01(\t\x12\n\n\x02ok\x18\x03 \x01(\x08\x12\r\n\x05\x65rror\x18\x04 \x01(\t\"~\n\x18\x45xportAnnotationsRequest\x12\'\n\x06\x66ormat\x18\x01 \x01(\x0e\x32\x17.AnnotationExportFormat\x12\x0e\n\x06origin\x18\x02 \x01(\t\x12\x1b\n\x13include_predictions\x18\x03 \x01(\x08\x12\x0c\n\x04tags\x18\x04 \x03(\t\"\x88\x01\n\x19\x45xportAnnotationsResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x0f\n\x07payload\x18\x03 \x01(\x0c\x12\x10\n\x08\x66ilename\x18\x04 \x01(\t\x12\x11\n\tmime_type\x18\x05 \x01(\t\x12\x13\n\x0bimage_count\x18\x06 \x01(\x05*d\n\x13WeightOperationType\x12\n\n\x06ZEROFY\x10\x00\x12\x10\n\x0cREINITIALIZE\x10\x01\x12\n\n\x06\x46REEZE\x10\x02\x12\x12\n\x0eREMOVE_NEURONS\x10\t\x12\x0f\n\x0b\x41\x44\x44_NEURONS\x10\n*o\n\x0fZerofyPredicate\x12\x19\n\x15ZEROFY_PREDICATE_NONE\x10\x00\x12 \n\x1cZEROFY_PREDICATE_WITH_FROZEN\x10\x01\x12\x1f\n\x1bZEROFY_PREDICATE_WITH_OLDER\x10\x02*M\n\x0f\x41gentIntentType\x12\x12\n\x0eINTENT_UNKNOWN\x10\x00\x12\x11\n\rINTENT_FILTER\x10\x01\x12\x13\n\x0fINTENT_ANALYSIS\x10\x02*I\n\x0eSampleEditType\x12\x11\n\rEDIT_OVERRIDE\x10\x00\x12\x13\n\x0f\x45\x44IT_ACCUMULATE\x10\x01\x12\x0f\n\x0b\x45\x44IT_REMOVE\x10\x02*,\n\x11\x41gentProviderType\x12\x17\n\x13PROVIDER_OPENROUTER\x10\x00*m\n\x16\x41nnotationExportFormat\x12\x16\n\x12\x45XPORT_FORMAT_CVAT\x10\x00\x12\x1e\n\x1a\x45XPORT_FORMAT_LABEL_STUDIO\x10\x01\x12\x1b\n\x17\x45XPORT_FORMAT_V7_DARWIN\x10\x02\x32\xe9\x0e\n\x11\x45xperimentService\x12P\n\x13GetLatestLoggerData\x12\x1b.GetLatestLoggerDataRequest\x1a\x1c.GetLatestLoggerDataResponse\x12\x36\n\x11\x45xperimentCommand\x12\x0f.TrainerCommand\x1a\x10.CommandResponse\x12H\n\x11ManipulateWeights\x12\x18.WeightsOperationRequest\x1a\x19.WeightsOperationResponse\x12/\n\nGetWeights\x12\x0f.WeightsRequest\x1a\x10.WeightsResponse\x12\x39\n\x0eGetActivations\x12\x12.ActivationRequest\x1a\x13.ActivationResponse\x12\x37\n\nGetSamples\x12\x13.BatchSampleRequest\x1a\x14.BatchSampleResponse\x12\x37\n\x0e\x41pplyDataQuery\x12\x11.DataQueryRequest\x1a\x12.DataQueryResponse\x12;\n\x0eGetDataSamples\x12\x13.DataSamplesRequest\x1a\x14.DataSamplesResponse\x12\x35\n\x0cGetHistogram\x12\x11.HistogramRequest\x1a\x12.HistogramResponse\x12\x38\n\x0bGetMetaData\x12\x13.GetMetaDataRequest\x1a\x14.GetMetaDataResponse\x12P\n\x13GetSignalTrajectory\x12\x1b.GetSignalTrajectoryRequest\x1a\x1c.GetSignalTrajectoryResponse\x12\x37\n\rGetPointCloud\x12\x12.PointCloudRequest\x1a\x10.PointCloudChunk0\x01\x12\x37\n\x0e\x45\x64itDataSample\x12\x11.DataEditsRequest\x1a\x12.DataEditsResponse\x12,\n\rGetDataSplits\x12\x06.Empty\x1a\x13.DataSplitsResponse\x12\x30\n\x10\x43heckAgentHealth\x12\x06.Empty\x1a\x14.AgentHealthResponse\x12\x44\n\x0fInitializeAgent\x12\x17.InitializeAgentRequest\x1a\x18.InitializeAgentResponse\x12G\n\x10\x43hangeAgentModel\x12\x18.ChangeAgentModelRequest\x1a\x19.ChangeAgentModelResponse\x12\x41\n\x0eGetAgentModels\x12\x16.GetAgentModelsRequest\x1a\x17.GetAgentModelsResponse\x12)\n\nResetAgent\x12\x06.Empty\x1a\x13.ResetAgentResponse\x12@\n\x0fRunNotebookCell\x12\x17.RunNotebookCellRequest\x1a\x12.NotebookCellChunk0\x01\x12V\n\x15InterruptNotebookCell\x12\x1d.InterruptNotebookCellRequest\x1a\x1e.InterruptNotebookCellResponse\x12(\n\x0bGetNotebook\x12\x06.Empty\x1a\x11.NotebookResponse\x12;\n\x0cSaveNotebook\x12\x14.SaveNotebookRequest\x1a\x15.SaveNotebookResponse\x12S\n\x14GenerateNotebookCode\x12\x1c.GenerateNotebookCodeRequest\x1a\x1d.GenerateNotebookCodeResponse\x12J\n\x11RestoreCheckpoint\x12\x19.RestoreCheckpointRequest\x1a\x1a.RestoreCheckpointResponse\x12J\n\x11TriggerEvaluation\x12\x19.TriggerEvaluationRequest\x1a\x1a.TriggerEvaluationResponse\x12P\n\x13GetEvaluationStatus\x12\x1b.GetEvaluationStatusRequest\x1a\x1c.GetEvaluationStatusResponse\x12G\n\x10\x43\x61ncelEvaluation\x12\x18.CancelEvaluationRequest\x1a\x19.CancelEvaluationResponse\x12J\n\x11\x45xportAnnotations\x12\x19.ExportAnnotationsRequest\x1a\x1a.ExportAnnotationsResponseb\x06proto3') +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n)weightslab/proto/experiment_service.proto\"\x89\x01\n\x1aGetLatestLoggerDataRequest\x12\x1c\n\x14request_full_history\x18\x01 \x01(\x08\x12\x12\n\nmax_points\x18\x02 \x01(\x05\x12\x17\n\x0f\x62reak_by_slices\x18\x03 \x01(\x08\x12\x0c\n\x04tags\x18\x04 \x03(\t\x12\x12\n\ngraph_name\x18\x05 \x01(\t\"1\n\rSignalOutlier\x12\x11\n\tsample_id\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\x02\"\xd2\x03\n\x0fLoggerDataPoint\x12\x13\n\x0bmetric_name\x18\x01 \x01(\t\x12\x11\n\tmodel_age\x18\x02 \x01(\x05\x12\x14\n\x0cmetric_value\x18\x03 \x01(\x02\x12\x17\n\x0f\x65xperiment_hash\x18\x04 \x01(\t\x12\x11\n\ttimestamp\x18\x05 \x01(\x03\x12\x11\n\tsample_id\x18\x06 \x01(\t\x12\x1c\n\x14is_evaluation_marker\x18\x07 \x01(\x08\x12\x12\n\nsplit_name\x18\x08 \x01(\t\x12\x17\n\x0f\x65valuation_tags\x18\t \x03(\t\x12\x12\n\npoint_note\x18\n \x01(\t\x12\x12\n\naudit_mode\x18\x0b \x01(\x08\x12 \n\x08outliers\x18\x0c \x03(\x0b\x32\x0e.SignalOutlier\x12\x15\n\routlier_count\x18\r \x01(\x05\x12\x14\n\x0csample_count\x18\x0e \x01(\x05\x12\x13\n\x0btrend_value\x18\x0f \x01(\x02\x12\x14\n\x0ctrend_margin\x18\x10 \x01(\x02\x12\x16\n\x0ehas_trend_band\x18\x11 \x01(\x08\x12\x11\n\tvalue_min\x18\x12 \x01(\x02\x12\x11\n\tvalue_max\x18\x13 \x01(\x02\x12\x17\n\x0fhas_value_range\x18\x14 \x01(\x08\"[\n\x1bGetLatestLoggerDataResponse\x12 \n\x06points\x18\x01 \x03(\x0b\x32\x10.LoggerDataPoint\x12\x1a\n\x12weightslab_version\x18\x02 \x01(\t\"\x07\n\x05\x45mpty\"/\n\x08NeuronId\x12\x10\n\x08layer_id\x18\x01 \x01(\x05\x12\x11\n\tneuron_id\x18\x02 \x01(\x05\"\x91\x02\n\x0fWeightOperation\x12*\n\x07op_type\x18\x01 \x01(\x0e\x32\x14.WeightOperationTypeH\x00\x88\x01\x01\x12\x15\n\x08layer_id\x18\x02 \x01(\x05H\x01\x88\x01\x01\x12\x1d\n\nneuron_ids\x18\x03 \x03(\x0b\x32\t.NeuronId\x12\x16\n\x0eneurons_to_add\x18\t \x01(\x05\x12 \n\x18zerofy_from_incoming_ids\x18\x0b \x03(\x05\x12\x1c\n\x14zerofy_to_neuron_ids\x18\x0c \x03(\x05\x12+\n\x11zerofy_predicates\x18\r \x03(\x0e\x32\x10.ZerofyPredicateB\n\n\x08_op_typeB\x0b\n\t_layer_id\"_\n\x17WeightsOperationRequest\x12/\n\x10weight_operation\x18\x01 \x01(\x0b\x32\x10.WeightOperationH\x00\x88\x01\x01\x42\x13\n\x11_weight_operation\"<\n\x18WeightsOperationResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"\xc1\x05\n\x0fHyperParameters\x12\x1c\n\x0f\x65xperiment_name\x18\x01 \x01(\tH\x00\x88\x01\x01\x12!\n\x14training_steps_to_do\x18\x02 \x01(\x05H\x01\x88\x01\x01\x12\x1a\n\rlearning_rate\x18\x03 \x01(\x02H\x02\x88\x01\x01\x12\x17\n\nbatch_size\x18\x04 \x01(\x05H\x03\x88\x01\x01\x12 \n\x13\x66ull_eval_frequency\x18\x05 \x01(\x05H\x04\x88\x01\x01\x12 \n\x13\x63heckpont_frequency\x18\x06 \x01(\x05H\x05\x88\x01\x01\x12\x18\n\x0bis_training\x18\x07 \x01(\x08H\x06\x88\x01\x01\x12\x15\n\x08nb_steps\x18\x08 \x01(\x05H\x07\x88\x01\x01\x12\x19\n\x0c\x61uditor_mode\x18\t \x01(\x08H\x08\x88\x01\x01\x12\x1d\n\x10train_batch_size\x18\n \x01(\x05H\t\x88\x01\x01\x12\x1b\n\x0eval_batch_size\x18\x0b \x01(\x05H\n\x88\x01\x01\x12\x1c\n\x0ftest_batch_size\x18\x0c \x01(\x05H\x0b\x88\x01\x01\x12\x1c\n\x0f\x65valuation_mode\x18\r \x01(\x08H\x0c\x88\x01\x01\x12\x1e\n\x11\x65valuation_config\x18\x0e \x01(\tH\r\x88\x01\x01\x42\x12\n\x10_experiment_nameB\x17\n\x15_training_steps_to_doB\x10\n\x0e_learning_rateB\r\n\x0b_batch_sizeB\x16\n\x14_full_eval_frequencyB\x16\n\x14_checkpont_frequencyB\x0e\n\x0c_is_trainingB\x0b\n\t_nb_stepsB\x0f\n\r_auditor_modeB\x13\n\x11_train_batch_sizeB\x11\n\x0f_val_batch_sizeB\x12\n\x10_test_batch_sizeB\x12\n\x10_evaluation_modeB\x14\n\x12_evaluation_config\",\n\rMetricsStatus\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\x02\"~\n\rAnnotatStatus\x12\x0c\n\x04name\x18\x01 \x01(\t\x12.\n\x08metadata\x18\x02 \x03(\x0b\x32\x1c.AnnotatStatus.MetadataEntry\x1a/\n\rMetadataEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\x02:\x02\x38\x01\"\x90\x02\n\x10TrainingStatusEx\x12\x16\n\ttimestamp\x18\x01 \x01(\tH\x00\x88\x01\x01\x12\x1c\n\x0f\x65xperiment_name\x18\x02 \x01(\tH\x01\x88\x01\x01\x12\x16\n\tmodel_age\x18\x03 \x01(\x05H\x02\x88\x01\x01\x12+\n\x0emetrics_status\x18\x04 \x01(\x0b\x32\x0e.MetricsStatusH\x03\x88\x01\x01\x12+\n\x0e\x61nnotat_status\x18\x05 \x01(\x0b\x32\x0e.AnnotatStatusH\x04\x88\x01\x01\x42\x0c\n\n_timestampB\x12\n\x10_experiment_nameB\x0c\n\n_model_ageB\x11\n\x0f_metrics_statusB\x11\n\x0f_annotat_status\"]\n\x15HyperParameterCommand\x12/\n\x10hyper_parameters\x18\x01 \x01(\x0b\x32\x10.HyperParametersH\x00\x88\x01\x01\x42\x13\n\x11_hyper_parameters\">\n\x14\x44\x65nySamplesOperation\x12\x12\n\nsample_ids\x18\x01 \x03(\t\x12\x12\n\naccumulate\x18\x02 \x01(\x08\"0\n\x17LoadCheckpointOperation\x12\x15\n\rcheckpoint_id\x18\x01 \x01(\x05\"b\n\x11PlotNoteOperation\x12\x13\n\x0bmetric_name\x18\x01 \x01(\t\x12\x17\n\x0f\x65xperiment_hash\x18\x02 \x01(\t\x12\x11\n\tmodel_age\x18\x03 \x01(\x05\x12\x0c\n\x04note\x18\x04 \x01(\t\"L\n\x17SaveCheckpointOperation\x12\x19\n\x11save_architecture\x18\x01 \x01(\x08\x12\x16\n\x0esave_optimizer\x18\x02 \x01(\x08\"\x1a\n\x18RestartInstanceOperation\"\x8d\x08\n\x0eTrainerCommand\x12\x1c\n\x14get_hyper_parameters\x18\x04 \x01(\x08\x12\x1e\n\x16get_interactive_layers\x18\x05 \x01(\x08\x12\x1d\n\x10get_data_records\x18\x06 \x01(\tH\x00\x88\x01\x01\x12%\n\x18get_single_layer_info_id\x18\x08 \x01(\x05H\x01\x88\x01\x01\x12;\n\x16hyper_parameter_change\x18\x01 \x01(\x0b\x32\x16.HyperParameterCommandH\x02\x88\x01\x01\x12:\n\x16\x64\x65ny_samples_operation\x18\x07 \x01(\x0b\x32\x15.DenySamplesOperationH\x03\x88\x01\x01\x12?\n\x1b\x64\x65ny_eval_samples_operation\x18\n \x01(\x0b\x32\x15.DenySamplesOperationH\x04\x88\x01\x01\x12@\n\x19load_checkpoint_operation\x18\t \x01(\x0b\x32\x18.LoadCheckpointOperationH\x05\x88\x01\x01\x12\x42\n\x1eremove_from_denylist_operation\x18\x0b \x01(\x0b\x32\x15.DenySamplesOperationH\x06\x88\x01\x01\x12G\n#remove_eval_from_denylist_operation\x18\x0c \x01(\x0b\x32\x15.DenySamplesOperationH\x07\x88\x01\x01\x12\x34\n\x13plot_note_operation\x18\r \x01(\x0b\x32\x12.PlotNoteOperationH\x08\x88\x01\x01\x12@\n\x19save_checkpoint_operation\x18\x0e \x01(\x0b\x32\x18.SaveCheckpointOperationH\t\x88\x01\x01\x12\x39\n\x11restart_operation\x18\x0f \x01(\x0b\x32\x19.RestartInstanceOperationH\n\x88\x01\x01\x42\x13\n\x11_get_data_recordsB\x1b\n\x19_get_single_layer_info_idB\x19\n\x17_hyper_parameter_changeB\x19\n\x17_deny_samples_operationB\x1e\n\x1c_deny_eval_samples_operationB\x1c\n\x1a_load_checkpoint_operationB!\n\x1f_remove_from_denylist_operationB&\n$_remove_eval_from_denylist_operationB\x16\n\x14_plot_note_operationB\x1c\n\x1a_save_checkpoint_operationB\x14\n\x12_restart_operation\"\x9d\x01\n\x12HyperParameterDesc\x12\r\n\x05label\x18\x01 \x01(\t\x12\x0c\n\x04name\x18\x02 \x01(\t\x12\x0c\n\x04type\x18\x03 \x01(\t\x12\x1c\n\x0fnumerical_value\x18\x04 \x01(\x02H\x00\x88\x01\x01\x12\x19\n\x0cstring_value\x18\x05 \x01(\tH\x01\x88\x01\x01\x42\x12\n\x10_numerical_valueB\x0f\n\r_string_value\"\xf2\x02\n\x10NeuronStatistics\x12!\n\tneuron_id\x18\x01 \x01(\x0b\x32\t.NeuronIdH\x00\x88\x01\x01\x12\x17\n\nneuron_age\x18\x02 \x01(\x05H\x01\x88\x01\x01\x12\x1f\n\x12train_trigger_rate\x18\x03 \x01(\x02H\x02\x88\x01\x01\x12\x1e\n\x11\x65val_trigger_rate\x18\x04 \x01(\x02H\x03\x88\x01\x01\x12\x1a\n\rlearning_rate\x18\x07 \x01(\x02H\x04\x88\x01\x01\x12\x36\n\x0bincoming_lr\x18\x08 \x03(\x0b\x32!.NeuronStatistics.IncomingLrEntry\x1a\x31\n\x0fIncomingLrEntry\x12\x0b\n\x03key\x18\x01 \x01(\x05\x12\r\n\x05value\x18\x02 \x01(\x02:\x02\x38\x01\x42\x0c\n\n_neuron_idB\r\n\x0b_neuron_ageB\x15\n\x13_train_trigger_rateB\x14\n\x12_eval_trigger_rateB\x10\n\x0e_learning_rate\"\xf0\x02\n\x13LayerRepresentation\x12\x15\n\x08layer_id\x18\x01 \x01(\x05H\x00\x88\x01\x01\x12\x17\n\nlayer_name\x18\x02 \x01(\tH\x01\x88\x01\x01\x12\x17\n\nlayer_type\x18\x03 \x01(\tH\x02\x88\x01\x01\x12\x1a\n\rneurons_count\x18\x04 \x01(\x05H\x03\x88\x01\x01\x12#\n\x16incoming_neurons_count\x18\x05 \x01(\x05H\x04\x88\x01\x01\x12\x18\n\x0bkernel_size\x18\x06 \x01(\x05H\x05\x88\x01\x01\x12\x13\n\x06stride\x18\x07 \x01(\x05H\x06\x88\x01\x01\x12-\n\x12neurons_statistics\x18\n \x03(\x0b\x32\x11.NeuronStatisticsB\x0b\n\t_layer_idB\r\n\x0b_layer_nameB\r\n\x0b_layer_typeB\x10\n\x0e_neurons_countB\x19\n\x17_incoming_neurons_countB\x0e\n\x0c_kernel_sizeB\t\n\x07_stride\"H\n\x11\x41\x63tivationRequest\x12\x10\n\x08layer_id\x18\x01 \x01(\x05\x12\x11\n\tsample_id\x18\x02 \x01(\t\x12\x0e\n\x06origin\x18\x03 \x01(\t\"H\n\rActivationMap\x12\x11\n\tneuron_id\x18\x01 \x01(\x05\x12\x0e\n\x06values\x18\x02 \x03(\x02\x12\t\n\x01H\x18\x03 \x01(\x05\x12\t\n\x01W\x18\x04 \x01(\x05\"d\n\x12\x41\x63tivationResponse\x12\x12\n\nlayer_type\x18\x01 \x01(\t\x12\x15\n\rneurons_count\x18\x02 \x01(\x05\x12#\n\x0b\x61\x63tivations\x18\x03 \x03(\x0b\x32\x0e.ActivationMap\"\x93\x01\n\tTaskField\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x15\n\x0b\x66loat_value\x18\x02 \x01(\x02H\x00\x12\x13\n\tint_value\x18\x03 \x01(\x05H\x00\x12\x16\n\x0cstring_value\x18\x04 \x01(\tH\x00\x12\x15\n\x0b\x62ytes_value\x18\x05 \x01(\x0cH\x00\x12\x14\n\nbool_value\x18\x06 \x01(\x08H\x00\x42\x07\n\x05value\"\x87\x03\n\x0eRecordMetadata\x12\x11\n\tsample_id\x18\x01 \x01(\t\x12\x14\n\x0csample_label\x18\x02 \x03(\x05\x12\x19\n\x11sample_prediction\x18\x03 \x03(\x05\x12=\n\x10sample_last_loss\x18\x04 \x03(\x0b\x32#.RecordMetadata.SampleLastLossEntry\x12\x19\n\x11sample_encounters\x18\x05 \x01(\x05\x12\x18\n\x10sample_discarded\x18\x06 \x01(\x08\x12 \n\x0c\x65xtra_fields\x18\x07 \x03(\x0b\x32\n.TaskField\x12\x16\n\x0eprediction_raw\x18\t \x01(\x0c\x12\x11\n\ttask_type\x18\n \x01(\t\x12\x19\n\x11sample_label_text\x18\x0b \x03(\t\x12\x1e\n\x16sample_prediction_text\x18\x0c \x03(\t\x1a\x35\n\x13SampleLastLossEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\x02:\x02\x38\x01\"\x93\x01\n\x10SampleStatistics\x12\x13\n\x06origin\x18\x06 \x01(\tH\x00\x88\x01\x01\x12\x19\n\x0csample_count\x18\x07 \x01(\x05H\x01\x88\x01\x01\x12\x11\n\ttask_type\x18\t \x01(\t\x12 \n\x07records\x18\x08 \x03(\x0b\x32\x0f.RecordMetadataB\t\n\x07_originB\x0f\n\r_sample_count\"\xe6\x01\n\x0f\x43ommandResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x33\n\x16hyper_parameters_descs\x18\x03 \x03(\x0b\x32\x13.HyperParameterDesc\x12\x33\n\x15layer_representations\x18\x04 \x03(\x0b\x32\x14.LayerRepresentation\x12\x31\n\x11sample_statistics\x18\x05 \x01(\x0b\x32\x11.SampleStatisticsH\x00\x88\x01\x01\x42\x14\n\x12_sample_statistics\"U\n\rSampleRequest\x12\x16\n\tsample_id\x18\x01 \x01(\tH\x00\x88\x01\x01\x12\x13\n\x06origin\x18\x02 \x01(\tH\x01\x88\x01\x01\x42\x0c\n\n_sample_idB\t\n\x07_origin\"\xad\x02\n\x15SampleRequestResponse\x12\x16\n\tsample_id\x18\x01 \x01(\tH\x00\x88\x01\x01\x12\x13\n\x06origin\x18\x02 \x01(\tH\x01\x88\x01\x01\x12\x12\n\x05label\x18\x03 \x01(\x05H\x02\x88\x01\x01\x12\x11\n\x04\x64\x61ta\x18\x04 \x01(\x0cH\x03\x88\x01\x01\x12\x1a\n\rerror_message\x18\x05 \x01(\tH\x04\x88\x01\x01\x12\x15\n\x08raw_data\x18\x06 \x01(\x0cH\x05\x88\x01\x01\x12\x11\n\x04mask\x18\x07 \x01(\x0cH\x06\x88\x01\x01\x12\x17\n\nprediction\x18\x08 \x01(\x0cH\x07\x88\x01\x01\x42\x0c\n\n_sample_idB\t\n\x07_originB\x08\n\x06_labelB\x07\n\x05_dataB\x10\n\x0e_error_messageB\x0b\n\t_raw_dataB\x07\n\x05_maskB\r\n\x0b_prediction\"\x92\x01\n\x12\x42\x61tchSampleRequest\x12\x12\n\nsample_ids\x18\x01 \x03(\t\x12\x0e\n\x06origin\x18\x02 \x01(\t\x12\x19\n\x0cresize_width\x18\x03 \x01(\x05H\x00\x88\x01\x01\x12\x1a\n\rresize_height\x18\x04 \x01(\x05H\x01\x88\x01\x01\x42\x0f\n\r_resize_widthB\x10\n\x0e_resize_height\">\n\x13\x42\x61tchSampleResponse\x12\'\n\x07samples\x18\x01 \x03(\x0b\x32\x16.SampleRequestResponse\".\n\x0eWeightsRequest\x12\x1c\n\tneuron_id\x18\x01 \x01(\x0b\x32\t.NeuronId\"\x9d\x02\n\x0fWeightsResponse\x12\x1c\n\tneuron_id\x18\x01 \x01(\x0b\x32\t.NeuronId\x12\x17\n\nlayer_name\x18\x02 \x01(\tH\x00\x88\x01\x01\x12\x17\n\nlayer_type\x18\x03 \x01(\tH\x01\x88\x01\x01\x12\x10\n\x08incoming\x18\x04 \x01(\x05\x12\x10\n\x08outgoing\x18\x05 \x01(\x05\x12\x18\n\x0bkernel_size\x18\x06 \x01(\x05H\x02\x88\x01\x01\x12\x0f\n\x07weights\x18\x07 \x03(\x02\x12\x0f\n\x07success\x18\x0b \x01(\x08\x12\x1a\n\rerror_message\x18\x0c \x01(\tH\x03\x88\x01\x01\x42\r\n\x0b_layer_nameB\r\n\x0b_layer_typeB\x0e\n\x0c_kernel_sizeB\x10\n\x0e_error_message\"R\n\x10\x44\x61taQueryRequest\x12\r\n\x05query\x18\x01 \x01(\t\x12\x12\n\naccumulate\x18\x02 \x01(\x08\x12\x1b\n\x13is_natural_language\x18\x03 \x01(\x08\"5\n\x11\x43\x61tegoricalTagDef\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x12\n\ncategories\x18\x02 \x03(\t\"\xa9\x02\n\x11\x44\x61taQueryResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x1d\n\x15number_of_all_samples\x18\x03 \x01(\x05\x12%\n\x1dnumber_of_samples_in_the_loop\x18\x04 \x01(\x05\x12#\n\x1bnumber_of_discarded_samples\x18\x05 \x01(\x05\x12\x13\n\x0bunique_tags\x18\x06 \x03(\t\x12+\n\x11\x61gent_intent_type\x18\x07 \x01(\x0e\x32\x10.AgentIntentType\x12\x17\n\x0f\x61nalysis_result\x18\x08 \x01(\t\x12,\n\x10\x63\x61tegorical_tags\x18\t \x03(\x0b\x32\x12.CategoricalTagDef\"\xc2\x01\n\x12\x44\x61taSamplesRequest\x12\x13\n\x0bstart_index\x18\x01 \x01(\x05\x12\x13\n\x0brecords_cnt\x18\x02 \x01(\x05\x12 \n\x18include_transformed_data\x18\x03 \x01(\x08\x12\x18\n\x10include_raw_data\x18\x04 \x01(\x08\x12\x19\n\x11stats_to_retrieve\x18\x05 \x03(\t\x12\x14\n\x0cresize_width\x18\x06 \x01(\x05\x12\x15\n\rresize_height\x18\x07 \x01(\x05\"m\n\x08\x44\x61taStat\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0c\n\x04type\x18\x02 \x01(\t\x12\r\n\x05shape\x18\x03 \x03(\x05\x12\r\n\x05value\x18\x04 \x03(\x02\x12\x14\n\x0cvalue_string\x18\x05 \x01(\t\x12\x11\n\tthumbnail\x18\x06 \x01(\x0c\">\n\nDataRecord\x12\x11\n\tsample_id\x18\x01 \x01(\t\x12\x1d\n\ndata_stats\x18\x02 \x03(\x0b\x32\t.DataStat\"Z\n\x13\x44\x61taSamplesResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12!\n\x0c\x64\x61ta_records\x18\x03 \x03(\x0b\x32\x0b.DataRecord\"C\n\x0fHistogramSubBar\x12\x0e\n\x06origin\x18\x01 \x01(\t\x12\x11\n\tdiscarded\x18\x02 \x01(\x08\x12\r\n\x05\x63ount\x18\x03 \x01(\x03\"h\n\x0cHistogramBin\x12\x0b\n\x03min\x18\x01 \x01(\x01\x12\x0b\n\x03max\x18\x02 \x01(\x01\x12\x0b\n\x03\x61vg\x18\x03 \x01(\x01\x12\r\n\x05\x63ount\x18\x04 \x01(\x03\x12\"\n\x08sub_bars\x18\x05 \x03(\x0b\x32\x10.HistogramSubBar\"[\n\x17\x43\x61tegoricalHistogramBar\x12\r\n\x05label\x18\x01 \x01(\t\x12\r\n\x05\x63ount\x18\x02 \x01(\x03\x12\"\n\x08sub_bars\x18\x03 \x03(\x0b\x32\x10.HistogramSubBar\"4\n\x10HistogramRequest\x12\x0e\n\x06\x63olumn\x18\x01 \x01(\t\x12\x10\n\x08max_bins\x18\x02 \x01(\x05\"\xb2\x01\n\x11HistogramResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x12\n\ntotal_rows\x18\x03 \x01(\x03\x12\x1b\n\x04\x62ins\x18\x04 \x03(\x0b\x32\r.HistogramBin\x12\x16\n\x0eis_categorical\x18\x05 \x01(\x08\x12\x32\n\x10\x63\x61tegorical_bars\x18\x06 \x03(\x0b\x32\x18.CategoricalHistogramBar\"W\n\x12GetMetaDataRequest\x12\x13\n\x0bstart_index\x18\x01 \x01(\x05\x12\x13\n\x0brecords_cnt\x18\x02 \x01(\x05\x12\x17\n\x0fmodal_sample_id\x18\x03 \x01(\t\"\x99\x01\n\x13GetMetaDataResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x1a\n\x12\x61ll_metadata_names\x18\x03 \x03(\t\x12!\n\x0cgrid_records\x18\x04 \x03(\x0b\x32\x0b.DataRecord\x12!\n\x0cmodal_record\x18\x05 \x01(\x0b\x32\x0b.DataRecord\"j\n\x12StepSamplesRequest\x12\x13\n\x0bmetric_name\x18\x01 \x01(\t\x12\x17\n\x0f\x65xperiment_hash\x18\x02 \x01(\t\x12\x11\n\tmodel_age\x18\x03 \x01(\x05\x12\x13\n\x0bmax_samples\x18\x04 \x01(\x05\"d\n\x13StepSamplesResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x12\n\nsample_ids\x18\x03 \x03(\t\x12\x17\n\x0ftotal_available\x18\x04 \x01(\x05\"Y\n\x1aGetSignalTrajectoryRequest\x12\x13\n\x0bsignal_name\x18\x01 \x01(\t\x12\x12\n\nsample_ids\x18\x02 \x03(\t\x12\x12\n\nmax_points\x18\x03 \x01(\x05\"4\n\x10SignalTrajectory\x12\x11\n\tsample_id\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x03(\x02\"}\n\x1bGetSignalTrajectoryResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x13\n\x0bsignal_name\x18\x03 \x01(\t\x12\'\n\x0ctrajectories\x18\x04 \x03(\x0b\x32\x11.SignalTrajectory\"J\n\x11PointCloudRequest\x12\x11\n\tsample_id\x18\x01 \x01(\t\x12\x0e\n\x06origin\x18\x02 \x01(\t\x12\x12\n\nmax_points\x18\x03 \x01(\x05\"\xbf\x01\n\x0fPointCloudChunk\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x12\n\nnum_points\x18\x03 \x01(\x05\x12\x14\n\x0cnum_features\x18\x04 \x01(\x05\x12\x10\n\x08pc_range\x18\x05 \x03(\x02\x12\x0c\n\x04\x64\x61ta\x18\x06 \x01(\x0c\x12\x13\n\x0b\x63hunk_index\x18\x07 \x01(\x05\x12\x14\n\x0ctotal_chunks\x18\x08 \x01(\x05\x12\x15\n\rfeature_names\x18\t \x03(\t\"\xdc\x01\n\x10\x44\x61taEditsRequest\x12\x11\n\tstat_name\x18\x01 \x01(\t\x12\x13\n\x0b\x66loat_value\x18\x02 \x01(\x02\x12\x14\n\x0cstring_value\x18\x03 \x01(\t\x12\x12\n\nbool_value\x18\x04 \x01(\x08\x12\x1d\n\x04type\x18\x05 \x01(\x0e\x32\x0f.SampleEditType\x12\x13\n\x0bsamples_ids\x18\x06 \x03(\t\x12\x16\n\x0esample_origins\x18\x07 \x03(\t\x12\x16\n\x0eis_categorical\x18\x08 \x01(\x08\x12\x12\n\ncategories\x18\t \x03(\t\"5\n\x11\x44\x61taEditsResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\":\n\x12\x44\x61taSplitsResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x13\n\x0bsplit_names\x18\x02 \x03(\t\"9\n\x13\x41gentHealthResponse\x12\x11\n\tavailable\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"^\n\x16InitializeAgentRequest\x12\x0f\n\x07\x61pi_key\x18\x01 \x01(\t\x12$\n\x08provider\x18\x02 \x01(\x0e\x32\x12.AgentProviderType\x12\r\n\x05model\x18\x03 \x01(\t\";\n\x17InitializeAgentResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"(\n\x17\x43hangeAgentModelRequest\x12\r\n\x05model\x18\x01 \x01(\t\"<\n\x18\x43hangeAgentModelResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"\x17\n\x15GetAgentModelsRequest\"J\n\x16GetAgentModelsResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0e\n\x06models\x18\x02 \x03(\t\x12\x0f\n\x07message\x18\x03 \x01(\t\"6\n\x12ResetAgentResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"3\n\x18RestoreCheckpointRequest\x12\x17\n\x0f\x65xperiment_hash\x18\x01 \x01(\t\"=\n\x19RestoreCheckpointResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"R\n\x18TriggerEvaluationRequest\x12\x12\n\nsplit_name\x18\x01 \x01(\t\x12\x0c\n\x04tags\x18\x02 \x03(\t\x12\x14\n\x0cuse_full_set\x18\x03 \x01(\x08\"=\n\x19TriggerEvaluationResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"\x1c\n\x1aGetEvaluationStatusRequest\"\x81\x01\n\x1bGetEvaluationStatusResponse\x12\x0e\n\x06status\x18\x01 \x01(\t\x12\x0f\n\x07\x63urrent\x18\x02 \x01(\x05\x12\r\n\x05total\x18\x03 \x01(\x05\x12\x0f\n\x07message\x18\x04 \x01(\t\x12\r\n\x05\x65rror\x18\x05 \x01(\t\x12\x12\n\nsplit_name\x18\x06 \x01(\t\")\n\x17\x43\x61ncelEvaluationRequest\x12\x0e\n\x06reason\x18\x01 \x01(\t\"<\n\x18\x43\x61ncelEvaluationResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"7\n\x16RunNotebookCellRequest\x12\x0c\n\x04\x63ode\x18\x01 \x01(\t\x12\x0f\n\x07\x63\x65ll_id\x18\x02 \x01(\t\"2\n\x10NotebookCellDone\x12\x12\n\nexec_count\x18\x01 \x01(\x05\x12\n\n\x02ok\x18\x02 \x01(\x08\"\x1e\n\x1cInterruptNotebookCellRequest\":\n\x1dInterruptNotebookCellResponse\x12\n\n\x02ok\x18\x01 \x01(\x08\x12\r\n\x05\x65rror\x18\x02 \x01(\t\"\xbd\x01\n\x11NotebookCellChunk\x12\x0f\n\x07\x63\x65ll_id\x18\x01 \x01(\t\x12\x10\n\x06stdout\x18\x02 \x01(\tH\x00\x12\x10\n\x06stderr\x18\x03 \x01(\tH\x00\x12\x15\n\x0bresult_text\x18\x04 \x01(\tH\x00\x12\x13\n\timage_png\x18\x05 \x01(\x0cH\x00\x12\x19\n\x0f\x65rror_traceback\x18\x06 \x01(\tH\x00\x12!\n\x04\x64one\x18\x07 \x01(\x0b\x32\x11.NotebookCellDoneH\x00\x42\t\n\x07payload\"S\n\x10NotebookResponse\x12\x12\n\nipynb_json\x18\x01 \x01(\t\x12\x0f\n\x07\x65xisted\x18\x02 \x01(\x08\x12\x0c\n\x04path\x18\x03 \x01(\t\x12\x0c\n\x04name\x18\x04 \x01(\t\"7\n\x13SaveNotebookRequest\x12\x12\n\nipynb_json\x18\x01 \x01(\t\x12\x0c\n\x04name\x18\x02 \x01(\t\"M\n\x14SaveNotebookResponse\x12\n\n\x02ok\x18\x01 \x01(\x08\x12\x0c\n\x04path\x18\x02 \x01(\t\x12\r\n\x05\x65rror\x18\x03 \x01(\t\x12\x0c\n\x04name\x18\x04 \x01(\t\"C\n\x1bGenerateNotebookCodeRequest\x12\x0e\n\x06prompt\x18\x01 \x01(\t\x12\x14\n\x0c\x63ontext_code\x18\x02 \x01(\t\"\\\n\x1cGenerateNotebookCodeResponse\x12\x0c\n\x04\x63ode\x18\x01 \x01(\t\x12\x13\n\x0b\x65xplanation\x18\x02 \x01(\t\x12\n\n\x02ok\x18\x03 \x01(\x08\x12\r\n\x05\x65rror\x18\x04 \x01(\t*d\n\x13WeightOperationType\x12\n\n\x06ZEROFY\x10\x00\x12\x10\n\x0cREINITIALIZE\x10\x01\x12\n\n\x06\x46REEZE\x10\x02\x12\x12\n\x0eREMOVE_NEURONS\x10\t\x12\x0f\n\x0b\x41\x44\x44_NEURONS\x10\n*o\n\x0fZerofyPredicate\x12\x19\n\x15ZEROFY_PREDICATE_NONE\x10\x00\x12 \n\x1cZEROFY_PREDICATE_WITH_FROZEN\x10\x01\x12\x1f\n\x1bZEROFY_PREDICATE_WITH_OLDER\x10\x02*M\n\x0f\x41gentIntentType\x12\x12\n\x0eINTENT_UNKNOWN\x10\x00\x12\x11\n\rINTENT_FILTER\x10\x01\x12\x13\n\x0fINTENT_ANALYSIS\x10\x02*I\n\x0eSampleEditType\x12\x11\n\rEDIT_OVERRIDE\x10\x00\x12\x13\n\x0f\x45\x44IT_ACCUMULATE\x10\x01\x12\x0f\n\x0b\x45\x44IT_REMOVE\x10\x02*,\n\x11\x41gentProviderType\x12\x17\n\x13PROVIDER_OPENROUTER\x10\x00\x32\xda\x0e\n\x11\x45xperimentService\x12P\n\x13GetLatestLoggerData\x12\x1b.GetLatestLoggerDataRequest\x1a\x1c.GetLatestLoggerDataResponse\x12\x36\n\x11\x45xperimentCommand\x12\x0f.TrainerCommand\x1a\x10.CommandResponse\x12H\n\x11ManipulateWeights\x12\x18.WeightsOperationRequest\x1a\x19.WeightsOperationResponse\x12/\n\nGetWeights\x12\x0f.WeightsRequest\x1a\x10.WeightsResponse\x12\x39\n\x0eGetActivations\x12\x12.ActivationRequest\x1a\x13.ActivationResponse\x12\x37\n\nGetSamples\x12\x13.BatchSampleRequest\x1a\x14.BatchSampleResponse\x12\x37\n\x0e\x41pplyDataQuery\x12\x11.DataQueryRequest\x1a\x12.DataQueryResponse\x12;\n\x0eGetDataSamples\x12\x13.DataSamplesRequest\x1a\x14.DataSamplesResponse\x12\x35\n\x0cGetHistogram\x12\x11.HistogramRequest\x1a\x12.HistogramResponse\x12\x38\n\x0bGetMetaData\x12\x13.GetMetaDataRequest\x1a\x14.GetMetaDataResponse\x12P\n\x13GetSignalTrajectory\x12\x1b.GetSignalTrajectoryRequest\x1a\x1c.GetSignalTrajectoryResponse\x12;\n\x0eGetStepSamples\x12\x13.StepSamplesRequest\x1a\x14.StepSamplesResponse\x12\x37\n\rGetPointCloud\x12\x12.PointCloudRequest\x1a\x10.PointCloudChunk0\x01\x12\x37\n\x0e\x45\x64itDataSample\x12\x11.DataEditsRequest\x1a\x12.DataEditsResponse\x12,\n\rGetDataSplits\x12\x06.Empty\x1a\x13.DataSplitsResponse\x12\x30\n\x10\x43heckAgentHealth\x12\x06.Empty\x1a\x14.AgentHealthResponse\x12\x44\n\x0fInitializeAgent\x12\x17.InitializeAgentRequest\x1a\x18.InitializeAgentResponse\x12G\n\x10\x43hangeAgentModel\x12\x18.ChangeAgentModelRequest\x1a\x19.ChangeAgentModelResponse\x12\x41\n\x0eGetAgentModels\x12\x16.GetAgentModelsRequest\x1a\x17.GetAgentModelsResponse\x12)\n\nResetAgent\x12\x06.Empty\x1a\x13.ResetAgentResponse\x12@\n\x0fRunNotebookCell\x12\x17.RunNotebookCellRequest\x1a\x12.NotebookCellChunk0\x01\x12V\n\x15InterruptNotebookCell\x12\x1d.InterruptNotebookCellRequest\x1a\x1e.InterruptNotebookCellResponse\x12(\n\x0bGetNotebook\x12\x06.Empty\x1a\x11.NotebookResponse\x12;\n\x0cSaveNotebook\x12\x14.SaveNotebookRequest\x1a\x15.SaveNotebookResponse\x12S\n\x14GenerateNotebookCode\x12\x1c.GenerateNotebookCodeRequest\x1a\x1d.GenerateNotebookCodeResponse\x12J\n\x11RestoreCheckpoint\x12\x19.RestoreCheckpointRequest\x1a\x1a.RestoreCheckpointResponse\x12J\n\x11TriggerEvaluation\x12\x19.TriggerEvaluationRequest\x1a\x1a.TriggerEvaluationResponse\x12P\n\x13GetEvaluationStatus\x12\x1b.GetEvaluationStatusRequest\x1a\x1c.GetEvaluationStatusResponse\x12G\n\x10\x43\x61ncelEvaluation\x12\x18.CancelEvaluationRequest\x1a\x19.CancelEvaluationResponseb\x06proto3') _globals = globals() _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) @@ -37,194 +37,194 @@ _globals['_NEURONSTATISTICS_INCOMINGLRENTRY']._serialized_options = b'8\001' _globals['_RECORDMETADATA_SAMPLELASTLOSSENTRY']._loaded_options = None _globals['_RECORDMETADATA_SAMPLELASTLOSSENTRY']._serialized_options = b'8\001' - _globals['_WEIGHTOPERATIONTYPE']._serialized_start=11246 - _globals['_WEIGHTOPERATIONTYPE']._serialized_end=11346 - _globals['_ZEROFYPREDICATE']._serialized_start=11348 - _globals['_ZEROFYPREDICATE']._serialized_end=11459 - _globals['_AGENTINTENTTYPE']._serialized_start=11461 - _globals['_AGENTINTENTTYPE']._serialized_end=11538 - _globals['_SAMPLEEDITTYPE']._serialized_start=11540 - _globals['_SAMPLEEDITTYPE']._serialized_end=11613 - _globals['_AGENTPROVIDERTYPE']._serialized_start=11615 - _globals['_AGENTPROVIDERTYPE']._serialized_end=11659 - _globals['_ANNOTATIONEXPORTFORMAT']._serialized_start=11661 - _globals['_ANNOTATIONEXPORTFORMAT']._serialized_end=11770 + _globals['_WEIGHTOPERATIONTYPE']._serialized_start=11449 + _globals['_WEIGHTOPERATIONTYPE']._serialized_end=11549 + _globals['_ZEROFYPREDICATE']._serialized_start=11551 + _globals['_ZEROFYPREDICATE']._serialized_end=11662 + _globals['_AGENTINTENTTYPE']._serialized_start=11664 + _globals['_AGENTINTENTTYPE']._serialized_end=11741 + _globals['_SAMPLEEDITTYPE']._serialized_start=11743 + _globals['_SAMPLEEDITTYPE']._serialized_end=11816 + _globals['_AGENTPROVIDERTYPE']._serialized_start=11818 + _globals['_AGENTPROVIDERTYPE']._serialized_end=11862 _globals['_GETLATESTLOGGERDATAREQUEST']._serialized_start=46 _globals['_GETLATESTLOGGERDATAREQUEST']._serialized_end=183 - _globals['_LOGGERDATAPOINT']._serialized_start=186 - _globals['_LOGGERDATAPOINT']._serialized_end=443 - _globals['_GETLATESTLOGGERDATARESPONSE']._serialized_start=445 - _globals['_GETLATESTLOGGERDATARESPONSE']._serialized_end=536 - _globals['_EMPTY']._serialized_start=538 - _globals['_EMPTY']._serialized_end=545 - _globals['_NEURONID']._serialized_start=547 - _globals['_NEURONID']._serialized_end=594 - _globals['_WEIGHTOPERATION']._serialized_start=597 - _globals['_WEIGHTOPERATION']._serialized_end=870 - _globals['_WEIGHTSOPERATIONREQUEST']._serialized_start=872 - _globals['_WEIGHTSOPERATIONREQUEST']._serialized_end=967 - _globals['_WEIGHTSOPERATIONRESPONSE']._serialized_start=969 - _globals['_WEIGHTSOPERATIONRESPONSE']._serialized_end=1029 - _globals['_HYPERPARAMETERS']._serialized_start=1032 - _globals['_HYPERPARAMETERS']._serialized_end=1737 - _globals['_METRICSSTATUS']._serialized_start=1739 - _globals['_METRICSSTATUS']._serialized_end=1783 - _globals['_ANNOTATSTATUS']._serialized_start=1785 - _globals['_ANNOTATSTATUS']._serialized_end=1911 - _globals['_ANNOTATSTATUS_METADATAENTRY']._serialized_start=1864 - _globals['_ANNOTATSTATUS_METADATAENTRY']._serialized_end=1911 - _globals['_TRAININGSTATUSEX']._serialized_start=1914 - _globals['_TRAININGSTATUSEX']._serialized_end=2186 - _globals['_HYPERPARAMETERCOMMAND']._serialized_start=2188 - _globals['_HYPERPARAMETERCOMMAND']._serialized_end=2281 - _globals['_DENYSAMPLESOPERATION']._serialized_start=2283 - _globals['_DENYSAMPLESOPERATION']._serialized_end=2345 - _globals['_LOADCHECKPOINTOPERATION']._serialized_start=2347 - _globals['_LOADCHECKPOINTOPERATION']._serialized_end=2395 - _globals['_PLOTNOTEOPERATION']._serialized_start=2397 - _globals['_PLOTNOTEOPERATION']._serialized_end=2495 - _globals['_SAVECHECKPOINTOPERATION']._serialized_start=2497 - _globals['_SAVECHECKPOINTOPERATION']._serialized_end=2573 - _globals['_RESTARTINSTANCEOPERATION']._serialized_start=2575 - _globals['_RESTARTINSTANCEOPERATION']._serialized_end=2601 - _globals['_TRAINERCOMMAND']._serialized_start=2604 - _globals['_TRAINERCOMMAND']._serialized_end=3641 - _globals['_HYPERPARAMETERDESC']._serialized_start=3644 - _globals['_HYPERPARAMETERDESC']._serialized_end=3801 - _globals['_NEURONSTATISTICS']._serialized_start=3804 - _globals['_NEURONSTATISTICS']._serialized_end=4174 - _globals['_NEURONSTATISTICS_INCOMINGLRENTRY']._serialized_start=4033 - _globals['_NEURONSTATISTICS_INCOMINGLRENTRY']._serialized_end=4082 - _globals['_LAYERREPRESENTATION']._serialized_start=4177 - _globals['_LAYERREPRESENTATION']._serialized_end=4545 - _globals['_ACTIVATIONREQUEST']._serialized_start=4547 - _globals['_ACTIVATIONREQUEST']._serialized_end=4619 - _globals['_ACTIVATIONMAP']._serialized_start=4621 - _globals['_ACTIVATIONMAP']._serialized_end=4693 - _globals['_ACTIVATIONRESPONSE']._serialized_start=4695 - _globals['_ACTIVATIONRESPONSE']._serialized_end=4795 - _globals['_TASKFIELD']._serialized_start=4798 - _globals['_TASKFIELD']._serialized_end=4945 - _globals['_RECORDMETADATA']._serialized_start=4948 - _globals['_RECORDMETADATA']._serialized_end=5339 - _globals['_RECORDMETADATA_SAMPLELASTLOSSENTRY']._serialized_start=5286 - _globals['_RECORDMETADATA_SAMPLELASTLOSSENTRY']._serialized_end=5339 - _globals['_SAMPLESTATISTICS']._serialized_start=5342 - _globals['_SAMPLESTATISTICS']._serialized_end=5489 - _globals['_COMMANDRESPONSE']._serialized_start=5492 - _globals['_COMMANDRESPONSE']._serialized_end=5722 - _globals['_SAMPLEREQUEST']._serialized_start=5724 - _globals['_SAMPLEREQUEST']._serialized_end=5809 - _globals['_SAMPLEREQUESTRESPONSE']._serialized_start=5812 - _globals['_SAMPLEREQUESTRESPONSE']._serialized_end=6113 - _globals['_BATCHSAMPLEREQUEST']._serialized_start=6116 - _globals['_BATCHSAMPLEREQUEST']._serialized_end=6262 - _globals['_BATCHSAMPLERESPONSE']._serialized_start=6264 - _globals['_BATCHSAMPLERESPONSE']._serialized_end=6326 - _globals['_WEIGHTSREQUEST']._serialized_start=6328 - _globals['_WEIGHTSREQUEST']._serialized_end=6374 - _globals['_WEIGHTSRESPONSE']._serialized_start=6377 - _globals['_WEIGHTSRESPONSE']._serialized_end=6662 - _globals['_DATAQUERYREQUEST']._serialized_start=6664 - _globals['_DATAQUERYREQUEST']._serialized_end=6746 - _globals['_CATEGORICALTAGDEF']._serialized_start=6748 - _globals['_CATEGORICALTAGDEF']._serialized_end=6801 - _globals['_DATAQUERYRESPONSE']._serialized_start=6804 - _globals['_DATAQUERYRESPONSE']._serialized_end=7101 - _globals['_DATASAMPLESREQUEST']._serialized_start=7104 - _globals['_DATASAMPLESREQUEST']._serialized_end=7298 - _globals['_DATASTAT']._serialized_start=7300 - _globals['_DATASTAT']._serialized_end=7409 - _globals['_DATARECORD']._serialized_start=7411 - _globals['_DATARECORD']._serialized_end=7473 - _globals['_DATASAMPLESRESPONSE']._serialized_start=7475 - _globals['_DATASAMPLESRESPONSE']._serialized_end=7565 - _globals['_HISTOGRAMSUBBAR']._serialized_start=7567 - _globals['_HISTOGRAMSUBBAR']._serialized_end=7634 - _globals['_HISTOGRAMBIN']._serialized_start=7636 - _globals['_HISTOGRAMBIN']._serialized_end=7740 - _globals['_CATEGORICALHISTOGRAMBAR']._serialized_start=7742 - _globals['_CATEGORICALHISTOGRAMBAR']._serialized_end=7833 - _globals['_HISTOGRAMREQUEST']._serialized_start=7835 - _globals['_HISTOGRAMREQUEST']._serialized_end=7887 - _globals['_HISTOGRAMRESPONSE']._serialized_start=7890 - _globals['_HISTOGRAMRESPONSE']._serialized_end=8068 - _globals['_GETMETADATAREQUEST']._serialized_start=8070 - _globals['_GETMETADATAREQUEST']._serialized_end=8157 - _globals['_GETMETADATARESPONSE']._serialized_start=8160 - _globals['_GETMETADATARESPONSE']._serialized_end=8313 - _globals['_GETSIGNALTRAJECTORYREQUEST']._serialized_start=8315 - _globals['_GETSIGNALTRAJECTORYREQUEST']._serialized_end=8404 - _globals['_SIGNALTRAJECTORY']._serialized_start=8406 - _globals['_SIGNALTRAJECTORY']._serialized_end=8458 - _globals['_GETSIGNALTRAJECTORYRESPONSE']._serialized_start=8460 - _globals['_GETSIGNALTRAJECTORYRESPONSE']._serialized_end=8585 - _globals['_POINTCLOUDREQUEST']._serialized_start=8587 - _globals['_POINTCLOUDREQUEST']._serialized_end=8661 - _globals['_POINTCLOUDCHUNK']._serialized_start=8664 - _globals['_POINTCLOUDCHUNK']._serialized_end=8855 - _globals['_DATAEDITSREQUEST']._serialized_start=8858 - _globals['_DATAEDITSREQUEST']._serialized_end=9078 - _globals['_DATAEDITSRESPONSE']._serialized_start=9080 - _globals['_DATAEDITSRESPONSE']._serialized_end=9133 - _globals['_DATASPLITSRESPONSE']._serialized_start=9135 - _globals['_DATASPLITSRESPONSE']._serialized_end=9193 - _globals['_AGENTHEALTHRESPONSE']._serialized_start=9195 - _globals['_AGENTHEALTHRESPONSE']._serialized_end=9252 - _globals['_INITIALIZEAGENTREQUEST']._serialized_start=9254 - _globals['_INITIALIZEAGENTREQUEST']._serialized_end=9348 - _globals['_INITIALIZEAGENTRESPONSE']._serialized_start=9350 - _globals['_INITIALIZEAGENTRESPONSE']._serialized_end=9409 - _globals['_CHANGEAGENTMODELREQUEST']._serialized_start=9411 - _globals['_CHANGEAGENTMODELREQUEST']._serialized_end=9451 - _globals['_CHANGEAGENTMODELRESPONSE']._serialized_start=9453 - _globals['_CHANGEAGENTMODELRESPONSE']._serialized_end=9513 - _globals['_GETAGENTMODELSREQUEST']._serialized_start=9515 - _globals['_GETAGENTMODELSREQUEST']._serialized_end=9538 - _globals['_GETAGENTMODELSRESPONSE']._serialized_start=9540 - _globals['_GETAGENTMODELSRESPONSE']._serialized_end=9614 - _globals['_RESETAGENTRESPONSE']._serialized_start=9616 - _globals['_RESETAGENTRESPONSE']._serialized_end=9670 - _globals['_RESTORECHECKPOINTREQUEST']._serialized_start=9672 - _globals['_RESTORECHECKPOINTREQUEST']._serialized_end=9723 - _globals['_RESTORECHECKPOINTRESPONSE']._serialized_start=9725 - _globals['_RESTORECHECKPOINTRESPONSE']._serialized_end=9786 - _globals['_TRIGGEREVALUATIONREQUEST']._serialized_start=9788 - _globals['_TRIGGEREVALUATIONREQUEST']._serialized_end=9870 - _globals['_TRIGGEREVALUATIONRESPONSE']._serialized_start=9872 - _globals['_TRIGGEREVALUATIONRESPONSE']._serialized_end=9933 - _globals['_GETEVALUATIONSTATUSREQUEST']._serialized_start=9935 - _globals['_GETEVALUATIONSTATUSREQUEST']._serialized_end=9963 - _globals['_GETEVALUATIONSTATUSRESPONSE']._serialized_start=9966 - _globals['_GETEVALUATIONSTATUSRESPONSE']._serialized_end=10095 - _globals['_CANCELEVALUATIONREQUEST']._serialized_start=10097 - _globals['_CANCELEVALUATIONREQUEST']._serialized_end=10138 - _globals['_CANCELEVALUATIONRESPONSE']._serialized_start=10140 - _globals['_CANCELEVALUATIONRESPONSE']._serialized_end=10200 - _globals['_RUNNOTEBOOKCELLREQUEST']._serialized_start=10202 - _globals['_RUNNOTEBOOKCELLREQUEST']._serialized_end=10257 - _globals['_NOTEBOOKCELLDONE']._serialized_start=10259 - _globals['_NOTEBOOKCELLDONE']._serialized_end=10309 - _globals['_INTERRUPTNOTEBOOKCELLREQUEST']._serialized_start=10311 - _globals['_INTERRUPTNOTEBOOKCELLREQUEST']._serialized_end=10341 - _globals['_INTERRUPTNOTEBOOKCELLRESPONSE']._serialized_start=10343 - _globals['_INTERRUPTNOTEBOOKCELLRESPONSE']._serialized_end=10401 - _globals['_NOTEBOOKCELLCHUNK']._serialized_start=10404 - _globals['_NOTEBOOKCELLCHUNK']._serialized_end=10593 - _globals['_NOTEBOOKRESPONSE']._serialized_start=10595 - _globals['_NOTEBOOKRESPONSE']._serialized_end=10678 - _globals['_SAVENOTEBOOKREQUEST']._serialized_start=10680 - _globals['_SAVENOTEBOOKREQUEST']._serialized_end=10735 - _globals['_SAVENOTEBOOKRESPONSE']._serialized_start=10737 - _globals['_SAVENOTEBOOKRESPONSE']._serialized_end=10814 - _globals['_GENERATENOTEBOOKCODEREQUEST']._serialized_start=10816 - _globals['_GENERATENOTEBOOKCODEREQUEST']._serialized_end=10883 - _globals['_GENERATENOTEBOOKCODERESPONSE']._serialized_start=10885 - _globals['_GENERATENOTEBOOKCODERESPONSE']._serialized_end=10977 - _globals['_EXPORTANNOTATIONSREQUEST']._serialized_start=10979 - _globals['_EXPORTANNOTATIONSREQUEST']._serialized_end=11105 - _globals['_EXPORTANNOTATIONSRESPONSE']._serialized_start=11108 - _globals['_EXPORTANNOTATIONSRESPONSE']._serialized_end=11244 - _globals['_EXPERIMENTSERVICE']._serialized_start=11773 - _globals['_EXPERIMENTSERVICE']._serialized_end=13670 + _globals['_SIGNALOUTLIER']._serialized_start=185 + _globals['_SIGNALOUTLIER']._serialized_end=234 + _globals['_LOGGERDATAPOINT']._serialized_start=237 + _globals['_LOGGERDATAPOINT']._serialized_end=703 + _globals['_GETLATESTLOGGERDATARESPONSE']._serialized_start=705 + _globals['_GETLATESTLOGGERDATARESPONSE']._serialized_end=796 + _globals['_EMPTY']._serialized_start=798 + _globals['_EMPTY']._serialized_end=805 + _globals['_NEURONID']._serialized_start=807 + _globals['_NEURONID']._serialized_end=854 + _globals['_WEIGHTOPERATION']._serialized_start=857 + _globals['_WEIGHTOPERATION']._serialized_end=1130 + _globals['_WEIGHTSOPERATIONREQUEST']._serialized_start=1132 + _globals['_WEIGHTSOPERATIONREQUEST']._serialized_end=1227 + _globals['_WEIGHTSOPERATIONRESPONSE']._serialized_start=1229 + _globals['_WEIGHTSOPERATIONRESPONSE']._serialized_end=1289 + _globals['_HYPERPARAMETERS']._serialized_start=1292 + _globals['_HYPERPARAMETERS']._serialized_end=1997 + _globals['_METRICSSTATUS']._serialized_start=1999 + _globals['_METRICSSTATUS']._serialized_end=2043 + _globals['_ANNOTATSTATUS']._serialized_start=2045 + _globals['_ANNOTATSTATUS']._serialized_end=2171 + _globals['_ANNOTATSTATUS_METADATAENTRY']._serialized_start=2124 + _globals['_ANNOTATSTATUS_METADATAENTRY']._serialized_end=2171 + _globals['_TRAININGSTATUSEX']._serialized_start=2174 + _globals['_TRAININGSTATUSEX']._serialized_end=2446 + _globals['_HYPERPARAMETERCOMMAND']._serialized_start=2448 + _globals['_HYPERPARAMETERCOMMAND']._serialized_end=2541 + _globals['_DENYSAMPLESOPERATION']._serialized_start=2543 + _globals['_DENYSAMPLESOPERATION']._serialized_end=2605 + _globals['_LOADCHECKPOINTOPERATION']._serialized_start=2607 + _globals['_LOADCHECKPOINTOPERATION']._serialized_end=2655 + _globals['_PLOTNOTEOPERATION']._serialized_start=2657 + _globals['_PLOTNOTEOPERATION']._serialized_end=2755 + _globals['_SAVECHECKPOINTOPERATION']._serialized_start=2757 + _globals['_SAVECHECKPOINTOPERATION']._serialized_end=2833 + _globals['_RESTARTINSTANCEOPERATION']._serialized_start=2835 + _globals['_RESTARTINSTANCEOPERATION']._serialized_end=2861 + _globals['_TRAINERCOMMAND']._serialized_start=2864 + _globals['_TRAINERCOMMAND']._serialized_end=3901 + _globals['_HYPERPARAMETERDESC']._serialized_start=3904 + _globals['_HYPERPARAMETERDESC']._serialized_end=4061 + _globals['_NEURONSTATISTICS']._serialized_start=4064 + _globals['_NEURONSTATISTICS']._serialized_end=4434 + _globals['_NEURONSTATISTICS_INCOMINGLRENTRY']._serialized_start=4293 + _globals['_NEURONSTATISTICS_INCOMINGLRENTRY']._serialized_end=4342 + _globals['_LAYERREPRESENTATION']._serialized_start=4437 + _globals['_LAYERREPRESENTATION']._serialized_end=4805 + _globals['_ACTIVATIONREQUEST']._serialized_start=4807 + _globals['_ACTIVATIONREQUEST']._serialized_end=4879 + _globals['_ACTIVATIONMAP']._serialized_start=4881 + _globals['_ACTIVATIONMAP']._serialized_end=4953 + _globals['_ACTIVATIONRESPONSE']._serialized_start=4955 + _globals['_ACTIVATIONRESPONSE']._serialized_end=5055 + _globals['_TASKFIELD']._serialized_start=5058 + _globals['_TASKFIELD']._serialized_end=5205 + _globals['_RECORDMETADATA']._serialized_start=5208 + _globals['_RECORDMETADATA']._serialized_end=5599 + _globals['_RECORDMETADATA_SAMPLELASTLOSSENTRY']._serialized_start=5546 + _globals['_RECORDMETADATA_SAMPLELASTLOSSENTRY']._serialized_end=5599 + _globals['_SAMPLESTATISTICS']._serialized_start=5602 + _globals['_SAMPLESTATISTICS']._serialized_end=5749 + _globals['_COMMANDRESPONSE']._serialized_start=5752 + _globals['_COMMANDRESPONSE']._serialized_end=5982 + _globals['_SAMPLEREQUEST']._serialized_start=5984 + _globals['_SAMPLEREQUEST']._serialized_end=6069 + _globals['_SAMPLEREQUESTRESPONSE']._serialized_start=6072 + _globals['_SAMPLEREQUESTRESPONSE']._serialized_end=6373 + _globals['_BATCHSAMPLEREQUEST']._serialized_start=6376 + _globals['_BATCHSAMPLEREQUEST']._serialized_end=6522 + _globals['_BATCHSAMPLERESPONSE']._serialized_start=6524 + _globals['_BATCHSAMPLERESPONSE']._serialized_end=6586 + _globals['_WEIGHTSREQUEST']._serialized_start=6588 + _globals['_WEIGHTSREQUEST']._serialized_end=6634 + _globals['_WEIGHTSRESPONSE']._serialized_start=6637 + _globals['_WEIGHTSRESPONSE']._serialized_end=6922 + _globals['_DATAQUERYREQUEST']._serialized_start=6924 + _globals['_DATAQUERYREQUEST']._serialized_end=7006 + _globals['_CATEGORICALTAGDEF']._serialized_start=7008 + _globals['_CATEGORICALTAGDEF']._serialized_end=7061 + _globals['_DATAQUERYRESPONSE']._serialized_start=7064 + _globals['_DATAQUERYRESPONSE']._serialized_end=7361 + _globals['_DATASAMPLESREQUEST']._serialized_start=7364 + _globals['_DATASAMPLESREQUEST']._serialized_end=7558 + _globals['_DATASTAT']._serialized_start=7560 + _globals['_DATASTAT']._serialized_end=7669 + _globals['_DATARECORD']._serialized_start=7671 + _globals['_DATARECORD']._serialized_end=7733 + _globals['_DATASAMPLESRESPONSE']._serialized_start=7735 + _globals['_DATASAMPLESRESPONSE']._serialized_end=7825 + _globals['_HISTOGRAMSUBBAR']._serialized_start=7827 + _globals['_HISTOGRAMSUBBAR']._serialized_end=7894 + _globals['_HISTOGRAMBIN']._serialized_start=7896 + _globals['_HISTOGRAMBIN']._serialized_end=8000 + _globals['_CATEGORICALHISTOGRAMBAR']._serialized_start=8002 + _globals['_CATEGORICALHISTOGRAMBAR']._serialized_end=8093 + _globals['_HISTOGRAMREQUEST']._serialized_start=8095 + _globals['_HISTOGRAMREQUEST']._serialized_end=8147 + _globals['_HISTOGRAMRESPONSE']._serialized_start=8150 + _globals['_HISTOGRAMRESPONSE']._serialized_end=8328 + _globals['_GETMETADATAREQUEST']._serialized_start=8330 + _globals['_GETMETADATAREQUEST']._serialized_end=8417 + _globals['_GETMETADATARESPONSE']._serialized_start=8420 + _globals['_GETMETADATARESPONSE']._serialized_end=8573 + _globals['_STEPSAMPLESREQUEST']._serialized_start=8575 + _globals['_STEPSAMPLESREQUEST']._serialized_end=8681 + _globals['_STEPSAMPLESRESPONSE']._serialized_start=8683 + _globals['_STEPSAMPLESRESPONSE']._serialized_end=8783 + _globals['_GETSIGNALTRAJECTORYREQUEST']._serialized_start=8785 + _globals['_GETSIGNALTRAJECTORYREQUEST']._serialized_end=8874 + _globals['_SIGNALTRAJECTORY']._serialized_start=8876 + _globals['_SIGNALTRAJECTORY']._serialized_end=8928 + _globals['_GETSIGNALTRAJECTORYRESPONSE']._serialized_start=8930 + _globals['_GETSIGNALTRAJECTORYRESPONSE']._serialized_end=9055 + _globals['_POINTCLOUDREQUEST']._serialized_start=9057 + _globals['_POINTCLOUDREQUEST']._serialized_end=9131 + _globals['_POINTCLOUDCHUNK']._serialized_start=9134 + _globals['_POINTCLOUDCHUNK']._serialized_end=9325 + _globals['_DATAEDITSREQUEST']._serialized_start=9328 + _globals['_DATAEDITSREQUEST']._serialized_end=9548 + _globals['_DATAEDITSRESPONSE']._serialized_start=9550 + _globals['_DATAEDITSRESPONSE']._serialized_end=9603 + _globals['_DATASPLITSRESPONSE']._serialized_start=9605 + _globals['_DATASPLITSRESPONSE']._serialized_end=9663 + _globals['_AGENTHEALTHRESPONSE']._serialized_start=9665 + _globals['_AGENTHEALTHRESPONSE']._serialized_end=9722 + _globals['_INITIALIZEAGENTREQUEST']._serialized_start=9724 + _globals['_INITIALIZEAGENTREQUEST']._serialized_end=9818 + _globals['_INITIALIZEAGENTRESPONSE']._serialized_start=9820 + _globals['_INITIALIZEAGENTRESPONSE']._serialized_end=9879 + _globals['_CHANGEAGENTMODELREQUEST']._serialized_start=9881 + _globals['_CHANGEAGENTMODELREQUEST']._serialized_end=9921 + _globals['_CHANGEAGENTMODELRESPONSE']._serialized_start=9923 + _globals['_CHANGEAGENTMODELRESPONSE']._serialized_end=9983 + _globals['_GETAGENTMODELSREQUEST']._serialized_start=9985 + _globals['_GETAGENTMODELSREQUEST']._serialized_end=10008 + _globals['_GETAGENTMODELSRESPONSE']._serialized_start=10010 + _globals['_GETAGENTMODELSRESPONSE']._serialized_end=10084 + _globals['_RESETAGENTRESPONSE']._serialized_start=10086 + _globals['_RESETAGENTRESPONSE']._serialized_end=10140 + _globals['_RESTORECHECKPOINTREQUEST']._serialized_start=10142 + _globals['_RESTORECHECKPOINTREQUEST']._serialized_end=10193 + _globals['_RESTORECHECKPOINTRESPONSE']._serialized_start=10195 + _globals['_RESTORECHECKPOINTRESPONSE']._serialized_end=10256 + _globals['_TRIGGEREVALUATIONREQUEST']._serialized_start=10258 + _globals['_TRIGGEREVALUATIONREQUEST']._serialized_end=10340 + _globals['_TRIGGEREVALUATIONRESPONSE']._serialized_start=10342 + _globals['_TRIGGEREVALUATIONRESPONSE']._serialized_end=10403 + _globals['_GETEVALUATIONSTATUSREQUEST']._serialized_start=10405 + _globals['_GETEVALUATIONSTATUSREQUEST']._serialized_end=10433 + _globals['_GETEVALUATIONSTATUSRESPONSE']._serialized_start=10436 + _globals['_GETEVALUATIONSTATUSRESPONSE']._serialized_end=10565 + _globals['_CANCELEVALUATIONREQUEST']._serialized_start=10567 + _globals['_CANCELEVALUATIONREQUEST']._serialized_end=10608 + _globals['_CANCELEVALUATIONRESPONSE']._serialized_start=10610 + _globals['_CANCELEVALUATIONRESPONSE']._serialized_end=10670 + _globals['_RUNNOTEBOOKCELLREQUEST']._serialized_start=10672 + _globals['_RUNNOTEBOOKCELLREQUEST']._serialized_end=10727 + _globals['_NOTEBOOKCELLDONE']._serialized_start=10729 + _globals['_NOTEBOOKCELLDONE']._serialized_end=10779 + _globals['_INTERRUPTNOTEBOOKCELLREQUEST']._serialized_start=10781 + _globals['_INTERRUPTNOTEBOOKCELLREQUEST']._serialized_end=10811 + _globals['_INTERRUPTNOTEBOOKCELLRESPONSE']._serialized_start=10813 + _globals['_INTERRUPTNOTEBOOKCELLRESPONSE']._serialized_end=10871 + _globals['_NOTEBOOKCELLCHUNK']._serialized_start=10874 + _globals['_NOTEBOOKCELLCHUNK']._serialized_end=11063 + _globals['_NOTEBOOKRESPONSE']._serialized_start=11065 + _globals['_NOTEBOOKRESPONSE']._serialized_end=11148 + _globals['_SAVENOTEBOOKREQUEST']._serialized_start=11150 + _globals['_SAVENOTEBOOKREQUEST']._serialized_end=11205 + _globals['_SAVENOTEBOOKRESPONSE']._serialized_start=11207 + _globals['_SAVENOTEBOOKRESPONSE']._serialized_end=11284 + _globals['_GENERATENOTEBOOKCODEREQUEST']._serialized_start=11286 + _globals['_GENERATENOTEBOOKCODEREQUEST']._serialized_end=11353 + _globals['_GENERATENOTEBOOKCODERESPONSE']._serialized_start=11355 + _globals['_GENERATENOTEBOOKCODERESPONSE']._serialized_end=11447 + _globals['_EXPERIMENTSERVICE']._serialized_start=11865 + _globals['_EXPERIMENTSERVICE']._serialized_end=13747 # @@protoc_insertion_point(module_scope) diff --git a/weightslab/proto/experiment_service_pb2_grpc.py b/weightslab/proto/experiment_service_pb2_grpc.py index 2967171a..7b61f4c4 100644 --- a/weightslab/proto/experiment_service_pb2_grpc.py +++ b/weightslab/proto/experiment_service_pb2_grpc.py @@ -89,6 +89,11 @@ def __init__(self, channel): request_serializer=weightslab_dot_proto_dot_experiment__service__pb2.GetSignalTrajectoryRequest.SerializeToString, response_deserializer=weightslab_dot_proto_dot_experiment__service__pb2.GetSignalTrajectoryResponse.FromString, _registered_method=True) + self.GetStepSamples = channel.unary_unary( + '/ExperimentService/GetStepSamples', + request_serializer=weightslab_dot_proto_dot_experiment__service__pb2.StepSamplesRequest.SerializeToString, + response_deserializer=weightslab_dot_proto_dot_experiment__service__pb2.StepSamplesResponse.FromString, + _registered_method=True) self.GetPointCloud = channel.unary_stream( '/ExperimentService/GetPointCloud', request_serializer=weightslab_dot_proto_dot_experiment__service__pb2.PointCloudRequest.SerializeToString, @@ -261,6 +266,15 @@ def GetSignalTrajectory(self, request, context): context.set_details('Method not implemented!') raise NotImplementedError('Method not implemented!') + def GetStepSamples(self, request, context): + """Every sample id that contributed to one step of one signal. Backs the plot's + right-click "Highlight step samples": the whole batch behind a point, not + just the samples that were flagged as off-trend. + """ + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + def GetPointCloud(self, request, context): """Raw point cloud of one sample (task_type "detection_pointcloud"), server-streamed in binary chunks for the interactive 3D viewer. @@ -443,6 +457,11 @@ def add_ExperimentServiceServicer_to_server(servicer, server): request_deserializer=weightslab_dot_proto_dot_experiment__service__pb2.GetSignalTrajectoryRequest.FromString, response_serializer=weightslab_dot_proto_dot_experiment__service__pb2.GetSignalTrajectoryResponse.SerializeToString, ), + 'GetStepSamples': grpc.unary_unary_rpc_method_handler( + servicer.GetStepSamples, + request_deserializer=weightslab_dot_proto_dot_experiment__service__pb2.StepSamplesRequest.FromString, + response_serializer=weightslab_dot_proto_dot_experiment__service__pb2.StepSamplesResponse.SerializeToString, + ), 'GetPointCloud': grpc.unary_stream_rpc_method_handler( servicer.GetPointCloud, request_deserializer=weightslab_dot_proto_dot_experiment__service__pb2.PointCloudRequest.FromString, @@ -841,6 +860,33 @@ def GetSignalTrajectory(request, metadata, _registered_method=True) + @staticmethod + def GetStepSamples(request, + target, + options=(), + channel_credentials=None, + call_credentials=None, + insecure=False, + compression=None, + wait_for_ready=None, + timeout=None, + metadata=None): + return grpc.experimental.unary_unary( + request, + target, + '/ExperimentService/GetStepSamples', + weightslab_dot_proto_dot_experiment__service__pb2.StepSamplesRequest.SerializeToString, + weightslab_dot_proto_dot_experiment__service__pb2.StepSamplesResponse.FromString, + options, + channel_credentials, + insecure, + call_credentials, + compression, + wait_for_ready, + timeout, + metadata, + _registered_method=True) + @staticmethod def GetPointCloud(request, target, diff --git a/weightslab/src.py b/weightslab/src.py index f2849818..2c9b3e48 100644 --- a/weightslab/src.py +++ b/weightslab/src.py @@ -6,6 +6,7 @@ import gc import os import sys +import math import time import types import ctypes @@ -1187,11 +1188,44 @@ def watch_or_edit(obj: Callable, obj_name: str = None, flag: str = None, **kwarg # unless the caller explicitly disables it. forced_model_wrapping = kwargs.pop('forced_model_wrapping', False) + # Per-step training-dynamics signals (gradient/weight norms, activation + # stats). Popped here rather than forwarded, because ModelInterface has + # no business knowing about them: they are hooks ON a wrapped model, not + # part of what wrapping means. See components/model_signals.py. + track_signals = kwargs.pop('track_model_signals', False) + signals_kwargs = { + key: kwargs.pop(key) + for key in ('model_signals_every_n_steps', 'model_signals_layer_ids') + if key in kwargs + } + # Now construct the wrapper and let it register into the ledger. wrapper = ModelInterface(obj, **kwargs) if forced_model_wrapping or _model == None else _model # No rebind here since the model wrapper is designed to be a drop-in replacement for the original model + if track_signals: + from weightslab.components.model_signals import METRICS, track_model_signals as _track + # `True` means "all of them"; a list/tuple narrows the set. + if track_signals is True: + metrics = METRICS + elif isinstance(track_signals, str): + metrics = (track_signals,) + else: + metrics = tuple(track_signals) + try: + _track( + _model if _model is not None else wrapper, + metrics=metrics, + every_n_steps=signals_kwargs.get('model_signals_every_n_steps', 1), + layer_ids=signals_kwargs.get('model_signals_layer_ids'), + ) + except Exception as exc: + # Instrumentation is observability, never a reason a training + # script fails to start -- the run is still fully usable + # without these curves. + logger.warning(f"Could not install model signal tracking: {exc}") + # Prefer returning the proxy (if one exists) so external callers hold # a stable reference that will see updates. If no proxy was # obtainable, return the wrapper itself. @@ -2782,6 +2816,119 @@ def save_group_signals( DATAFRAME_M.update_by_groups_bulk(origin=origin, group_ids=active_group_ids, updates_list=all_updates) +def save_model_signals( + signals: dict, + step: int | None = None, +): + """Save **per-step** scalars that describe the MODEL, not any sample. + + This is the step-keyed sibling of :func:`save_signals` (per sample), + :func:`save_instance_signals` (per annotation) and + :func:`save_group_signals` (per group). Those three all write onto + dataframe rows, because every value they record belongs to something in + the dataset. A gradient norm does not: it belongs to the optimization + step that produced it, and the batch behind it is incidental. So nothing + here touches the dataframe — each value becomes one point on its own + signal curve, plotted exactly like a watched loss. + + Reach for this for training-dynamics values: gradient norms, weight + norms, activation statistics, learning rate, gradient-to-weight ratios. + Anything you would otherwise have had to fake by broadcasting one number + across a whole batch of ``batch_ids`` — which pollutes every one of those + samples' history with a value that was never about them. + + Naming is what groups the curves in the UI, since ``/`` is a path + separator there. The convention the shipped examples use: + + ``metrics/global/`` whole-model values + ``metrics/layer//`` per-layer values + + Args: + signals (dict): ``{name: value}``. Values may be Python numbers, or + 0-d / reducible tensors and arrays (mean-reduced to one scalar). + Non-finite values (NaN/inf) are dropped rather than plotted — + a diverging run should break the curve, not the dashboard. + step (int, optional): Training step. Inferred from the watched + model's age when omitted, same as every other ``save_*`` verb. + + Examples: + Global gradient norm, straight after ``backward()``:: + + total = sum(p.grad.pow(2).sum() for p in model.parameters() + if p.grad is not None) + wl.save_model_signals({"metrics/global/grad_norm": total.sqrt()}) + + Per-layer weight norm:: + + wl.save_model_signals({ + f"metrics/layer/{lid}/weights_norm": layer.weight.norm() + for lid, layer in tracked.items() + }) + + See also: + :func:`track_model_signals` — collects all of the above for you via + hooks, so a script never writes this loop by hand. + """ + if not signals: + return + + step = _get_step(step=step) + + for name, value in signals.items(): + # One scalar per name per step. `_extract_scalar_from_tensor` already + # handles 0-d tensors, arrays and plain numbers (mean-reducing + # anything with extra dims), so a caller can hand over whatever shape + # their metric naturally has. + scalar, _ = _extract_scalar_from_tensor(value) + if scalar is None: + try: + scalar = float(value) + except (TypeError, ValueError): + logger.debug(f"save_model_signals: skipping non-numeric signal {name!r}") + continue + + # A NaN/inf gradient norm is real information ("this run just blew + # up"), but it is not a plottable point — it rescales the whole axis + # and hides every healthy value before it. Drop the point and say so + # once, so the gap in the curve is the signal. + if not math.isfinite(scalar): + logger.warning(f"save_model_signals: {name} is {scalar} at step {step}; point dropped") + continue + + try: + _logger_obj = get_logger() + except Exception: + _logger_obj = None + if _logger_obj is None or not hasattr(_logger_obj, 'add_scalars'): + return + + # `aggregate_by_step=False` + no per-sample map is the immediate path: + # the point is appended to this signal's history as-is instead of + # being averaged into a per-step bucket (see LoggerQueue.add_scalars). + # That is what we want — there is only ever one value per step here, + # so there is nothing to aggregate, and buffering it would delay the + # point by a step for no benefit. + _logger_obj.add_scalars( + name, + {name: scalar}, + global_step=step, + signal_per_sample=None, + aggregate_by_step=False, + ) + + +def track_model_signals(model=None, **kwargs): + """Instrument a watched model so its training dynamics log themselves. + + Thin re-export of + :func:`weightslab.components.model_signals.track_model_signals`; see that + function for the full argument list. Equivalent to passing + ``track_model_signals=True`` to ``watch_or_edit(model, flag="model")``. + """ + from weightslab.components.model_signals import track_model_signals as _track + return _track(model, **kwargs) + + def clear_all(): """Clear all WeightsLab registries (models, dataloaders, etc.).""" ledgers.clear_all() diff --git a/weightslab/trainer/services/experiment_service.py b/weightslab/trainer/services/experiment_service.py index 7cd85b62..9c9460aa 100644 --- a/weightslab/trainer/services/experiment_service.py +++ b/weightslab/trainer/services/experiment_service.py @@ -74,6 +74,82 @@ def _downsample_uniform(series, max_points: int): return picked +def _outliers_pb(entry) -> list: + """Convert a signal entry's stored outliers into ``SignalOutlier`` messages.""" + raw = entry.get("outliers") or [] + if not isinstance(raw, list): + return [] + out = [] + for item in raw: + if not isinstance(item, dict): + continue + sample_id = str(item.get("sample_id", "")) + if not sample_id: + continue + try: + value = float(item.get("value", 0.0) or 0.0) + except (TypeError, ValueError): + continue + out.append(pb2.SignalOutlier(sample_id=sample_id, value=value)) + + +def _logger_point_pb(metric_name: str, entry: dict, sample_id: str = "") -> "pb2.LoggerDataPoint": + """Build one ``LoggerDataPoint`` from a logger history/queue entry dict. + + Shared by the full-history and live-queue paths so the two can't drift on + which fields they forward. + """ + return pb2.LoggerDataPoint( + metric_name=metric_name, + model_age=entry.get("model_age", 0), + metric_value=entry.get("metric_value", 0.0), + experiment_hash=entry.get("experiment_hash", "N.A."), + timestamp=int(entry.get("timestamp", time.time())), + sample_id=sample_id, + is_evaluation_marker=bool(entry.get("is_evaluation_marker", False)), + split_name=str(entry.get("split_name", "")), + evaluation_tags=[str(tag) for tag in entry.get("evaluation_tags", []) or []], + point_note=str(entry.get("point_note", "")), + audit_mode=bool(entry.get("audit_mode", False)), + outliers=_outliers_pb(entry), + outlier_count=int(entry.get("outlier_count", 0) or 0), + sample_count=int(entry.get("sample_count", 0) or 0), + trend_value=float(entry.get("trend_value") or 0.0), + trend_margin=float(entry.get("trend_margin") or 0.0), + # Explicit flags: a band of (0, 0) is indistinguishable from "no band" on + # the wire, since proto3 scalars have no presence. + has_trend_band=entry.get("trend_value") is not None and entry.get("trend_margin") is not None, + value_min=float(entry.get("value_min") or 0.0), + value_max=float(entry.get("value_max") or 0.0), + has_value_range=entry.get("value_min") is not None and entry.get("value_max") is not None, + ) + + +def _downsample_preserving_outliers(signal_history, max_points: int): + """Stride ``signal_history`` down to ~``max_points``, never dropping a spike. + + Plain striding is fine for the smooth body of a curve but would throw away + exactly the anomalies the plot is meant to surface — an outlier is by nature + a single step, so a stride of 5 loses 4 out of 5 of them. Points carrying + outliers are therefore always kept, on top of the strided baseline. + + The result can exceed ``max_points`` on a pathologically spiky run; that is + the intended trade (completeness of anomalies over an exact cap). + """ + n = len(signal_history) + if max_points <= 0 or n <= max_points: + return signal_history + + stride = max(1, n // max_points) + kept, seen = [], set() + for index, entry in enumerate(signal_history): + if index % stride == 0 or entry.get("outliers"): + if index not in seen: + seen.add(index) + kept.append(entry) + return kept + + class ExperimentService(pb2_grpc.ExperimentServiceServicer): """ Domain-level experiment service that orchestrates model/data services @@ -179,6 +255,51 @@ def GetLatestLoggerData(self, request, context): _level("GetLatestLoggerData: done elapsed=%.1fms in_flight_peak=%d client_active=%s", elapsed_ms, _in_flight, _active) + def GetStepSamples(self, request, context): + """Return every sample id behind one plotted step. + + The plot's "Highlight step samples" action shows the whole batch for a + point, so the answer must come from the per-sample table rather than from + the outlier list recorded on the aggregated point. + """ + self._ctx.ensure_components() + signal_logger = self._ctx.components.get("signal_logger") + if signal_logger is None: + return pb2.StepSamplesResponse( + success=False, message="Signal logger unavailable") + + metric_name = str(request.metric_name or "") + if not metric_name: + return pb2.StepSamplesResponse( + success=False, message="metric_name is required") + + try: + sample_ids, total = signal_logger.get_step_sample_ids( + metric_name, + str(request.experiment_hash or ""), + int(request.model_age), + int(request.max_samples or 0), + ) + except Exception as exc: + logger.exception("GetStepSamples failed for %s", metric_name) + return pb2.StepSamplesResponse(success=False, message=str(exc)) + + if not sample_ids: + # Not an error: signals that log only an aggregate have no per-sample + # rows, and the UI needs to tell that apart from a failure. + return pb2.StepSamplesResponse( + success=True, + message=f"No per-sample data recorded for {metric_name} at step {request.model_age}", + sample_ids=[], total_available=0, + ) + + return pb2.StepSamplesResponse( + success=True, + message=f"{len(sample_ids)} of {total} sample(s)", + sample_ids=sample_ids, + total_available=total, + ) + def _get_latest_logger_data_impl(self, request, context): self._ctx.ensure_components() components = self._ctx.components @@ -313,28 +434,12 @@ def _get_latest_logger_data_impl(self, request, context): # Keep deterministic order by model_age before sampling signal_history = sorted(signal_history, key=lambda item: item.get("model_age", 0)) - # Downsample if we have more than 1000 points - if len(signal_history) > max_points: - # Calculate step to downsample (e.g., if 5000 points, step=5 to get ~1000) - step = max(1, len(signal_history) // max_points) - signal_history = signal_history[::step] + # Downsample if we have more than max_points, keeping every + # outlier-bearing step (see _downsample_preserving_outliers). + signal_history = _downsample_preserving_outliers(signal_history, max_points) for s in signal_history: - points.append( - pb2.LoggerDataPoint( - metric_name=metric_name, - model_age=s.get("model_age", 0), - metric_value=s.get("metric_value", 0.0), - experiment_hash=s.get("experiment_hash", "N.A."), - timestamp=int(s.get("timestamp", time.time())), - sample_id="", # No sample_id in aggregated mode - is_evaluation_marker=bool(s.get("is_evaluation_marker", False)), - split_name=str(s.get("split_name", "")), - evaluation_tags=[str(tag) for tag in s.get("evaluation_tags", []) or []], - point_note=str(s.get("point_note", "")), - audit_mode=bool(s.get("audit_mode", False)), - ) - ) + points.append(_logger_point_pb(metric_name, s)) else: # Return only queue (new data since last poll) if context and not context.is_active(): @@ -346,21 +451,7 @@ def _get_latest_logger_data_impl(self, request, context): if _tq_ms > 200: logger.warning("get_and_clear_queue() took %.1fms (slow — possible lock contention)", _tq_ms) for s in queue_data: - points.append( - pb2.LoggerDataPoint( - metric_name=s.get("metric_name", ""), - model_age=s.get("model_age", 0), - metric_value=s.get("metric_value", 0.0), - experiment_hash=s.get("experiment_hash", "N.A."), - timestamp=int(s.get("timestamp", time.time())), - sample_id="", # No sample_id in queue mode - is_evaluation_marker=bool(s.get("is_evaluation_marker", False)), - split_name=str(s.get("split_name", "")), - evaluation_tags=[str(tag) for tag in s.get("evaluation_tags", []) or []], - point_note=str(s.get("point_note", "")), - audit_mode=bool(s.get("audit_mode", False)), - ) - ) + points.append(_logger_point_pb(s.get("metric_name", ""), s)) return pb2.GetLatestLoggerDataResponse(points=points)