Plots outliers interactions and Model Tracking Upgrade

AGENTS.md

@@ -132,6 +132,37 @@ Conventions that matter for correctness:
   supports both `hp.get("lr")` and `hp["lr"]` (subscript == `.get`), and stays
   live — reads reflect in-place updates and re-registration.
 
+Recording values (pick by what the value is *about*, not by convenience):
+
+| Value describes | Verb | Example |
+|---|---|---|
+| one sample | `wl.save_signals(signals={...}, batch_ids=ids, ...)` | an image's loss |
+| one annotation | `wl.save_instance_signals(...)` | one box's IoU |
+| a group of samples | `wl.save_group_signals(signals={...}, group_ids=[...])` | a pair's contrastive loss |
+| one training **step** | `wl.save_model_signals(signals={...})` | a gradient norm |
+
+The first three write dataframe rows (sortable/filterable in the grid); the
+fourth only plots a curve. Never fake a step-level value by broadcasting it
+across `batch_ids` — that writes a number onto samples it was never about.
+
+- **Training-dynamics signals** come free with
+  `wl.watch_or_edit(model, flag="model", track_model_signals=True,
+  model_signals_every_n_steps=N)`. That emits, per step and with no call in the
+  training loop: `metrics/global/{grad_norm,weights_norm}` and
+  `metrics/layer/<layer_id>/{grad_norm,weights_norm,activation_mean,
+  activation_std,activation_max,activation_min}`. `<layer_id>` is the same id
+  architecture ops (freeze/reset) address, so a bad curve names the layer to act
+  on. Only collects inside `guard_training_context`, so eval never contaminates
+  it. Implementation: `weightslab/weightslab/components/model_signals.py`;
+  example: `examples/Usecases/wl-fashion-mnist-signals`.
+  - Diagnosing from these: early-layer `grad_norm` → 0 with healthy late layers
+    = vanishing gradient; `grad_norm` spiking orders of magnitude = exploding;
+    `activation_std` → 0 on a layer = that layer went constant (dead
+    ReLU/saturated BN); `weights_norm` growing while loss flattens = needs decay.
+  - Per-layer curves need per-layer modules: an `nn.Sequential` block resolves
+    to ONE layer id, so a model built out of Sequentials gets one curve for the
+    whole block.
+
 ---
 
 ## 4. Configuration (environment variables)

docs/_static/examples-gallery.js

@@ -82,6 +82,13 @@
       tags: ['loss analysis', 'signal', 'categorical tag', 'per-sample', 'trajectory'],
       url: 'examples/usecases/loss_shape_classification.html',
       colab: COLAB + 'Usecases/wl-segmentation-loss-shapes-classification.ipynb'
+    },
+    {
+      badge: 'Usecase', color: 'usecase',
+      title: 'Model Signals — Fashion-MNIST',
+      desc: 'Per-step training dynamics: global and per-layer gradient norms, weight norms and activation statistics, from one argument on the model wrap.',
+      tags: ['model signals', 'gradient norm', 'activations', 'per-layer', 'training dynamics'],
+      url: 'examples/usecases/model_signals.html'
     }
   ];
 

docs/examples/usecases/index.rst

@@ -4,7 +4,7 @@ Specific User Usecases
 ======================
 
 Task-specific integrations that go beyond the standard loop: point-cloud
-inputs and per-sample loss trajectory analysis.
+inputs, per-sample loss trajectory analysis, and per-layer training dynamics.
 
 .. raw:: html
 
@@ -18,3 +18,4 @@ inputs and per-sample loss trajectory analysis.
 
    lidar_detection
    loss_shape_classification
+   model_signals

docs/examples/usecases/model_signals.rst

@@ -0,0 +1,162 @@
+Model Signals on Fashion-MNIST
+===============================
+
+.. raw:: html
+
+   <div class="wl-eg-page-tags">
+     <span class="wl-eg-badge wl-eg-badge--usecase">Usecase</span>
+     <span class="wl-eg-tag">model signals</span>
+     <span class="wl-eg-tag">gradient norm</span>
+     <span class="wl-eg-tag">activations</span>
+     <span class="wl-eg-tag">per-layer</span>
+     <span class="wl-eg-tag">training dynamics</span>
+   </div>
+
+**Example:** ``weightslab/examples/Usecases/wl-fashion-mnist-signals``
+
+This use case trains a small CNN on Fashion-MNIST and adds one thing on top of
+the plain per-sample logging: the run plots **its own training dynamics**. A
+loss curve tells you whether the model is learning; these curves tell you
+*where* in the model something went wrong.
+
+Everything below comes from one argument.
+
+The integration
+---------------
+
+.. code-block:: python
+
+   model = wl.watch_or_edit(
+       FashionCNN(),
+       flag="model",
+       device=device,
+       track_model_signals=True,          # <- the whole feature
+       model_signals_every_n_steps=1,
+   )
+
+No hooks to write, and **no call anywhere in the training loop** — the loop is
+byte-for-byte the same as ``wl-classification``'s. Pass a list instead of
+``True`` to narrow the set, e.g. ``track_model_signals=["grad_norm",
+"activation_std"]``.
+
+What gets plotted
+-----------------
+
+.. code-block:: text
+
+   metrics/global/grad_norm                whole-model gradient L2 norm
+   metrics/global/weights_norm             whole-model parameter L2 norm
+   metrics/layer/<layer_id>/grad_norm      per-layer parameter gradients
+   metrics/layer/<layer_id>/weights_norm   per-layer parameters
+   metrics/layer/<layer_id>/activation_mean
+   metrics/layer/<layer_id>/activation_std
+   metrics/layer/<layer_id>/activation_max
+   metrics/layer/<layer_id>/activation_min
+
+Layers with parameters get all eight; parameter-free layers (``ReLU``,
+``MaxPool2d``) get the four activation curves only. Containers and shape-only
+ops (``Sequential``, ``Flatten``, ``Identity``, ``Dropout``) are skipped, since
+their output statistics duplicate the layer before them.
+
+For the model in this example — three conv blocks and a two-layer head — that
+is 74 curves: 14 layers × 4 activation stats, 8 parameterized layers × 2 norms,
+and the 2 global norms.
+
+The layer legend
+----------------
+
+``metrics/layer/7/grad_norm`` says nothing on its own, so the example prints the
+mapping at startup:
+
+.. code-block:: text
+
+    layer_id  module          shape
+           1  Conv2d          (16, 1, 3, 3)
+           2  BatchNorm2d     (16,)
+           3  ReLU            -
+           4  MaxPool2d       -
+           5  Conv2d          (32, 16, 3, 3)
+           6  BatchNorm2d     (32,)
+           7  ReLU            -
+           8  MaxPool2d       -
+           9  Conv2d          (64, 32, 3, 3)
+          10  BatchNorm2d     (64,)
+          11  ReLU            -
+          12  Flatten         -
+          13  Linear          (128, 3136)
+          14  ReLU            -
+          15  Linear          (10, 128)
+
+These are the same ids the model panel and every architecture op (freeze /
+reset / operate) use — so a curve that looks wrong names the layer you then act
+on, whether from the UI, the CLI, or the agent.
+
+Note that every module in this example's model is a **named attribute** rather
+than a member of an ``nn.Sequential``. That is deliberate: a Sequential block
+resolves to one layer id, and therefore one curve, which defeats the purpose of
+per-layer signals.
+
+Reading the curves
+------------------
+
+Fashion-MNIST is small enough to make each failure mode legible:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 34 66
+
+   * - What you see
+     - What it means
+   * - ``grad_norm`` collapsing toward 0 in the **early** layers while the late
+       ones stay healthy
+     - Vanishing gradient. The run keeps "training" and stops learning. Act
+       from the layer where it dies.
+   * - ``grad_norm`` spiking by orders of magnitude
+     - Exploding gradient. Compare against the loss curve to see which moved
+       first.
+   * - ``activation_std`` → 0 on a layer
+     - That layer has gone constant (dead ReLUs, saturated BatchNorm). Still
+       consuming compute, contributing nothing.
+   * - ``activation_min`` pinned at exactly 0.0 across a whole ReLU
+     - The same story from the other side — nothing is getting through.
+   * - ``weights_norm`` climbing without bound while the loss flattens
+     - The model is growing weights instead of learning structure. Add decay.
+
+Cost, and how it is kept low
+----------------------------
+
+Three things keep the per-step overhead small enough to leave on by default:
+
+- **Activations are reduced on-device** into 0-d tensors and held there. The
+  whole step costs *one* host↔device sync no matter how many layers are
+  tracked.
+- **Gradients are captured by post-accumulate hooks**, so nothing walks the
+  parameter list a second time — and nothing depends on where your loop calls
+  ``optimizer.zero_grad()``.
+- **``model_signals_every_n_steps``** samples every Nth step. On a large model,
+  10–50 makes the cost negligible while the curves stay just as readable. Reach
+  for this before dropping metrics.
+
+Collection only happens inside ``guard_training_context``, so the evaluation
+pass contributes nothing — a gradient or activation curve never contains values
+the optimizer did not see. This holds even for eval loops that skip
+``model.eval()`` or ``torch.no_grad()``.
+
+Custom dynamics values
+----------------------
+
+``track_model_signals`` is a collector over ``wl.save_model_signals``, which is
+the step-keyed write path in its own right. Use it directly for anything the
+collector does not compute:
+
+.. code-block:: python
+
+   # gradient-to-weight ratio: how big a step is this, relative to the weights?
+   wl.save_model_signals({
+       "metrics/global/update_ratio": grad_norm / (weight_norm + 1e-12),
+       "metrics/global/lr": optimizer.param_groups[0]["lr"],
+   })
+
+See :ref:`save_model_signals <model-signals>` for the full reference, and
+:doc:`../../model_interaction` for how these fit alongside the rest of the
+model surface.

docs/index.rst

@@ -129,31 +129,31 @@ Weightslab is a Python SDK to inspect, monitor, and edit training behavior for c
 
 .. toctree::
    :maxdepth: 2
-   :caption: Getting Started
+   :caption: GETTING STARTED
    :hidden:
 
    quickstart
 
 
 .. toctree::
    :maxdepth: 2
-   :caption: Usage
+   :caption: USAGE
    :hidden:
 
    usage/good_practice
 
 
 .. toctree::
    :maxdepth: 3
-   :caption: Examples
+   :caption: EXAMPLES
    :hidden:
 
    examples/index
 
 
 .. toctree::
    :maxdepth: 2
-   :caption: Core Concepts
+   :caption: CORE CONCEPTS
    :hidden:
 
    four_way_approach
@@ -171,7 +171,7 @@ Weightslab is a Python SDK to inspect, monitor, and edit training behavior for c
 
 .. toctree::
    :maxdepth: 2
-   :caption: External Library Integration
+   :caption: INTEGRATIONS
    :hidden:
 
    pytorch_lightning
@@ -180,17 +180,28 @@ Weightslab is a Python SDK to inspect, monitor, and edit training behavior for c
 
 .. toctree::
    :maxdepth: 1
-   :caption: Configuration
+   :caption: CONFIGURATION
    :hidden:
 
    configuration
 
 
 .. toctree::
    :maxdepth: 2
-   :caption: Reference
+   :caption: REFERENCE
    :hidden:
 
    user_functions
    user_commands
    grpc/index
+
+
+.. toctree::
+   :maxdepth: 2
+   :caption: MIGRATION
+   :hidden:
+
+   From Weights & Biases
+   From Voxel 51
+   From Tensorboard
+

docs/logger.rst

@@ -8,8 +8,68 @@ What gets logged
 
 - Scalar signals (losses, metrics)
 - Per-sample signal vectors
+- Per-step **model** signals (gradient/weight norms, activation statistics)
 - Optional predictions/targets for deeper analysis
 
+Two kinds of signal
+-------------------
+
+Signals divide by what a value is *about*, and that decides which verb records
+it:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 22 30 48
+
+   * - Keyed by
+     - Verb
+     - Example
+   * - Sample
+     - ``wl.save_signals``
+     - The classification loss of one image.
+   * - Annotation
+     - ``wl.save_instance_signals``
+     - The IoU of one bounding box.
+   * - Group
+     - ``wl.save_group_signals``
+     - A contrastive loss over an image pair.
+   * - **Step**
+     - ``wl.save_model_signals``
+     - The gradient norm of layer 5 at step 900.
+
+The first three write onto dataframe rows; the sample grid can then be sorted
+and filtered by them. The fourth does not — a gradient norm belongs to the
+optimization step that produced it, not to any of the samples in the batch, so
+it is plotted as a curve and nothing else. Recording it with ``save_signals``
+would mean broadcasting one number across a whole batch of ids and polluting
+every one of those samples' history with a value that was never about them.
+
+Default plot order
+------------------
+
+The plots board groups curves by signal-name prefix, in this order:
+
+1. **Your experiment's signals** — losses, metrics, and the whole-model
+   ``metrics/global/*`` norms. These are what the board is for, so they stay at
+   the top.
+2. **Per-layer model signals** — everything under ``metrics/layer/``
+   (see :ref:`track_model_signals <model-signals>`).
+3. **Resource monitors** — everything under ``resource/`` (CPU, memory, disk,
+   network, GPU and process telemetry).
+
+The grouping exists because arrival order stops being usable once model signals
+are on: ``track_model_signals`` can emit dozens of ``metrics/layer/*`` curves in
+a single step (74 for the Fashion-MNIST example) and resource monitoring is
+enabled by default, so an unordered board buries the loss curve under
+telemetry. Note that ``metrics/global/*`` deliberately sits in the *first*
+group — a whole-model gradient norm is read next to the loss, not scrolled past
+70 per-layer curves.
+
+This is only a default. Dragging a card puts it exactly where you drop it and
+that arrangement is remembered, per browser; signals that appear later (a
+``metrics/layer/*`` curve showing up once training starts) are filed into their
+group without disturbing anything you have already arranged.
+
 Start services
 --------------