[NEW FEATURE] Goodness-of-fit outlier rejection

[jeipollack]

Problem Description
Poor-quality observations, contaminated stars, or samples that are not well represented by the WaveDiff model can adversely affect model training and degrade PSF reconstruction performance.

Given a previously-trained WaveDiff model, evaluate the reconstruction quality of each observation to identify observations with poor reconstruction quality, so they can be excluded from subsequent training iterations. This provides a data-driven mechanism for improving the quality of the training dataset beyond catalogue-based star selection.

Proposed Solution
Implement a configurable goodness-of-fit evaluation as part of the data preprocessing workflow.
The feature should:

• evaluate each star using a previously trained WaveDiff PSF model;
• compute one or more configurable goodness-of-fit metrics (e.g. reduced χ² or residual-based statistics);
• classify observations as acceptable or outliers using configurable thresholds;
• exclude rejected observations from downstream training (and optionally validation);
• generate summary statistics describing the rejected sample.

The design should remain extensible so that additional goodness-of-fit metrics can be introduced without modifying the overall workflow.

Alternatives Considered

• No automated filtering — retain all selected stars regardless of reconstruction quality, relying entirely on the optimisation process to handle poor observations.
• Manual inspection of residuals — useful during development but not suitable for automated pipeline execution.
• Fixed quality cuts from catalogue metadata — simple but unable to detect stars that appear valid initially yet exhibit poor agreement with the trained model.

Additional Context
This feature is one component of the Outlier Rejection milestone.
Unlike pixel mask obscuration, which identifies potential data quality issues directly from the observations, this method uses the performance of a trained WaveDiff model to identify observations that are inconsistent with the learned PSF representation.

The intended workflow is:

1. Load a trained WaveDiff model.
2. Evaluate goodness-of-fit for observation.
3. Reject observations exceeding configurable thresholds.
4. Train the model using the filtered dataset.
5. Compare model performance before and after outlier rejection.

Tasks-to-complete

• Define goodness-of-fit metric(s)
• Implement metric computation
• Add configurable threshold(s)
• Integrate with preprocessing workflow
• Generate rejection statistics
• Add unit tests
• Update documentation

[jeipollack]

after more thought, the configuration settings for this metric extends beyond just thresholds. We have to indicate which model to use for inferring the PSFs of the new dataset. The configuration pattern could be something like:

outlier_rejection:

  goodness_of_fit:
    enabled: true

    model:
      inference_config: inference_config.yaml

    metric:
      type: reduced_chi_square

    threshold: 3.0

For threshold I made up sth as a placeholder. it can be updated with whatever you implemented in your PR or new ideas.

[jeipollack]

hi @tobias-liaudat , for inferring the PSFs for outlier rejection using chi-square do we want the inference configuration to apply centroid corrections? see below:

inference:
  # Inference batch size
  batch_size: 16

  # Cycle to use for inference. Can be: 1, 2, ...
  cycle: 2

  # Dataset schema used to interpret input fields and control downstream conversion.
  # This does NOT modify dataset content; it only controls how fields are interpreted during conversion.
  # This affects required fields and available conversion handlers.
  #
  # Choices:
  # - INFERENCE: standard inference mode (expects positions and seds)
  # - EVALUATION: evaluation mode (may include additional fields such as sources or masks)
  schema_mode: INFERENCE

  # Paths to the configuration files and trained model directory
  configs:
    # Path to the directory containing the trained model
    trained_model_path: /path/to/trained/model/

    # Subdirectory name of the trained model, e.g. psf_model
    model_subdir: model

    # Relative Path to the training configuration file used to train the model
    trained_model_config_path: config/training_config.yaml

    # Path to the data config file (this could contain prior information)
    data_config_path:

  # The following parameters will overwrite the `model_params` in the training config file.
  model_params:
    # Num of wavelength bins to reconstruct polychromatic objects.
    n_bins_lambda: 8 (actual number of lambda bins)

    # Downsampling rate to match the oversampled model to the specified telescope's sampling.
    output_Q: 1

    # Dimension of the pixel PSF postage stamp
    output_dim: 32

    # Flag to perform centroid error correction
    correct_centroids: False

    # Flag to perform CCD misalignment error correction
    add_ccd_misalignments: True