Skip to content

python-guide.md

Latest commit

 

History

History
631 lines (476 loc) · 21.2 KB

python-guide.md

File metadata and controls

631 lines (476 loc) · 21.2 KB
Raw

Python Guide

This guide covers the full Python API for HOLA. For installation instructions, see Getting Started.

Overview

HOLA's Python API centers on the Study class, which can operate in two modes:

Mode Where the engine runs When to use it
Study(...) In your Python process (Rust engine loaded inside the interpreter) Notebooks, single-machine scripts, anything that should not depend on a server
Study.connect(url) In a running HOLA server (returns an HTTP client) Workers on other machines, language-agnostic workers, sharing one study across many processes

Both modes expose the same methods (ask, tell, top_k, …). You pick one based on process layout, not on different math.

The Python API exposes these classes:

Class Purpose
Study In-process engine. Pass Space and objectives here; also provides Study.connect(url) for remote.
Space Named parameter space builder
Trial A pending trial returned by ask(), with .trial_id and .params
CompletedTrial A completed trial with .trial_id, .params, .metrics, .scores, .score_vector, .rank, .pareto_front, .completed_at
Real Real-valued parameter with configurable scale (linear, log, log10)
Integer Integer parameter within an inclusive range
Categorical Choice from a list of string labels
Minimize Minimize an objective field
Maximize Maximize an objective field
Gmm GMM strategy configuration (refit cadence, elite fraction, exploration, and work limits)
Sobol Sobol strategy configuration
Random Random strategy configuration

All classes are imported from the hola_opt module.

from hola_opt import (
    Study, Space, Trial, CompletedTrial,
    Real, Integer, Categorical,
    Minimize, Maximize,
    Gmm, Sobol, Random,
)

Exceptions

HOLA exposes a small exception hierarchy so callers can distinguish failures without parsing message text:

Exception Meaning
HolaError Base class for errors raised by HOLA
ConfigurationError Invalid space, objective, strategy, study, URL, or timeout configuration
CheckpointError Checkpoint loading or saving failed
RemoteError Remote transport, HTTP status, response schema, or protocol failure
ObjectiveError Objective metrics have the wrong shape or violate the declared contract

All five classes subclass ValueError, so code written for earlier releases that catches ValueError remains compatible. Exceptions raised by the user's objective function itself are propagated unchanged, including their traceback.

from hola_opt import ConfigurationError, RemoteError, Study

try:
    remote = Study.connect("https://hola.example.com", request_timeout=30)
    trial = remote.ask()
except ConfigurationError as error:
    print(f"Invalid client configuration: {error}")
except RemoteError as error:
    print(f"Server request failed: {error}")

Defining Parameter Spaces

A Space is built by passing parameter builders as keyword arguments. The keyword names become the parameter names in trial dicts.

Real

A real-valued (floating-point) parameter with a configurable scale. The scale keyword argument accepts "linear" (default), "log", or "log10".

Linear scale (default). We sample values uniformly from $[\min, \max]$.

Space(temperature=Real(0.0, 2.0))

Log scale. For values that span orders of magnitude, we sample uniformly in $\ln$ space. Both bounds must be positive.

Space(lr=Real(1e-4, 0.1, scale="log"))

Log10 scale. Similar to log but uses $\log_{10}$ internally. Both bounds must be positive.

Space(lr=Real(1e-4, 0.1, scale="log10"))

Real(min, max, scale="linear"): min and max are specified in actual values (not exponents), regardless of scale. Internally, HOLA samples uniformly in the chosen scale's transformed space.

Integer

An integer parameter within an inclusive range.

Space(layers=Integer(1, 10))

Integer(min, max): values are integers from min to max, inclusive.

Categorical

A parameter that chooses from a fixed set of string labels.

Space(optimizer=Categorical(["adam", "sgd", "rmsprop"]))

Categorical(choices): choices is a list of strings. The selected label is returned as a string in trial params.

Mixed Spaces

Combine any parameter types in a single space.

space = Space(
    lr=Real(1e-4, 0.1, scale="log10"),
    layers=Integer(1, 10),
    dropout=Real(0.0, 0.5),
    optimizer=Categorical(["adam", "sgd", "rmsprop", "adamw"]),
)

Defining Objectives

Objectives tell HOLA which fields in your metrics dict to optimize and in which direction.

Single Objective

objectives = [Minimize("loss")]

Your objective function must return a dict containing the field name (here "loss").

Maximize

objectives=[Maximize("accuracy")]

Internally, maximization is converted to minimization by negating the value.

Multi-Objective

Pass multiple objectives to optimize several metrics simultaneously.

objectives=[
    Minimize("error"),
    Minimize("latency"),
]

Because group is omitted here, each field becomes its own priority group and the leaderboard uses Pareto/NSGA-II ranking over the two group costs. HOLA sums priority-weighted objective contributions only within one shared group. To request scalar ranking for several fields, give them the same group label.

Target-Limit-Priority (TLP) Objectives

For fine-grained control, use target, limit, and priority.

objectives=[
    Minimize("loss", target=0.0, limit=1.0, priority=1.0),
    Minimize("latency", target=100, limit=500, priority=0.5),
]
  • target. The "good enough" value. Trials at or better than target score 0 for this objective.
  • limit. The worst acceptable boundary. At the limit, an objective scores priority; crossing beyond it makes the trial infeasible and scores infinity.
  • priority. The objective's score at the limit and its relative weight within a group ($P_i$). The linear segment's slope is $P_i / (\text{limit} - \text{target})$; priority is not itself a slope.

The TLP formula is $\varphi_i = P_i \times (\text{value} - \text{target}) / (\text{limit} - \text{target})$.

Between target and limit, the score is interpolated linearly and scaled by priority. See Concepts: TLP Scalarization for the full explanation.

Priority Groups

To control Pareto axes explicitly, assign objectives to groups using the group parameter. Objectives in the same group are summed into a single group cost; distinct groups form the axes of the Pareto ranking.

objectives=[
    Minimize("error", target=0.05, limit=0.5, priority=1.0, group="quality"),
    Minimize("calibration", target=0.01, limit=0.1, priority=0.5, group="quality"),
    Minimize("latency", target=20, limit=100, priority=1.0, group="cost"),
]

Here, "error" and "calibration" share the "quality" group; their TLP scores are summed into a single quality cost. The "latency" objective forms its own "cost" group. The Pareto front is then computed over the two group axes (quality, cost).

When group is omitted, each objective defaults to its own group (keyed by field name). A study with a single group uses scalar ranking; multiple groups enable Pareto front via study.pareto_front().

Pass these objective lists to the Study constructor as the objectives parameter, as shown in the next section.

Creating a Study

study = Study(
    space=Space(x=Real(0.0, 1.0)),
    objectives=[Minimize("loss")],
    strategy="gmm",  # default
    seed=42,           # optional: for reproducible runs
)

Parameters

Parameter Type Default Description
space Space required The parameter space to search
objectives list required List of Minimize / Maximize objectives (at least one)
strategy str or strategy class "gmm" Search strategy. Pass a string ("gmm", "sobol", "random") for defaults, or a configuration class (Gmm(...), Sobol(), Random()) for fine-grained control.
seed int or None None Random seed for reproducibility. When set, the same seed produces the same candidate sequence.
max_trials int or None None Maximum number of trials. When set, ask() raises after this many trials have been dispatched.

The Ask/Tell Loop

The core optimization loop has two steps:

  1. Ask: get the next trial to evaluate.
  2. Tell: report the result.
for i in range(100):
    trial = study.ask()                    # Trial with .trial_id and .params
    metrics = my_function(trial.params)    # Your evaluation code
    study.tell(trial.trial_id, metrics)    # Report results

study.ask() -> Trial

Returns a Trial object with:

  • trial.trial_id: a unique integer identifier (monotonically increasing, starting from 0).
  • trial.params: a dict mapping parameter names to values.
trial = study.ask()
print(trial)           # Trial(trial_id=0, params={'x': 0.4321, 'layers': 5})
print(trial.trial_id)  # 0
print(trial.params)    # {'x': 0.4321, 'layers': 5}

study.tell(trial_id, metrics) -> CompletedTrial

Reports the result of a trial. metrics must be a dict containing at least the fields specified in your objectives. Returns a CompletedTrial.

completed = study.tell(trial.trial_id, {"loss": 0.42, "accuracy": 0.91})
print(completed.score_vector)  # scalarized score
print(completed.metrics)       # {"loss": 0.42, "accuracy": 0.91}

Extra fields beyond what your objectives require are stored in the trial as metrics and can be inspected later.

!!! note For infeasible trials (where a metric exceeds its TLP limit), the corresponding entries in .scores and .score_vector are float('inf'). You can check for this with math.isinf().

!!! warning Each trial ID can only be told once. Calling tell with the same ID twice raises a ValueError.

The run() Convenience Method

For simple workflows, study.run() automates the ask/tell loop.

study = Study(
    space=Space(x=Real(0.0, 1.0)),
    objectives=[Minimize("loss")],
)

def objective(params):
    return {"loss": train_model(params)}

study.run(objective, n_trials=100)

Parameters

Parameter Type Default Description
func callable required Function that takes a params dict and returns a metrics dict
n_trials int required Number of trials to run
n_workers int 1 Parallel workers: <=1 = sequential, N = N parallel threads

run() returns self, so you can chain.

best = study.run(objective, n_trials=100).top_k(1)[0]

Parallel Evaluation

With n_workers > 1, run() dispatches trials concurrently using Python's ThreadPoolExecutor. It keeps at most n_workers evaluations in flight, processes each result as soon as that evaluation finishes, and immediately replenishes the free slot. A slow early trial therefore does not hold up faster later results, and exceptions cancel any still-pending trials before the pool is shut down.

# Use 4 parallel workers
study.run(objective, n_trials=100, n_workers=4)

# Sequential (no thread pool overhead)
study.run(objective, n_trials=100, n_workers=1)

Inspecting Results

All index fields (trial_id, rank, pareto_front) are 0-indexed.

study.top_k(k) -> list[CompletedTrial]

Returns the top k trials found so far, as a list of CompletedTrial objects. Returns an empty list if no trials have been completed.

top = study.top_k(1)
if top:
    best = top[0]
    print(best.score_vector)  # scalarized score
    print(best.params)        # {"x": 0.73}
    print(best.trial_id)      # 17
    print(best.metrics)       # original metrics dict
    print(best.scores)        # per-objective scores
    print(best.rank)          # rank in leaderboard
    print(best.completed_at)  # completion timestamp

study.trial_count() -> int

Returns the number of completed trials.

print(f"Completed {study.trial_count()} trials")

study.trials(sorted_by="index", include_infeasible=True) -> list[CompletedTrial]

Returns all trials as CompletedTrial objects. Each has .trial_id, .params, .score_vector, .scores, .metrics, .rank, .pareto_front, and .completed_at. Useful for plotting convergence traces or custom analysis.

# Compute running-best convergence trace
import math

best_so_far = float("inf")
trace = []
for trial in study.trials():
    sv = trial.score_vector  # dict of {objective group: scalarized score}
    obs = sum(sv.values()) if sv else float("inf")
    if math.isfinite(obs):
        best_so_far = min(best_so_far, obs)
    trace.append(best_so_far)

study.pareto_front(front=0, include_infeasible=False) -> list[CompletedTrial]

Returns the Pareto front (non-dominated trials) for multi-objective studies, specifically those with objectives assigned to distinct groups. Each element is a CompletedTrial with .trial_id, .params, .scores, .metrics, etc. The front parameter is 0-indexed: front=0 returns the first (best) Pareto front, front=1 returns the second front, and so on. The .pareto_front field on each CompletedTrial is also 0-indexed.

study = Study(
    space=Space(x=Real(0.0, 1.0)),
    objectives=[
        Minimize("loss", target=0.0, limit=5.0, priority=1.0, group="quality"),
        Minimize("latency", target=0.0, limit=100.0, priority=1.0, group="cost"),
    ],
    seed=42,
)
study.run(objective, n_trials=200, n_workers=1)

for trial in study.pareto_front():
    print(trial.scores)  # {"loss": 0.3, "latency": 42.0}

Returns an empty list for single-group (scalar) studies.

Choosing a Strategy

Pass a string shortcut for defaults, or a strategy configuration class for fine-grained control.

# String shortcut (default settings)
Study(strategy="gmm", ...)

# Configuration class (custom settings)
Study(strategy=Gmm(refit_interval=10, elite_fraction=0.1), ...)

GMM (default)

Gaussian Mixture Model strategy. Uses Sobol exploration followed by GMM exploitation. Refits a GMM to the top elite_fraction (default 25%) of trials every refit_interval (default 20) completed trials. With multiple objective groups, elites are ordered by non-domination rank and then descending crowding distance. The exploration budget counts issued ask suggestions, including pending trials. If concurrent asks reach that boundary before any empirical fit is installed, HOLA continues the Sobol' sequence rather than sampling the uninformed GMM prior. Uses the HOLA algorithm. GMM exploitation uses seeded Owen-scrambled Gauss–Sobol' points: one Sobol' coordinate selects the component, and inverse-normal coordinates sample within it. Each successfully installed GMM starts a new epoch-specific scramble at its first point.

  • Best for larger budgets (50+ trials) where exploration can transition to exploitation
  • Concentrates samples in promising regions after warmup
# Default GMM - equivalent to strategy="gmm"
Study(strategy=Gmm(), ...)

# Customized: refit more often, use top 10% of trials
Study(strategy=Gmm(refit_interval=10, elite_fraction=0.1), ...)

Gmm parameters

Parameter Type Default Description
refit_interval int or None 20 How often the GMM is refit, in completed trials
elite_fraction float or None 0.25 Fraction of top trials used for refitting. Must be in (0, 1].
exploration_budget int or None auto Number of issued Sobol exploration suggestions before GMM exploitation begins. Pending asks count against this budget. When omitted, computed automatically from the total budget and number of dimensions.
ongoing_exploration_period int or None 5 Continue global Sobol' exploration every Nth post-warmup suggestion. Use 0 to disable; explicit periods must be at least 2.
max_components int or None 3 Maximum fitted mixture components. The effective count can be lower when the elite set is small.
min_elite_samples int or None 1 Minimum feasible elite workset required before fitting. Must not exceed max_refit_samples.
max_refit_samples int or None 4096 Maximum elite samples used by one GMM fit. Must be at least 1.
max_refit_candidates int or None 16384 Maximum retained trials ranked during elite selection. Must be at least max_refit_samples; longer histories use deterministic stratified coverage.

Sobol

Owen-scrambled Sobol sequences provide quasi-random sampling with better coverage than pure random. Good for initial exploration and moderate-budget optimizations.

  • Deterministic given a seed
  • Fills the space more evenly than random sampling
  • Works well for up to ~100--200 trials in moderate dimensions
Study(strategy="sobol", ...)   # or
Study(strategy=Sobol(), ...)

Random

Uniform pseudo-random sampling. A simple baseline.

  • Deterministic given a seed
  • No spatial structure; samples are independent.
Study(strategy="random", ...)   # or
Study(strategy=Random(), ...)

Going Distributed

Hosting a Server

You can start a REST server directly from a local Study, making it accessible to remote workers.

study = Study(space=space, objectives=objectives)

# Blocking - serves until interrupted (Ctrl+C)
study.serve(port=8000)

# Background - serves in a background thread, study remains usable
study.serve(port=8000, background=True)
study.run(objective, n_trials=100)  # runs locally while server is active
Parameter Type Default Description
port int 8000 TCP port to listen on
background bool False If True, runs in a background thread and returns immediately
dashboard_path str or None None Path to a dashboard directory to serve the bundled UI. When omitted, no dashboard is served. Use str(dashboard_dir()) to serve the bundled dashboard.

When background=True, the study continues to work locally. Both local calls and remote HTTP requests share the same engine state, so trials from any source appear in the same leaderboard.

Study.connect()

Connect to a running HOLA server (started via study.serve(), hola serve, or any other means) using Study.connect(). The returned object exposes the same methods as a local Study, but forwards all calls as HTTP requests. The server holds the leaderboard and strategy state.

from hola_opt import Study

remote = Study.connect("http://localhost:8000")

# The same ask/tell/top_k interface as Study
trial = remote.ask()
remote.tell(trial.trial_id, {"loss": 0.42})
top = remote.top_k(1)

# Convenience method - automates the ask/tell loop
remote.run(my_function, n_trials=100, n_workers=4)

# Inspect results
print(remote.trial_count())    # number of completed trials
for t in remote.trials():      # all trials in insertion order
    print(t.trial_id, t.score_vector)

# Multi-objective: Pareto front
for t in remote.pareto_front():
    print(t.scores)

Remote requests use a 10-second connection timeout and a 30-second whole-request timeout by default. Both are configurable, and a bearer token is sent with every endpoint when provided:

import os

remote = Study.connect(
    "https://hola.example.com",
    token=os.environ["HOLA_TOKEN"],
    connect_timeout=5.0,
    request_timeout=60.0,
)

Switching from local to distributed is mostly replacing Study(...) with Study.connect("http://...") (you no longer pass space / objectives here, since the server already has them configured). All inspection methods (top_k(), trial_count(), trials(), pareto_front(), and run()) work on both modes. See the Overview for a comparison of the two modes.

For the wire format, see the REST API Reference.

Examples

The hola-py/examples/ directory contains complete runnable examples:

Example Description
basic_optimization.py Minimizes 1D Forrester and 2D Branin functions. Shows both study.run() and the manual ask/tell loop.
categorical_demo.py Mixed space with Categorical, Real (log10 scale), and Integer parameters. Simulates an optimizer hyperparameter search.
gmm_explore_exploit.py Compares Sobol vs GMM strategies on Branin and Rastrigin. Shows how GMM concentrates samples after warmup.
ml_hyperparameters.py Tunes a scikit-learn GradientBoostingRegressor with Real, log-scale Real, Integer, and Categorical parameters. Requires scikit-learn.
multi_objective.py Optimizes error vs latency with TLP scoring and priority groups. Demonstrates Pareto-front optimization via study.pareto_front().

Run an example.

uv run --directory hola-py python examples/basic_optimization.py

Run that command from the repository root. The --directory option selects the hola-py project and its virtual environment explicitly.