Skip to content

fix: Do not load the native extension to build an empty rules list - #342

Open
leongdl wants to merge 1 commit into
OpenJobDescription:mainlinefrom
leongdl:expr-extension-load-only-when-used
Open

fix: Do not load the native extension to build an empty rules list#342
leongdl wants to merge 1 commit into
OpenJobDescription:mainlinefrom
leongdl:expr-extension-load-only-when-used

Conversation

@leongdl

@leongdl leongdl commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

The gap #341 left

#341 stopped import openjd.sessions from loading the native
openjd._openjd_rs extension. But Session.__init__ calls
_build_expr_host_rules() unconditionally, and that imported openjd.expr
a facade over the extension — before checking whether there was anything to
convert. With no path mapping rules it imported the engine, iterated an empty
list, and returned [].

So a worker that never evaluates an EXPR expression still loaded the extension on
its first session. 49d1804 moved the load from import time to
first-session time rather than eliminating it. Caught by a reviewer reading the
call site; confirmed by measurement:

import openjd.sessions              -> rs_loaded=False   (#341 holds)
Session(...)  no rules, no EXPR     -> rs_loaded=True     _expr_host_rules == []

The fix

Return [] directly when there are no rules — byte-identical to what the loop
produced.

It must stay [] and not None: the session is still host scope, so
apply_path_mapping() has to remain available with an empty rule set. Seeding
[] is free — SymbolTable stores expr_host_rules untouched as
Optional[list[Any]] and only crosses into Rust at evaluation time in
_expr_support.symtab_to_expr_values. Verified that assigning None, assigning
[], and running a full non-EXPR resolve all leave the extension unloaded.

Audited every load path, not just this one

Rather than grepping for imports, I installed a sys.meta_path finder that
records the stack at the moment openjd._openjd_rs is first imported, and ran
each flow in its own fresh interpreter.

Flow Before After Expected
import openjd.sessions pure pure pure
Session() no rules LEAK pure pure
Session(path_mapping_rules=None) LEAK pure pure
non-EXPR run_task end to end LEAK pure pure
non-EXPR enter_environment LEAK pure pure
non-EXPR argument resolution pure pure pure
non-EXPR optional-int resolution pure pure pure
Session(path_mapping_rules=[rule]) loads loads loads — see below
EXPR evaluation loads loads loads
openjd.sessions._v1 loads loads loads

All four leaks traced to the same single frame:

openjd/expr/__init__.py <- _session.py:1657 in _build_expr_host_rules <- _session.py:476 in __init__

After the fix every non-EXPR flow is clean and the only remaining loads are the
three that should load.

Known limitation, asserted rather than implied

A session with real path mapping rules still loads the extension, because the
rules must be converted into openjd.expr.PathMappingRule engine objects to seed
host context. openjd-sessions cannot avoid that alone — closing it needs
openjd-model to accept unconverted rules and convert them at the Rust boundary it
already crosses.

test_path_mapping_rules_do_load_the_extension pins the current behaviour so a
future fix is noticed here instead of passing silently.

This matters in practice: a worker with path mapping rules configured — the
common case — still loads the extension on its first session. This PR fixes the
no-rules case, which covers the CLI default and most tests, and narrows the
remaining exposure to one well-understood site. It does not deliver "a non-EXPR
worker never loads the extension" on its own.

Correcting my own earlier audit

When I audited crossings for #341 I classified this site as "already
function-local (lazy)" and moved on. Lazy is not conditional. A
function-local import still runs unconditionally the moment its function is
called, and this one is called from __init__. The existing purity tests covered
import time plus one runtime path, so nothing would have caught this.

Testing

Mutation-checked: 4 mutants, 0 survivors. Neutralising the fast path,
returning None instead of [], firing it for every session, and inverting its
condition are each caught by a named test.

913 passed, 39 skipped, 16 xfailed
ruff clean · black clean · mypy clean · mypy --platform win32 clean

Session.__init__ calls _build_expr_host_rules() unconditionally, and that
imported openjd.expr -- a facade over the native openjd._openjd_rs
extension -- before checking whether there was anything to convert. With no
path mapping rules it imported the engine, iterated an empty list, and
returned []. So a worker that never evaluates an EXPR expression still
loaded the extension on its FIRST SESSION: 49d1804 moved the load from
import time to first-session time rather than eliminating it. Reported by
a reviewer reading the call site; confirmed by measurement.

Return [] directly when there are no rules. That is byte-identical to what
the loop produced, and it must stay [] rather than None because the session
is still host scope and apply_path_mapping() has to remain available with an
empty rule set. Seeding [] is free: SymbolTable stores expr_host_rules
untouched as Optional[list[Any]] and only crosses into Rust at evaluation
time, in _expr_support.symtab_to_expr_values -- verified that assigning
None, [], and running a full non-EXPR resolve all leave the extension
unloaded.

Audited every load path rather than grepping for imports, using a
sys.meta_path finder that records the stack at the moment
openjd._openjd_rs is first imported, one flow per fresh interpreter.
Before: four flows leaked -- Session() with no rules, with rules=None, a
non-EXPR run_task, and a non-EXPR enter_environment -- and all four traced
to the same single frame, _session.py:476 __init__ ->
_build_expr_host_rules. After: every non-EXPR flow is clean, and the only
remaining loads are the three that should load. Import of openjd.sessions,
non-EXPR argument resolution, and non-EXPR optional-int resolution were
already clean and stayed clean.

Known limitation, asserted rather than left implicit: a session with real
path mapping rules still loads the extension, because the rules must be
converted into openjd.expr.PathMappingRule engine objects to seed host
context. openjd-sessions cannot avoid that alone; closing it needs
openjd-model to accept unconverted rules and convert them at the Rust
boundary it already crosses. test_path_mapping_rules_do_load_the_extension
pins the current behaviour so that a future fix is noticed here instead of
passing silently. This matters in practice: a worker with path mapping
rules configured -- the common case -- still loads the extension.

Correcting my own earlier audit: I classified this site as "already
function-local (lazy)" and moved on. Lazy is not conditional. A
function-local import still runs unconditionally the moment its function is
called, and this one is called from __init__. The existing purity tests
covered import time plus one runtime path, so nothing would have caught it.

Mutation-checked, 4 mutants, 0 survivors: neutralising the fast path,
returning None instead of [], firing it for every session, and inverting
its condition are each caught by name.

Verified: 913 passed / 39 skipped / 16 xfailed; ruff, black, mypy native
and mypy --platform win32 all clean.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
@leongdl
leongdl requested a review from a team as a code owner July 30, 2026 22:46
@wyongzhi

Copy link
Copy Markdown

Might be a parity gap, related to this area (not caused by this PR): rules added after construction are honored by Rust but not Python v0. Rust's extend_path_mapping_rules rebuilds the expr library (session.rs:759); v0 has no extend API, so callers append to _path_mapping_rules directly — live consumers see the new rules, but _expr_host_rules (built once at _session.py:458) doesn't, so apply_path_mapping() silently resolves against stale rules. Job-attachment style workflows always add rules post-construction, so this will bite once EXPR templates use them together. Not sure how you think about it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants