feat: expose the PST as typed nodes via policies_to_pst

[h0rv]

Rebased on current main. Nine commits, all part of this change.

The last five are a response to @skuenzli's review, and I have answered the questions on the PR itself. Short version: the two node types nothing could produce are gone, entity_uids no longer reports an empty result for input it cannot walk, the round trip he asked about is here, and there is now a test that fails CI when cedar-policy adds a PST variant.

What it adds

policies_to_pst(policies) -> cedarpy.pst.PolicySet. It parses Cedar policy text and hands back the PST as typed Python objects, one frozen dataclass per cedar_policy::pst node kind. You match on real types instead of digging through string keys:

match clause.expr:
    case BinaryOp(op="eq", left=GetAttr(base=Var(name="resource"), attr="status")):
        ...

Rust builds the objects directly. It walks pst::Expr once and calls the matching Python constructor for each variant, with the child nodes already built. There is no JSON step and no second walker.

PolicySet.from_pst(nodes) and PolicySet.to_pst() are the other direction, so a set can be taken apart as nodes, changed, and handed back to the engine:

nodes = policies_to_pst('permit(principal == User::"alice", action, resource);')
policy = nodes.static_policies["policy0"]
retargeted = dataclasses.replace(policy, principal=ScopeEq(EntityUid(EntityType("User"), "bob")))
edited = dataclasses.replace(nodes, static_policies={"policy0": retargeted})

PolicySet.from_pst(edited)     # authorizes for bob, not alice

Reading back needs no crate internals: cedar_policy::Policy::from_pst, Template::from_pst and PolicySet::from_pst are all public in 4.12. The enums are #[non_exhaustive], which blocks an exhaustive match from outside the crate but not constructing a variant that exists, so the reader builds pst::Expr values directly and lets the engine's own from_pst do the validating. policies_to_pst(text) is now just PolicySet.from_str(text).to_pst().

Why typed objects instead of JSON

Cedar already checked all of this on the Rust side. A clause cannot hold a slot. A clause cannot hold an unknown. Every expression is well formed. Handing back a JSON string throws those guarantees away and makes every caller re-check them by matching on strings, usually getting it wrong. A typed object carries the guarantee in the type, so there is nothing left to re-check.

The same idea drives the rest of the design: anything Cedar models as a closed set stays a closed set in Python, so a type checker can catch a mistake instead of a caller finding it at runtime.

Operator names are closed. UnaryOp.op and BinaryOp.op are Literal aliases listing every name Rust can emit (UnaryOpName, 18 names; BinaryOpName, 24). Template.effect, Var.name, and Slot.name are the same. So a match over one can be proven exhaustive with assert_never, and a typo like op == "equals" is a type error rather than a branch that never runs.

Bool and Long literals are separate nodes. bool is a subclass of int in Python, so one node holding either could not tell Cedar's Bool from its Long: isinstance(v, int) is true for True. They are BoolLit and LongLit, alongside StringLit and EntityLit.

Entity type names keep their namespace. EntityType(basename="User", namespace=("MyApp",)) instead of the string "MyApp::User", so no caller has to split on :: to get the namespace back. str() gives the Cedar form.

Every collection is parameterized. tuple[Expr, ...], tuple[Clause, ...], and so on. A bare tuple is tuple[Any, ...], which means the union evaporates as soon as you index into it.

HasAttr.attrs rejects an empty path, which is the NonEmpty<SmolStr> invariant Rust holds.

Nodes are values. Frozen, slotted (a policy set is many small objects, and there is no reason for each to carry a __dict__), and hashable, so they work as dict keys and set members. Mapping fields are declared Mapping, which has no mutating methods, and hold a FrozenMap at runtime, which is a dict subclass that raises on mutation. So a write to record.fields fails both in the type checker and at runtime, while dataclasses.asdict(), json.dumps(), and comparing against a plain dict all still work.

pst.entity_uids(node) returns every entity uid named anywhere under a node, at any depth, which is what you need in order to decide which entities to load before evaluating. It takes a node, or a mapping or tuple of nodes such as a PolicySet's templates. Anything else raises TypeError, rather than returning an empty frozenset that reads as "this policy names no entities". That mattered: passing the engine's own PolicySet handle instead of a pst.PolicySet used to silently report nothing.

__all__ declares the module's public surface.

Keeping up with cedar-policy

Every pst enum is #[non_exhaustive], so no exhaustive match is possible from outside the crate and the compiler cannot tell us when a variant is added. Two things stand in for that.

tests/unit/test_pst_variant_coverage.py reads the variant names, and the cargo feature each one needs, out of the cedar-policy-core source that Cargo.lock resolves to, and compares them against pst_variants.json in the repo. A bump that adds, removes or renames a variant fails and names it. It is feature-aware, because which variants exist depends on what Cargo.toml turns on: ResidualError needs tpe, VariadicOp needs variadic-is-in-range. The modelled side is read off cedarpy.pst.Expr rather than listed in the test, so a node type added to the union without a matching Rust variant fails too. Regenerate with python tests/unit/test_pst_variant_coverage.py after deciding what to do about the new variant.

The corpus suite also runs all 7,600 fuzzer-generated policies through policies_to_pst and back through from_pst. That catches a regression in what is already modelled, though not a new variant, since a new variant needs syntax the existing corpus does not contain. Today 7,551 convert and rebuild into an identical node tree. The other 49 are cedar's own PST construction rejecting extension functions called with the wrong arity (lessThan with 5 arguments, decimal with 4, and so on); none is the "unrepresentable variant" error this code raises for a node kind it does not model, which is what the test actually asserts.

What this does not cover

Static policies and unlinked templates only. A residual from is_authorized_partial cannot be represented this way. pst::Template's clauses are pub(crate) and can only be added through methods that validate each one, and that validation rejects any clause holding an unresolved unknown(...) node. Every non-trivial residual from is_authorized_partial has one. Confirmed by calling to_pst() on real residuals for a permit and a forbid; both raised.

For the same reason Unknown is not a node type here any more, and neither is ResidualError: that variant is behind cedar's tpe feature, which this PR does not enable, so it could not occur at all. ResidualError arrives in #109, where the feature is on and TPE really does emit it.

is_authorized_partial and residuals are untouched.

Notes for review

pst::Expr and several of its operator enums are #[non_exhaustive] upstream, so the Rust match needs a wildcard arm no matter what. That arm raises ValueError naming the variant it could not handle. It should never fire on this cedar-policy version. It is there so a future engine bump that adds a variant fails loudly instead of quietly building the wrong thing, and the audit above is what turns that into a CI failure rather than a runtime surprise.

No Cargo change is needed. There is no pst cargo feature: cedar_policy::pst is re-exported ungated in 4.12. Adding features = ["pst"] fails to resolve, and cargo helpfully points at tpe as the similar name, which is #109's feature.

I looked at pyo3 complex enums first (a #[pyclass] on a Rust enum with struct variants). They work, including the recursive case with Py<T> fields, and pyo3 generates __match_args__ so both keyword and positional patterns match. I did not use them: calling a plain Python constructor from Rust gives real dataclasses with no intermediate representation, and asdict / json.dumps come for free. A pyo3 complex enum would need a hand-written dict conversion instead.

One thing worth knowing if you touch the mapping fields: types.MappingProxyType cannot be used here, because dataclasses.asdict() calls copy.deepcopy and a mappingproxy cannot be pickled. That is why FrozenMap subclasses dict rather than wrapping one.

The audit test decodes cargo metadata as UTF-8 explicitly rather than using subprocess text=True. Cargo's metadata is UTF-8 and is not ASCII-only, and text=True decodes with the locale codec, which fails partway through on a Windows runner. That is a real failure I hit on CI here, not a precaution.

Testing

Built with maturin develop on macOS arm64, Python 3.12.13.

Check	Result
pytest tests/unit	269 passed, 14 subtests
pytest tests/integration/test_cedar_integration_tests.py	78 passed
pytest tests/integration/test_cedar_corpus_tests.py	60801 passed
mypy --strict cedarpy/pst.py	clean
cargo clippy --all-targets	10 warnings, the same 10 as on main, none in this diff

Tests cover every clause and scope constraint shape, every literal kind, set and record, has (including the multi-attribute path, checked against real parser output: resource has a.b lowers to two chained single-attribute nodes, not one node holding two attrs), like, is / is in, if / then / else, operators, a template with a slot, keyword and positional pattern matching, asdict / json.dumps, and error handling. Plus the namespaced entity type, BoolLit against LongLit, every emitted operator name being present in its Literal alias, mapping fields refusing mutation, nodes working as dict keys, the empty HasAttr path being rejected, and entity_uids across scope, conditions, sets, records and template links.

The new ones cover the round trip (static policies, a template with slots, a linked template, an edited node tree that authorizes differently afterwards, and both error paths), entity_uids refusing what it cannot walk, and Unknown being rejected by cedar rather than modelled here.

I checked the typing claims with mypy rather than assuming them. An exhaustive match over Expr ending in assert_never passes, and each of these is an error: op="nope", Var("principle"), EntityUid(type="User", ...), a list where a tuple belongs, e.op == "equals", and writing to a mapping field.

Not run: make benchmark-compare. The repo's own guidance says that comparison is load-sensitive and the machine was busy.

[skuenzli]

Hi @h0rv - thank you for the PR!

I've spent quite a while working to understand this change and I have several questions:

1. Do you consider this feature addition complete? Both this PR and feat: expose type-aware partial evaluation (TPE) as tpe_authorize #109 changed significantly during my review.

2. How are you using this new API in your own application(s)? I'm trying to understand the use case and how broad it is.

3. What is the maintenance process for adding coverage to the PST entity types?

4. How should cedar-py verify coverage of the PST types on an ongoing basis? One thought was to run some portion of the corpus test policies or cedar-policy integration test policies through policies_to_pst to verify they parse.

5. Should we add cedarpy.PolicySet#from_pst so that we can initialize a PolicySet from a pst? This would enable round-tripping policies. That would enable testing and other things like surgical edits, I'm guessing.

6. I'm confused about residuals. Can you please help me understand how entity_uids is meant to be used with them? The commit message says residuals are the motivation, but the PR body says residuals can't convert to a PST at all (pst::Template rejects unknown(...)), and the tests only ever call entity_uids on static policies and templates. What do you actually call it on in your application?

[h0rv]

All six are handled in code. #108 is now 1c59266 and #109 is 60ae834.

1. Yes, complete. Sorry about the churn. I marked these ready and then pushed the typing rework and two rebases the same day, which is a bad thing to review against. The heads are stable now.

2. We read policies as data rather than as text. policies_to_json_str already allowed that, but it meant walking nested dicts keyed by strings, and nothing tells you when you have missed a node kind. With typed nodes a type checker proves the match is exhaustive.

3 and 4. Every pst enum is #[non_exhaustive], so an exhaustive match is impossible from outside the crate and the compiler can never report a new variant. I added tests/unit/test_pst_variant_coverage.py. It reads the variant names, and the cargo feature each one needs, from the cedar-policy-core source that Cargo.lock resolves to, then compares them to a snapshot in the repo. A version bump that adds or renames a variant fails and names it. I also added your corpus idea. All 7,600 fuzzer policies now go through policies_to_pst and back. 7,551 convert and rebuild into an identical node tree. The other 49 fail inside cedar's own arity checks, not ours.

5. Added, with to_pst as the other half. Policy::from_pst and PolicySet::from_pst are public in 4.12, so this needs no crate internals.

6. You are right, and the commit message was the wrong part. The motivation is TPE residuals in feat: expose type-aware partial evaluation (TPE) as tpe_authorize #109, which are pst.Template values. Residuals from is_authorized_partial cannot convert at all. I reworded the message and added the missing tests on feat: expose type-aware partial evaluation (TPE) as tpe_authorize #109.

Two things I fixed while checking that. entity_uids returned an empty set for input it could not walk, so passing the engine's PolicySet handle looked like "this policy names no entities". It raises now. I also dropped Unknown and ResidualError from this PR, because neither can occur here. ResidualError arrives in #109, where the tpe feature is on.

[skuenzli]

Thanks for the substantial updates, which I think improved the PR a lot. I'll review and share feedback.

[skuenzli]

👍

[skuenzli]

👍

[skuenzli]

👍

[skuenzli]

👍

[skuenzli]

Hi @h0rv - thank you for the updates. I reviewed the revised PR and it was very good. Rather than round-trip more feedback, I pushed my review changes directly to your branch (309f1d54..56bfab36, 15 commits). Summary:

Behavior change (the one to look at first):

• Expression nesting is now capped at 100 levels in both conversion directions. build_expr and read_expr track depth and raise ValueError past the limit; deep enough input previously meant a RecursionError from Python or a stack overflow (process abort) from the Rust builder. I measured the fuzzer corpus to size the limit: the deepest expression across all 7,551 convertible policies is ~13 levels, so 100 gives ~6x headroom. The limit is a documented behavioral contract: raising it later is backwards compatible, lowering it would be breaking. Boundary pinned by unit tests (98 nested sets convert, 99 raise).
• entity_uids's walk is now iterative (explicit work stack), so it has no depth ceiling at all.

Tests:

• The corpus conversion test now asserts cedar-side rejections by category instead of the 0.99 conversion-rate floor. All 49 current rejections are one category, extension-function arity, so the first rejection of any new kind fails the test and names the file. Corpus shrinkage is guarded directly (> 7000 files).

Docs:

• Documented the cedarpy.pst compatibility contract (tracks the engine; unmodelled syntax raises ValueError) in the module docstring, the to_pst/from_pst/policies_to_pst pydocs, and the README.
• Moved the full PST documentation to docs/guides/policy-syntax-tree-guide.md, following the pattern the partial-authorization guide set. The README keeps one section: intro, the match example, the engine-tracking line, and a pointer. This reflects a README doctrine I codified in CLAUDE.md while reviewing: the README serves new and intermediate users; advanced features get guides.
• Editorial pass on the copy, plus docstrings explaining the pst_variants.json format and why FrozenMap is hand-rolled rather than a dependency.
• CHANGELOG: fixed a stale README pointer, added the feat: expose the PST as typed nodes via policies_to_pst #108 credit, and corrected the engine-upgrade entry (pst is public API, not an experimental feature - it was just unused at 4.8.2).

Everything verified locally at the pushed tip: release build, 274 unit / 78 integration / 60,801 corpus tests pass. Please flag anything you'd have done differently - particularly the depth limit, since it constrains input your applications might produce.