Skip to content

refactor: implement mutation backends and migrate handlers#237

Draft
retran wants to merge 2 commits intomendixlabs:mainfrom
retran:feature/mutation-backends
Draft

refactor: implement mutation backends and migrate handlers#237
retran wants to merge 2 commits intomendixlabs:mainfrom
retran:feature/mutation-backends

Conversation

@retran
Copy link
Copy Markdown
Contributor

@retran retran commented Apr 19, 2026

Why

PR #236 defined the mutation interfaces — this PR provides the MPR-backed implementations that actually perform the BSON manipulation. The goal is to move all BSON mutation logic out of the executor and behind the backend abstraction, so the executor becomes a thin orchestrator.

What changed (incremental from PR #236)

28 files changed, +4,511/-3,523

New backend implementations:

  • mdl/backend/mpr/page_mutator.go (1,554 lines) — full PageMutator implementation: widget tree traversal, property updates, widget add/drop/move, layout changes, pluggable widget operations, all operating on BSON documents
  • mdl/backend/mpr/workflow_mutator.go (771 lines) — full WorkflowMutator implementation: activity insertion/removal, outcome management, path operations, boundary events
  • mdl/backend/mpr/widget_builder.go (1,007 lines) — widget construction and serialization: WidgetObjectBuilder, property type resolution, default widget scaffolding

Executor simplification:

  • cmd_alter_page.go: 1,721 → 256 lines — rewritten as thin orchestrator calling PageMutator methods
  • cmd_alter_workflow.go: 887 → 178 lines — same pattern with WorkflowMutator
  • widget_engine.go refactored to use WidgetObjectBuilder interface instead of direct BSON construction
  • Deleted from executor: widget_defaults.go, widget_operations.go, widget_templates.go (logic moved to backend)

Interface integration from #236:

Tests relocated: ALTER PAGE and ALTER WORKFLOW tests moved from executor to mdl/backend/mpr/ where the implementation now lives.

Stack

PR 3/5 in the shared-types refactoring stack. This is the largest PR in the stack.
Merge order: #232#235#236this#238#239

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR continues the shared-types refactor by moving Mendix project value types and mutation/serialization responsibilities into backend interfaces, reducing direct sdk/mpr / BSON coupling in the executor and improving testability via mock backends.

Changes:

  • Introduces mdl/types (WASM-safe shared types) and migrates multiple handlers/readers from sdk/mpr types to mdl/types.
  • Adds backend interfaces for page/workflow mutation and widget serialization/building, and wires the executor to use them.
  • Adds extensive mock-based tests for executor commands and updates widget engine/registry validation accordingly.

Reviewed changes

Copilot reviewed 117 out of 137 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
mdl/types/doc.go Adds package docs clarifying purpose and conversion layer location.
mdl/types/asyncapi.go Adds AsyncAPI 2.x parsing into WASM-safe shared types.
mdl/linter/rules/page_navigation_security.go Switches navigation menu traversal to mdl/types menu item types.
mdl/executor/widget_templates.go Removes executor-side BSON widget template helpers (moved behind backend).
mdl/executor/widget_registry_test.go Updates tests to match new operation validation approach.
mdl/executor/widget_registry.go Replaces OperationRegistry with a fixed knownOperations set for validation.
mdl/executor/widget_operations.go Removes executor-side operation registry and operation implementations.
mdl/executor/widget_engine_test.go Updates tests to validate knownOperations and removes registry tests.
mdl/executor/widget_defaults.go Removes executor-side object-list default population (moved behind backend).
mdl/executor/mock_test_helpers_test.go Adds shared mock context/model factories + assertion helpers for executor tests.
mdl/executor/helpers.go Migrates folder info + ID generation to mdl/types.
mdl/executor/executor.go Migrates cached unit/folder info to mdl/types.
mdl/executor/cmd_workflows_write.go Migrates workflow ID generation to mdl/types.
mdl/executor/cmd_workflows_mock_test.go Adds mock-based tests for workflow show/describe.
mdl/executor/cmd_settings_mock_test.go Adds mock-based tests for settings show/describe.
mdl/executor/cmd_security_write.go Migrates security member-access and revocation types to mdl/types.
mdl/executor/cmd_security_mock_test.go Adds mock-based tests for multiple security commands.
mdl/executor/cmd_rest_clients_mock_test.go Adds mock-based tests for REST client show/describe.
mdl/executor/cmd_rename.go Migrates rename hit types to mdl/types.
mdl/executor/cmd_published_rest_mock_test.go Adds mock-based tests for published REST services.
mdl/executor/cmd_pages_mock_test.go Adds mock-based tests for pages/snippets/layouts listing.
mdl/executor/cmd_pages_create_v3.go Wires widgetBackend into the page builder for V3 creation.
mdl/executor/cmd_pages_builder_v3_pluggable.go Uses bsonutil IDs and backend widget serialization hook.
mdl/executor/cmd_pages_builder_v3_layout.go Migrates layout/widget ID generation to mdl/types.
mdl/executor/cmd_pages_builder_input_filters.go Uses bsonutil and mdl/types IDs for filter widget BSON creation.
mdl/executor/cmd_pages_builder_input_cloning_test.go Updates cloning tests to use bsonutil IDs.
mdl/executor/cmd_pages_builder_input_cloning.go Replaces sdk/mpr ID helpers with bsonutil/mdl/types.
mdl/executor/cmd_pages_builder_input.go Removes executor-side datasource serialization and uses bsonutil IDs for refs.
mdl/executor/cmd_pages_builder.go Adds widget backend dependency and migrates folder info caching to mdl/types.
mdl/executor/cmd_odata_mock_test.go Adds mock-based tests for OData commands.
mdl/executor/cmd_odata.go Switches EDMX parsing to mdl/types and ID generation to mdl/types.
mdl/executor/cmd_notconnected_mock_test.go Adds mock-based “not connected” tests for many commands.
mdl/executor/cmd_navigation_mock_test.go Adds mock-based tests for navigation show/describe.
mdl/executor/cmd_navigation.go Migrates navigation document/spec types to mdl/types.
mdl/executor/cmd_modules_mock_test.go Adds mock-based tests for modules listing.
mdl/executor/cmd_misc_mock_test.go Adds mock-based tests for version output.
mdl/executor/cmd_microflows_mock_test.go Adds mock-based tests for microflows/nanoflows show/describe.
mdl/executor/cmd_microflows_create.go Migrates microflow IDs to mdl/types.
mdl/executor/cmd_microflows_builder_workflow.go Migrates workflow-related microflow builder IDs to mdl/types.
mdl/executor/cmd_microflows_builder_graph.go Migrates graph builder IDs to mdl/types.
mdl/executor/cmd_microflows_builder_flows.go Migrates sequence flow IDs to mdl/types.
mdl/executor/cmd_microflows_builder_control.go Migrates split/merge/loop IDs to mdl/types.
mdl/executor/cmd_microflows_builder_annotations.go Migrates annotation and event IDs to mdl/types.
mdl/executor/cmd_mermaid_mock_test.go Adds mock-based test for Mermaid domain model rendering.
mdl/executor/cmd_jsonstructures_mock_test.go Adds mock-based tests for JSON structures listing.
mdl/executor/cmd_jsonstructures.go Migrates JSON structure types/helpers to mdl/types.
mdl/executor/cmd_javascript_actions_mock_test.go Adds mock-based tests for JavaScript actions.
mdl/executor/cmd_javaactions_mock_test.go Adds mock-based tests for Java actions.
mdl/executor/cmd_javaactions.go Migrates Java action ID generation to mdl/types.
mdl/executor/cmd_import_mappings_mock_test.go Adds mock-based tests for import mappings listing.
mdl/executor/cmd_import_mappings.go Migrates JSON element types (schema alignment) to mdl/types.
mdl/executor/cmd_imagecollections_mock_test.go Adds mock-based tests for image collections show/describe.
mdl/executor/cmd_imagecollections.go Migrates image collection types to mdl/types.
mdl/executor/cmd_fragments_mock_test.go Adds mock-based tests for fragments listing.
mdl/executor/cmd_folders.go Migrates folder info usage to mdl/types.
mdl/executor/cmd_export_mappings_mock_test.go Adds mock-based tests for export mappings listing.
mdl/executor/cmd_export_mappings.go Migrates JSON element types (schema alignment) to mdl/types.
mdl/executor/cmd_enumerations_mock_test.go Refactors enumeration tests to common mock helpers and adds describe tests.
mdl/executor/cmd_entities_mock_test.go Adds mock-based tests for entities listing.
mdl/executor/cmd_entities.go Migrates entity-related ID generation to mdl/types.
mdl/executor/cmd_diff_local.go Moves microflow parsing from sdk/mpr to backend method.
mdl/executor/cmd_dbconnection_mock_test.go Adds mock-based tests for database connections show/describe.
mdl/executor/cmd_datatransformer_mock_test.go Adds mock-based tests for data transformers list/describe.
mdl/executor/cmd_constants_mock_test.go Adds mock-based tests for constants show/describe.
mdl/executor/cmd_businessevents_mock_test.go Adds mock-based tests for business event service commands.
mdl/executor/cmd_businessevents.go Migrates ID generation to mdl/types.
mdl/executor/cmd_associations_mock_test.go Adds mock-based tests for association listing.
mdl/executor/cmd_agenteditor_mock_test.go Adds mock-based tests for agent editor commands.
mdl/catalog/builder_references.go Migrates navigation menu item reference extraction to mdl/types.
mdl/catalog/builder_navigation.go Migrates navigation indexing functions to mdl/types.
mdl/catalog/builder_contract.go Migrates EDMX/AsyncAPI parsing to mdl/types.
mdl/catalog/builder.go Updates CatalogReader interface to use mdl/types shared structs.
mdl/bsonutil/bsonutil.go Adds BSON-safe UUID helpers without sdk/mpr dependency.
mdl/backend/workflow.go Migrates image collection types to mdl/types.
mdl/backend/security.go Migrates member access + revocation types to mdl/types.
mdl/backend/navigation.go Migrates navigation types/specs to mdl/types.
mdl/backend/mutation.go Adds mutation + widget builder/serialization backend interfaces.
mdl/backend/mock/mock_workflow.go Updates mock backend to use mdl/types image collection types.
mdl/backend/mock/mock_security.go Updates mock backend to use mdl/types revocation types.
mdl/backend/mock/mock_navigation.go Updates mock backend to use mdl/types navigation types/specs.
mdl/backend/mock/mock_mutation.go Adds mock methods for new mutation/serialization/builder interfaces.
mdl/backend/mock/mock_module.go Updates folder listing mocks to mdl/types.
mdl/backend/mock/mock_microflow.go Adds mock hook for backend microflow parsing from raw BSON maps.
mdl/backend/mock/mock_mapping.go Updates JSON structure types to mdl/types.
mdl/backend/mock/mock_java.go Updates Java/JavaScript action types to mdl/types.
mdl/backend/mock/mock_infrastructure.go Updates raw unit + rename + widget type mocks to mdl/types; makes agent editor create/delete overrideable.
mdl/backend/mock/mock_connection.go Migrates version/project version types to mdl/types.
mdl/backend/mock/backend.go Extends mock backend surface to cover new interfaces and mdl/types types.
mdl/backend/microflow.go Adds ParseMicroflowFromRaw to backend API.
mdl/backend/mapping.go Migrates mapping backend JSON structure types to mdl/types.
mdl/backend/java.go Migrates Java backend list/read API types to mdl/types.
mdl/backend/infrastructure.go Migrates rename/raw unit/metadata/widget type APIs to mdl/types.
mdl/backend/doc.go Updates docs to reflect mdl/types extraction.
mdl/backend/connection.go Migrates connection + folder backend types to mdl/types.
mdl/backend/backend.go Extends FullBackend to include mutation + serialization + widget builder backends.
cmd/mxcli/project_tree.go Migrates menu tree building to mdl/types.NavMenuItem.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 316 to 322
currentContainerID = newFolderID

// Add to cache
pb.foldersCache = append(pb.foldersCache, &mpr.FolderInfo{
pb.foldersCache = append(pb.foldersCache, &types.FolderInfo{
ID: newFolderID,
ContainerID: currentContainerID,
Name: part,
Copy link

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ContainerID is being set to currentContainerID after currentContainerID has been updated to newFolderID, which makes the new folder appear as its own container. This will break subsequent folder lookups that rely on correct parentage. Store the prior container ID (the parent) and use that for ContainerID, then update currentContainerID to newFolderID.

Copilot uses AI. Check for mistakes.
Comment on lines +93 to +101
raw := pb.widgetBackend.SerializeWidgetToOpaque(widget)
if raw == nil {
return nil, nil
}
bsonD, ok := raw.(bson.D)
if !ok {
return nil, mdlerrors.NewValidationf("SerializeWidgetToOpaque returned unexpected type %T", raw)
}
return bsonD, nil
Copy link

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SerializeWidgetToOpaque is defined to return an opaque any, but the executor immediately asserts it must be bson.D. This undermines the 'opaque' contract and will fail for any backend that returns a different representation (e.g., bytes). Either (mandatory) change the interface to return bson.D (if BSON is the required storage form for this path), or (optional) keep the interface opaque and move the BSON conversion/casting behind the backend (e.g., provide an executor-facing SerializeWidgetToBSON method).

Copilot uses AI. Check for mistakes.
Comment thread mdl/types/asyncapi.go Outdated
}

// Resolve messages from components
for name, msg := range raw.Components.Messages {
Copy link

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These loops iterate over Go maps, so doc.Messages, doc.Channels, and AsyncAPIMessage.Properties will have non-deterministic ordering. If any downstream output (e.g., catalog ingestion, diagnostics, or tests) expects stable ordering, this can lead to flaky diffs/results. Consider collecting keys, sorting them, and iterating deterministically when constructing slices.

Copilot uses AI. Check for mistakes.
Comment thread mdl/types/asyncapi.go Outdated
}

// Resolve channels
for channelName, channel := range raw.Channels {
Copy link

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These loops iterate over Go maps, so doc.Messages, doc.Channels, and AsyncAPIMessage.Properties will have non-deterministic ordering. If any downstream output (e.g., catalog ingestion, diagnostics, or tests) expects stable ordering, this can lead to flaky diffs/results. Consider collecting keys, sorting them, and iterating deterministically when constructing slices.

Copilot uses AI. Check for mistakes.
Comment thread mdl/types/asyncapi.go Outdated

func resolveAsyncSchemaProperties(schema yamlAsyncSchema) []*AsyncAPIProperty {
var props []*AsyncAPIProperty
for name, prop := range schema.Properties {
Copy link

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These loops iterate over Go maps, so doc.Messages, doc.Channels, and AsyncAPIMessage.Properties will have non-deterministic ordering. If any downstream output (e.g., catalog ingestion, diagnostics, or tests) expects stable ordering, this can lead to flaky diffs/results. Consider collecting keys, sorting them, and iterating deterministically when constructing slices.

Copilot uses AI. Check for mistakes.
Comment on lines +23 to +38
// knownOperations is the set of operation names supported by the widget engine.
var knownOperations = map[string]bool{
"attribute": true,
"association": true,
"primitive": true,
"selection": true,
"expression": true,
"datasource": true,
"widgets": true,
"texttemplate": true,
"action": true,
"attributeObjects": true,
}

// NewWidgetRegistryWithOps creates a registry pre-loaded with embedded definitions,
// validating operations against the provided OperationRegistry.
func NewWidgetRegistryWithOps(opReg *OperationRegistry) (*WidgetRegistry, error) {
// NewWidgetRegistry creates a registry pre-loaded with embedded definitions.
func NewWidgetRegistry() (*WidgetRegistry, error) {
Copy link

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change removes the ability to validate/extend operations via an injected OperationRegistry (and removes NewWidgetRegistryWithOps). If custom widget definitions previously relied on additional operations (even just for forward compatibility), they will now fail validation. If extensibility is still required, reintroduce an injection point (e.g., NewWidgetRegistry(ops map[string]struct{}) or WithKnownOperations(...)) while keeping a default set for the common case.

Copilot uses AI. Check for mistakes.
Comment thread mdl/types/asyncapi.go
Comment on lines +46 to +55
// ParseAsyncAPI parses an AsyncAPI YAML string into an AsyncAPIDocument.
func ParseAsyncAPI(yamlStr string) (*AsyncAPIDocument, error) {
if yamlStr == "" {
return nil, fmt.Errorf("empty AsyncAPI document")
}

var raw yamlAsyncAPI
if err := yaml.Unmarshal([]byte(yamlStr), &raw); err != nil {
return nil, fmt.Errorf("failed to parse AsyncAPI YAML: %w", err)
}
Copy link

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New parsing behavior is introduced without corresponding tests in this diff. Adding unit tests for ParseAsyncAPI would help lock in behavior for empty input, YAML parse failures, $ref resolution, and channel/message extraction (and would also guard against ordering-related flakiness if deterministic ordering is implemented).

Copilot uses AI. Check for mistakes.
@retran retran marked this pull request as draft April 19, 2026 18:09
@retran retran force-pushed the feature/mutation-backends branch 2 times, most recently from 0adddbc to 55b21eb Compare April 20, 2026 08:01
@ako
Copy link
Copy Markdown
Collaborator

ako commented Apr 20, 2026

Code Review

Overview

This is the largest PR in the stack (+12,710/-5,703 lines, 28 files) and delivers the actual implementation behind the interfaces defined in PR #236. The executor's cmd_alter_page.go shrinks from 1,721 → 256 lines and cmd_alter_workflow.go from 887 → 178 lines. Both become thin orchestrators. The BSON mutation logic moves to page_mutator.go (1,554 lines), workflow_mutator.go (771 lines), and widget_builder.go (1,007 lines) in mdl/backend/mpr/. This is a well-executed refactoring overall.

Note: This PR is currently in DRAFT state.


Positive Aspects

  • Compile-time interface satisfaction checks are present. Both page_mutator.go and workflow_mutator.go have var _ backend.PageMutator = (*mprPageMutator)(nil) guards — exactly what was flagged as missing in the PR refactor: define mutation backend interfaces #236 review.
  • BSON tree walking is thorough. findBsonWidget and findBsonWidgetInSnippet correctly handle all container types: FormCall, LayoutGrid (rows/columns), TabContainer, DataView (Widgets + FooterWidgets), ControlBar, CustomWidget (pluggable properties). No container type is obviously missing.
  • Widget builder uses templates, not hand-constructed BSON. widget_builder.go loads pre-built templates via widgets.GetTemplateFullBSON() and modifies them — which is the safe approach established in the codebase.
  • Thin orchestrators are genuinely thin. cmd_alter_page.go is a clean dispatcher with no BSON logic. cmd_alter_workflow.go maps all 10+ operation types through a single switch and delegates every case to a mutator call.
  • page_mutator_test.go (725 lines) has solid coverage — widget tree walking, drop/set operations, scope extraction, BSON array marker preservation.

Issues & Concerns

1. Potential method visibility issue in workflow_mutator.go — verify before merge

The analysis flagged that the MprBackend factory method may be defined with a lowercase name (openWorkflowForMutation vs the interface's OpenWorkflowForMutation). If real, this would be a compile error. Since this is a DRAFT it may be intentional/WIP — but worth explicitly verifying the method name matches the interface before promoting to ready-for-review.

2. Workflow mutation operations are undertested

workflow_mutator_test.go (484 lines) covers property setting, activity finding, and drop operations. It does not test:

  • InsertAfterActivity
  • ReplaceActivity
  • InsertOutcome / DropOutcome
  • InsertPath / DropPath
  • InsertBranch / DropBranch
  • InsertBoundaryEvent
  • SetActivityProperty (only one test for DUE_DATE)

These are the most complex mutation operations and the most likely to have edge case bugs (e.g., position handling, sub-flow construction, deduplication). The corresponding page_mutator_test.go has better coverage of its operations.

3. BSON helpers are duplicated across three files

bson_helpers.go (executor-private), page_mutator.go, and workflow_mutator.go all define the same low-level helpers (dGet, dGetDoc, dSet, dGetArrayElements, etc.). This is pragmatic but creates a maintenance risk — a fix to one copy won't propagate. These belong in mdl/bsonutil or a shared internal package.

4. No MockPageMutator / MockWorkflowMutator types

mock_mutation.go mocks the backend factory methods but doesn't provide types that implement the mutator interfaces themselves. The test files construct mprPageMutator/mprWorkflowMutator directly (testing the real implementation), which is fine for backend tests. But executor-level tests that want to test cmd_alter_page.go logic in isolation (without BSON roundtrips) have no mock mutator to inject.

5. SetLayout (96 lines in page_mutator.go) warrants close review

The parameter remapping logic when changing a page's layout is the most complex method in the mutator. It correctly handles old/new layout qualified names and argument list remapping, but the complexity makes it a candidate for dedicated tests — none exist currently.


Checklist Against Project Standards

Item Status
Interface satisfaction compile-time checks ✅ Present in both mutator files
Thin orchestrators ✅ Both executor files genuinely delegate all logic
BSON container type coverage ✅ All known container types handled
Widget builder uses templates ✅ No hand-constructed BSON
page_mutator_test.go coverage ✅ Tree walking, drops, properties, scope
workflow_mutator_test.go coverage ⚠️ Missing insert/replace/outcome/path/branch tests
BSON helper deduplication ⚠️ Three copies across executor and backend
Mock mutator types ⚠️ Factory mocked; mutator interface not mockable

Verdict

Strong work overall — the refactoring goal is achieved. The executor is genuinely decoupled from BSON, the implementations are thorough, and the page-side tests are solid. The main gaps before this moves out of DRAFT are: (1) verify the OpenWorkflowForMutation method name; (2) add tests for the write-path workflow operations (insert/replace/outcomes/paths/branches) — these are the riskiest operations and currently have zero test coverage.

Implement PageMutator, WorkflowMutator, and WidgetBuilderBackend in
mdl/backend/mpr/. Rewrite ALTER PAGE (1721→256 lines) and ALTER
WORKFLOW (887→178 lines) as thin orchestrators using mutator sessions.
Implement PluggableWidgetEngine with WidgetObjectBuilder interface,
eliminating all BSON from widget_engine.go.

- Create mdl/backend/mpr/page_mutator.go (1554 lines)
- Create mdl/backend/mpr/workflow_mutator.go (771 lines)
- Create mdl/backend/mpr/widget_builder.go (1007 lines)
- Migrate SerializeWidget/ClientAction/DataSource to backend interface
- Add ParseMicroflowFromRaw to MicroflowBackend interface
- Delete widget_operations.go, widget_templates.go, widget_defaults.go
- Move ALTER PAGE/WORKFLOW tests to backend/mpr/ package
@retran retran force-pushed the feature/mutation-backends branch from 55b21eb to 96b12d4 Compare April 20, 2026 09:30
Cover all forward conversion functions (convertProjectVersion, convertFolderInfoSlice,
convertNavDoc, convertJsonStructure, convertImageCollection, etc.) and reverse
unconvert functions (unconvertNavProfileSpec, unconvertNavMenuItemSpec,
unconvertEntityMemberAccessSlice, unconvertEntityAccessRevocation,
unconvertJsonStructure, unconvertImageCollection) with fully-populated struct
inputs, nil/error passthrough, and recursive structure verification.

Includes Slice/Ptr wrapper tests for all wrapper variants with error
passthrough and nil handling. 33 tests total in convert_roundtrip_test.go.
@retran retran force-pushed the feature/mutation-backends branch from 96b12d4 to a2d0752 Compare April 20, 2026 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants