refactor: implement mutation backends and migrate handlers#237
refactor: implement mutation backends and migrate handlers#237retran wants to merge 2 commits intomendixlabs:mainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR continues the shared-types refactor by moving Mendix project value types and mutation/serialization responsibilities into backend interfaces, reducing direct sdk/mpr / BSON coupling in the executor and improving testability via mock backends.
Changes:
- Introduces
mdl/types(WASM-safe shared types) and migrates multiple handlers/readers fromsdk/mprtypes tomdl/types. - Adds backend interfaces for page/workflow mutation and widget serialization/building, and wires the executor to use them.
- Adds extensive mock-based tests for executor commands and updates widget engine/registry validation accordingly.
Reviewed changes
Copilot reviewed 117 out of 137 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| mdl/types/doc.go | Adds package docs clarifying purpose and conversion layer location. |
| mdl/types/asyncapi.go | Adds AsyncAPI 2.x parsing into WASM-safe shared types. |
| mdl/linter/rules/page_navigation_security.go | Switches navigation menu traversal to mdl/types menu item types. |
| mdl/executor/widget_templates.go | Removes executor-side BSON widget template helpers (moved behind backend). |
| mdl/executor/widget_registry_test.go | Updates tests to match new operation validation approach. |
| mdl/executor/widget_registry.go | Replaces OperationRegistry with a fixed knownOperations set for validation. |
| mdl/executor/widget_operations.go | Removes executor-side operation registry and operation implementations. |
| mdl/executor/widget_engine_test.go | Updates tests to validate knownOperations and removes registry tests. |
| mdl/executor/widget_defaults.go | Removes executor-side object-list default population (moved behind backend). |
| mdl/executor/mock_test_helpers_test.go | Adds shared mock context/model factories + assertion helpers for executor tests. |
| mdl/executor/helpers.go | Migrates folder info + ID generation to mdl/types. |
| mdl/executor/executor.go | Migrates cached unit/folder info to mdl/types. |
| mdl/executor/cmd_workflows_write.go | Migrates workflow ID generation to mdl/types. |
| mdl/executor/cmd_workflows_mock_test.go | Adds mock-based tests for workflow show/describe. |
| mdl/executor/cmd_settings_mock_test.go | Adds mock-based tests for settings show/describe. |
| mdl/executor/cmd_security_write.go | Migrates security member-access and revocation types to mdl/types. |
| mdl/executor/cmd_security_mock_test.go | Adds mock-based tests for multiple security commands. |
| mdl/executor/cmd_rest_clients_mock_test.go | Adds mock-based tests for REST client show/describe. |
| mdl/executor/cmd_rename.go | Migrates rename hit types to mdl/types. |
| mdl/executor/cmd_published_rest_mock_test.go | Adds mock-based tests for published REST services. |
| mdl/executor/cmd_pages_mock_test.go | Adds mock-based tests for pages/snippets/layouts listing. |
| mdl/executor/cmd_pages_create_v3.go | Wires widgetBackend into the page builder for V3 creation. |
| mdl/executor/cmd_pages_builder_v3_pluggable.go | Uses bsonutil IDs and backend widget serialization hook. |
| mdl/executor/cmd_pages_builder_v3_layout.go | Migrates layout/widget ID generation to mdl/types. |
| mdl/executor/cmd_pages_builder_input_filters.go | Uses bsonutil and mdl/types IDs for filter widget BSON creation. |
| mdl/executor/cmd_pages_builder_input_cloning_test.go | Updates cloning tests to use bsonutil IDs. |
| mdl/executor/cmd_pages_builder_input_cloning.go | Replaces sdk/mpr ID helpers with bsonutil/mdl/types. |
| mdl/executor/cmd_pages_builder_input.go | Removes executor-side datasource serialization and uses bsonutil IDs for refs. |
| mdl/executor/cmd_pages_builder.go | Adds widget backend dependency and migrates folder info caching to mdl/types. |
| mdl/executor/cmd_odata_mock_test.go | Adds mock-based tests for OData commands. |
| mdl/executor/cmd_odata.go | Switches EDMX parsing to mdl/types and ID generation to mdl/types. |
| mdl/executor/cmd_notconnected_mock_test.go | Adds mock-based “not connected” tests for many commands. |
| mdl/executor/cmd_navigation_mock_test.go | Adds mock-based tests for navigation show/describe. |
| mdl/executor/cmd_navigation.go | Migrates navigation document/spec types to mdl/types. |
| mdl/executor/cmd_modules_mock_test.go | Adds mock-based tests for modules listing. |
| mdl/executor/cmd_misc_mock_test.go | Adds mock-based tests for version output. |
| mdl/executor/cmd_microflows_mock_test.go | Adds mock-based tests for microflows/nanoflows show/describe. |
| mdl/executor/cmd_microflows_create.go | Migrates microflow IDs to mdl/types. |
| mdl/executor/cmd_microflows_builder_workflow.go | Migrates workflow-related microflow builder IDs to mdl/types. |
| mdl/executor/cmd_microflows_builder_graph.go | Migrates graph builder IDs to mdl/types. |
| mdl/executor/cmd_microflows_builder_flows.go | Migrates sequence flow IDs to mdl/types. |
| mdl/executor/cmd_microflows_builder_control.go | Migrates split/merge/loop IDs to mdl/types. |
| mdl/executor/cmd_microflows_builder_annotations.go | Migrates annotation and event IDs to mdl/types. |
| mdl/executor/cmd_mermaid_mock_test.go | Adds mock-based test for Mermaid domain model rendering. |
| mdl/executor/cmd_jsonstructures_mock_test.go | Adds mock-based tests for JSON structures listing. |
| mdl/executor/cmd_jsonstructures.go | Migrates JSON structure types/helpers to mdl/types. |
| mdl/executor/cmd_javascript_actions_mock_test.go | Adds mock-based tests for JavaScript actions. |
| mdl/executor/cmd_javaactions_mock_test.go | Adds mock-based tests for Java actions. |
| mdl/executor/cmd_javaactions.go | Migrates Java action ID generation to mdl/types. |
| mdl/executor/cmd_import_mappings_mock_test.go | Adds mock-based tests for import mappings listing. |
| mdl/executor/cmd_import_mappings.go | Migrates JSON element types (schema alignment) to mdl/types. |
| mdl/executor/cmd_imagecollections_mock_test.go | Adds mock-based tests for image collections show/describe. |
| mdl/executor/cmd_imagecollections.go | Migrates image collection types to mdl/types. |
| mdl/executor/cmd_fragments_mock_test.go | Adds mock-based tests for fragments listing. |
| mdl/executor/cmd_folders.go | Migrates folder info usage to mdl/types. |
| mdl/executor/cmd_export_mappings_mock_test.go | Adds mock-based tests for export mappings listing. |
| mdl/executor/cmd_export_mappings.go | Migrates JSON element types (schema alignment) to mdl/types. |
| mdl/executor/cmd_enumerations_mock_test.go | Refactors enumeration tests to common mock helpers and adds describe tests. |
| mdl/executor/cmd_entities_mock_test.go | Adds mock-based tests for entities listing. |
| mdl/executor/cmd_entities.go | Migrates entity-related ID generation to mdl/types. |
| mdl/executor/cmd_diff_local.go | Moves microflow parsing from sdk/mpr to backend method. |
| mdl/executor/cmd_dbconnection_mock_test.go | Adds mock-based tests for database connections show/describe. |
| mdl/executor/cmd_datatransformer_mock_test.go | Adds mock-based tests for data transformers list/describe. |
| mdl/executor/cmd_constants_mock_test.go | Adds mock-based tests for constants show/describe. |
| mdl/executor/cmd_businessevents_mock_test.go | Adds mock-based tests for business event service commands. |
| mdl/executor/cmd_businessevents.go | Migrates ID generation to mdl/types. |
| mdl/executor/cmd_associations_mock_test.go | Adds mock-based tests for association listing. |
| mdl/executor/cmd_agenteditor_mock_test.go | Adds mock-based tests for agent editor commands. |
| mdl/catalog/builder_references.go | Migrates navigation menu item reference extraction to mdl/types. |
| mdl/catalog/builder_navigation.go | Migrates navigation indexing functions to mdl/types. |
| mdl/catalog/builder_contract.go | Migrates EDMX/AsyncAPI parsing to mdl/types. |
| mdl/catalog/builder.go | Updates CatalogReader interface to use mdl/types shared structs. |
| mdl/bsonutil/bsonutil.go | Adds BSON-safe UUID helpers without sdk/mpr dependency. |
| mdl/backend/workflow.go | Migrates image collection types to mdl/types. |
| mdl/backend/security.go | Migrates member access + revocation types to mdl/types. |
| mdl/backend/navigation.go | Migrates navigation types/specs to mdl/types. |
| mdl/backend/mutation.go | Adds mutation + widget builder/serialization backend interfaces. |
| mdl/backend/mock/mock_workflow.go | Updates mock backend to use mdl/types image collection types. |
| mdl/backend/mock/mock_security.go | Updates mock backend to use mdl/types revocation types. |
| mdl/backend/mock/mock_navigation.go | Updates mock backend to use mdl/types navigation types/specs. |
| mdl/backend/mock/mock_mutation.go | Adds mock methods for new mutation/serialization/builder interfaces. |
| mdl/backend/mock/mock_module.go | Updates folder listing mocks to mdl/types. |
| mdl/backend/mock/mock_microflow.go | Adds mock hook for backend microflow parsing from raw BSON maps. |
| mdl/backend/mock/mock_mapping.go | Updates JSON structure types to mdl/types. |
| mdl/backend/mock/mock_java.go | Updates Java/JavaScript action types to mdl/types. |
| mdl/backend/mock/mock_infrastructure.go | Updates raw unit + rename + widget type mocks to mdl/types; makes agent editor create/delete overrideable. |
| mdl/backend/mock/mock_connection.go | Migrates version/project version types to mdl/types. |
| mdl/backend/mock/backend.go | Extends mock backend surface to cover new interfaces and mdl/types types. |
| mdl/backend/microflow.go | Adds ParseMicroflowFromRaw to backend API. |
| mdl/backend/mapping.go | Migrates mapping backend JSON structure types to mdl/types. |
| mdl/backend/java.go | Migrates Java backend list/read API types to mdl/types. |
| mdl/backend/infrastructure.go | Migrates rename/raw unit/metadata/widget type APIs to mdl/types. |
| mdl/backend/doc.go | Updates docs to reflect mdl/types extraction. |
| mdl/backend/connection.go | Migrates connection + folder backend types to mdl/types. |
| mdl/backend/backend.go | Extends FullBackend to include mutation + serialization + widget builder backends. |
| cmd/mxcli/project_tree.go | Migrates menu tree building to mdl/types.NavMenuItem. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| currentContainerID = newFolderID | ||
|
|
||
| // Add to cache | ||
| pb.foldersCache = append(pb.foldersCache, &mpr.FolderInfo{ | ||
| pb.foldersCache = append(pb.foldersCache, &types.FolderInfo{ | ||
| ID: newFolderID, | ||
| ContainerID: currentContainerID, | ||
| Name: part, |
There was a problem hiding this comment.
ContainerID is being set to currentContainerID after currentContainerID has been updated to newFolderID, which makes the new folder appear as its own container. This will break subsequent folder lookups that rely on correct parentage. Store the prior container ID (the parent) and use that for ContainerID, then update currentContainerID to newFolderID.
| raw := pb.widgetBackend.SerializeWidgetToOpaque(widget) | ||
| if raw == nil { | ||
| return nil, nil | ||
| } | ||
| bsonD, ok := raw.(bson.D) | ||
| if !ok { | ||
| return nil, mdlerrors.NewValidationf("SerializeWidgetToOpaque returned unexpected type %T", raw) | ||
| } | ||
| return bsonD, nil |
There was a problem hiding this comment.
SerializeWidgetToOpaque is defined to return an opaque any, but the executor immediately asserts it must be bson.D. This undermines the 'opaque' contract and will fail for any backend that returns a different representation (e.g., bytes). Either (mandatory) change the interface to return bson.D (if BSON is the required storage form for this path), or (optional) keep the interface opaque and move the BSON conversion/casting behind the backend (e.g., provide an executor-facing SerializeWidgetToBSON method).
| } | ||
|
|
||
| // Resolve messages from components | ||
| for name, msg := range raw.Components.Messages { |
There was a problem hiding this comment.
These loops iterate over Go maps, so doc.Messages, doc.Channels, and AsyncAPIMessage.Properties will have non-deterministic ordering. If any downstream output (e.g., catalog ingestion, diagnostics, or tests) expects stable ordering, this can lead to flaky diffs/results. Consider collecting keys, sorting them, and iterating deterministically when constructing slices.
| } | ||
|
|
||
| // Resolve channels | ||
| for channelName, channel := range raw.Channels { |
There was a problem hiding this comment.
These loops iterate over Go maps, so doc.Messages, doc.Channels, and AsyncAPIMessage.Properties will have non-deterministic ordering. If any downstream output (e.g., catalog ingestion, diagnostics, or tests) expects stable ordering, this can lead to flaky diffs/results. Consider collecting keys, sorting them, and iterating deterministically when constructing slices.
|
|
||
| func resolveAsyncSchemaProperties(schema yamlAsyncSchema) []*AsyncAPIProperty { | ||
| var props []*AsyncAPIProperty | ||
| for name, prop := range schema.Properties { |
There was a problem hiding this comment.
These loops iterate over Go maps, so doc.Messages, doc.Channels, and AsyncAPIMessage.Properties will have non-deterministic ordering. If any downstream output (e.g., catalog ingestion, diagnostics, or tests) expects stable ordering, this can lead to flaky diffs/results. Consider collecting keys, sorting them, and iterating deterministically when constructing slices.
| // knownOperations is the set of operation names supported by the widget engine. | ||
| var knownOperations = map[string]bool{ | ||
| "attribute": true, | ||
| "association": true, | ||
| "primitive": true, | ||
| "selection": true, | ||
| "expression": true, | ||
| "datasource": true, | ||
| "widgets": true, | ||
| "texttemplate": true, | ||
| "action": true, | ||
| "attributeObjects": true, | ||
| } | ||
|
|
||
| // NewWidgetRegistryWithOps creates a registry pre-loaded with embedded definitions, | ||
| // validating operations against the provided OperationRegistry. | ||
| func NewWidgetRegistryWithOps(opReg *OperationRegistry) (*WidgetRegistry, error) { | ||
| // NewWidgetRegistry creates a registry pre-loaded with embedded definitions. | ||
| func NewWidgetRegistry() (*WidgetRegistry, error) { |
There was a problem hiding this comment.
This change removes the ability to validate/extend operations via an injected OperationRegistry (and removes NewWidgetRegistryWithOps). If custom widget definitions previously relied on additional operations (even just for forward compatibility), they will now fail validation. If extensibility is still required, reintroduce an injection point (e.g., NewWidgetRegistry(ops map[string]struct{}) or WithKnownOperations(...)) while keeping a default set for the common case.
| // ParseAsyncAPI parses an AsyncAPI YAML string into an AsyncAPIDocument. | ||
| func ParseAsyncAPI(yamlStr string) (*AsyncAPIDocument, error) { | ||
| if yamlStr == "" { | ||
| return nil, fmt.Errorf("empty AsyncAPI document") | ||
| } | ||
|
|
||
| var raw yamlAsyncAPI | ||
| if err := yaml.Unmarshal([]byte(yamlStr), &raw); err != nil { | ||
| return nil, fmt.Errorf("failed to parse AsyncAPI YAML: %w", err) | ||
| } |
There was a problem hiding this comment.
New parsing behavior is introduced without corresponding tests in this diff. Adding unit tests for ParseAsyncAPI would help lock in behavior for empty input, YAML parse failures, $ref resolution, and channel/message extraction (and would also guard against ordering-related flakiness if deterministic ordering is implemented).
0adddbc to
55b21eb
Compare
Code ReviewOverviewThis is the largest PR in the stack (+12,710/-5,703 lines, 28 files) and delivers the actual implementation behind the interfaces defined in PR #236. The executor's Note: This PR is currently in DRAFT state. Positive Aspects
Issues & Concerns1. Potential method visibility issue in The analysis flagged that the 2. Workflow mutation operations are undertested
These are the most complex mutation operations and the most likely to have edge case bugs (e.g., position handling, sub-flow construction, deduplication). The corresponding 3. BSON helpers are duplicated across three files
4. No
5. The parameter remapping logic when changing a page's layout is the most complex method in the mutator. It correctly handles old/new layout qualified names and argument list remapping, but the complexity makes it a candidate for dedicated tests — none exist currently. Checklist Against Project Standards
VerdictStrong work overall — the refactoring goal is achieved. The executor is genuinely decoupled from BSON, the implementations are thorough, and the page-side tests are solid. The main gaps before this moves out of DRAFT are: (1) verify the |
Implement PageMutator, WorkflowMutator, and WidgetBuilderBackend in mdl/backend/mpr/. Rewrite ALTER PAGE (1721→256 lines) and ALTER WORKFLOW (887→178 lines) as thin orchestrators using mutator sessions. Implement PluggableWidgetEngine with WidgetObjectBuilder interface, eliminating all BSON from widget_engine.go. - Create mdl/backend/mpr/page_mutator.go (1554 lines) - Create mdl/backend/mpr/workflow_mutator.go (771 lines) - Create mdl/backend/mpr/widget_builder.go (1007 lines) - Migrate SerializeWidget/ClientAction/DataSource to backend interface - Add ParseMicroflowFromRaw to MicroflowBackend interface - Delete widget_operations.go, widget_templates.go, widget_defaults.go - Move ALTER PAGE/WORKFLOW tests to backend/mpr/ package
55b21eb to
96b12d4
Compare
Cover all forward conversion functions (convertProjectVersion, convertFolderInfoSlice, convertNavDoc, convertJsonStructure, convertImageCollection, etc.) and reverse unconvert functions (unconvertNavProfileSpec, unconvertNavMenuItemSpec, unconvertEntityMemberAccessSlice, unconvertEntityAccessRevocation, unconvertJsonStructure, unconvertImageCollection) with fully-populated struct inputs, nil/error passthrough, and recursive structure verification. Includes Slice/Ptr wrapper tests for all wrapper variants with error passthrough and nil handling. 33 tests total in convert_roundtrip_test.go.
96b12d4 to
a2d0752
Compare
Why
PR #236 defined the mutation interfaces — this PR provides the MPR-backed implementations that actually perform the BSON manipulation. The goal is to move all BSON mutation logic out of the executor and behind the backend abstraction, so the executor becomes a thin orchestrator.
What changed (incremental from PR #236)
28 files changed, +4,511/-3,523
New backend implementations:
mdl/backend/mpr/page_mutator.go(1,554 lines) — fullPageMutatorimplementation: widget tree traversal, property updates, widget add/drop/move, layout changes, pluggable widget operations, all operating on BSON documentsmdl/backend/mpr/workflow_mutator.go(771 lines) — fullWorkflowMutatorimplementation: activity insertion/removal, outcome management, path operations, boundary eventsmdl/backend/mpr/widget_builder.go(1,007 lines) — widget construction and serialization:WidgetObjectBuilder, property type resolution, default widget scaffoldingExecutor simplification:
cmd_alter_page.go: 1,721 → 256 lines — rewritten as thin orchestrator callingPageMutatormethodscmd_alter_workflow.go: 887 → 178 lines — same pattern withWorkflowMutatorwidget_engine.gorefactored to useWidgetObjectBuilderinterface instead of direct BSON constructionwidget_defaults.go,widget_operations.go,widget_templates.go(logic moved to backend)Interface integration from #236:
ContainerKind,InsertPosition,PluggablePropertyOpconstants defined in PR refactor: define mutation backend interfaces #236InsertWidgetsignature usesInsertPositiontyped parameter (case-insensitive comparison in implementation)Tests relocated: ALTER PAGE and ALTER WORKFLOW tests moved from executor to
mdl/backend/mpr/where the implementation now lives.Stack
PR 3/5 in the shared-types refactoring stack. This is the largest PR in the stack.
Merge order: #232 → #235 → #236 → this → #238 → #239