Feature/mqtt connector - #64
Merged
Merged
Conversation
…cation serialization
…equirements files
There was a problem hiding this comment.
Pull request overview
This PR introduces MQTT connector support to the machine data model and refactors the connector API to use a resolved RemoteResource object instead of passing (path, remote_resource_spec) pairs, while also improving connector lifecycle management and YAML (de)serialization.
Changes:
- Added MQTT connector implementation (topics, codecs, subscriptions) plus YAML constructors/representers and examples/tests.
- Introduced
RemoteResourceand updated core node/connector APIs (OPC UA + variable/method nodes) to use it end-to-end. - Improved connector cleanup semantics in
DataModel(idempotent close + failure cleanup + finalizer) and hardened async connector event-loop management.
Reviewed changes
Copilot reviewed 32 out of 35 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/nodes/connectors/test_abstract_async_connector.py | Tests event-loop recreation on reconnect |
| tests/nodes/connectors/opcua/test_opcua_connector.py | Updates OPC UA tests to RemoteResource |
| tests/nodes/connectors/opcua/conftest.py | Stabilizes OPC UA server startup logic |
| tests/nodes/connectors/mqtt/test_mqtt_remote_resource_spec.py | Tests MQTT topic resolution/validation |
| tests/nodes/connectors/mqtt/test_mqtt_integration.py | Docker-based MQTT integration tests |
| tests/nodes/connectors/mqtt/test_mqtt_data_model.py | Tests DataModel remote resource reuse + close semantics |
| tests/nodes/connectors/mqtt/test_mqtt_connector.py | Unit tests for MQTT connector + codecs |
| tests/nodes/connectors/mqtt/conftest.py | Mosquitto Docker fixture + readiness wait |
| tests/builder/test_data_model_dumper.py | YAML dump/load coverage for MQTT types |
| tests/builder/test_data_model_builder.py | YAML build coverage for MQTT types |
| scripts/gen_requirements.sh | Script to export requirements from Poetry |
| requirements.txt | Updated exported runtime requirements |
| requirements-dev.txt | Updated exported dev requirements |
| README.md | Adds connector docs and MQTT example |
| pyproject.toml | Adds aiomqtt/msgpack deps |
| poetry.lock | Lockfile updated (Poetry v2 format) |
| machine_data_model/nodes/variable_node.py | Uses RemoteResource; avoids overwriting defaults with None |
| machine_data_model/nodes/method_node.py | Uses RemoteResource for remote method calls |
| machine_data_model/nodes/data_model_node.py | Adds cached/lazy remote_resource resolution |
| machine_data_model/nodes/connectors/remote_resource.py | New RemoteResource type (path+spec+weak node ref) |
| machine_data_model/nodes/connectors/opcua/opcua_remote_resource_spec.py | Removes parent pointer; adds clone_for_child |
| machine_data_model/nodes/connectors/opcua/opcua_connector.py | Migrates OPC UA connector API to RemoteResource |
| machine_data_model/nodes/connectors/mqtt/mqtt_remote_resource_spec.py | New MQTT remote spec (topics, QoS, retain) |
| machine_data_model/nodes/connectors/mqtt/mqtt_payload_codec.py | New MQTT payload codecs (string/json/msgpack) |
| machine_data_model/nodes/connectors/mqtt/mqtt_connector.py | New MQTT connector implementation |
| machine_data_model/nodes/connectors/mqtt/init.py | Exposes MQTT public API |
| machine_data_model/nodes/connectors/abstract_remote_resource_spec.py | Adds node-aware get_remote_path, clone_for_child |
| machine_data_model/nodes/connectors/abstract_connector.py | Connector API now takes RemoteResource |
| machine_data_model/nodes/connectors/abstract_async_connector.py | Async connector API + event-loop ownership fixes |
| machine_data_model/nodes/connectors/init.py | Exports shared connector types |
| machine_data_model/data_model.py | Connector cleanup/finalizer + remote resource configuration |
| machine_data_model/builder/data_model_dumper.py | YAML representers for MQTT + shared connector field helpers |
| machine_data_model/builder/data_model_builder.py | YAML constructors for MQTT types |
| examples/opcua/connect.py | Updates example to RemoteResource API |
| examples/mqtt/basic_mqtt.yml | New MQTT example data model YAML |
Comment on lines
+263
to
+283
| connectors: | ||
| - !!OpcUaConnector | ||
| name: "opc_ua_server" | ||
| ip: "127.0.0.1" | ||
| port: 4840 | ||
|
|
||
| root: | ||
| !!FolderNode | ||
| name: "Objects" | ||
| connector_name: "opc_ua_server" | ||
| children: | ||
| - !!NumericalVariableNode | ||
| name: "Temperature" | ||
| default_value: 20.0 | ||
| - !!BooleanVariableNode | ||
| name: "StartCommand" | ||
| default_value: false | ||
| remote_resource_spec: | ||
| !!OpcUaRemoteResourceSpec | ||
| node_id: "ns=2;s=StartCommand" | ||
| ``` |
Comment on lines
34
to
36
| container = docker_client.containers.run( | ||
| "mcr.microsoft.com/iotedge/opc-plc@sha256:" | ||
| "1fda0e687dee9bd86e1d40ad3e1e4a81d087777d59cab5abd62fdba9c3eefa9e", | ||
| "mcr.microsoft.com/iotedge/opc-plc:latest", | ||
| "--pn=50000 --autoaccept --sph --sn=5 --sr=10 --st=uint " |
Comment on lines
486
to
489
| raise ValueError( | ||
| f"Couldn't retrieve node '{path}', {remote_resource_spec}", | ||
| f" using the '{self.name}' " f"connector: the node doesn't exist", | ||
| ) |
…add related tests
…g address and auth fields
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request adds support for MQTT connectors in the data model, alongside improvements to connector resource management and YAML serialization/deserialization. The main themes are: introducing MQTT support, updating YAML handling for connectors, and improving connector lifecycle management and error handling.