diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f6ae28d..6afb175 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -18,16 +18,16 @@ jobs: - name: Setup .NET uses: actions/setup-dotnet@v1 with: - dotnet-version: 5.0.x + dotnet-version: 10.0.x - name: Restore dependencies run: dotnet restore - name: Build - run: dotnet build --no-restore + run: dotnet build -c Release --no-restore - name: Test - run: dotnet test --no-build --verbosity normal - - - name: Publish - run: dotnet publish + run: dotnet test -c Release --no-build --verbosity normal + + - name: Pack + run: dotnet pack -c Release --no-build diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 8224a75..95f4224 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -17,25 +17,25 @@ jobs: - name: Setup .NET uses: actions/setup-dotnet@v1 with: - dotnet-version: 5.0.x + dotnet-version: 10.0.x - name: Restore dependencies run: dotnet restore - name: Build - run: dotnet build -c Release --no-restore --version-suffix="$GITHUB_REF_NAME.$GITHUB_RUN_NUMBER" - - - name: Publish - run: dotnet publish -c Release --no-build --version-suffix="$GITHUB_REF_NAME.$GITHUB_RUN_NUMBER" + run: dotnet build -c Release --no-restore + + - name: Test + run: dotnet test -c Release --no-build --verbosity normal - name: Pack - run: dotnet pack -c Release --no-build --version-suffix="$GITHUB_REF_NAME.$GITHUB_RUN_NUMBER" + run: dotnet pack -c Release --no-build - name: Upload Artifacts uses: actions/upload-artifact@v2 with: name: dpp.cot - path: dpp.cot/bin/*/*.nupkg + path: dpp.cot/bin/Release/*.nupkg - name: Publish To Nuget run: dotnet nuget push dpp.cot/bin/Release/*.nupkg -k $NUGET_AUTH_TOKEN -s https://api.nuget.org/v3/index.json diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..0ad95a5 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,10 @@ +# Changelog + +## 2.0.0 + +- separate the XML/domain model from protobuf wire DTOs and explicit mapping +- add TAK streaming envelope support and stronger framed message parsing +- preserve unknown and mixed `detail` XML more faithfully, including typed elements with extra attributes +- add public-source-backed fixture coverage from ATAK CIV and `taky` +- document source inventory, fixture provenance, compatibility policy, and type inference +- multi-target the library for `netstandard2.0` and `net10.0` diff --git a/docs/compatibility-policy.md b/docs/compatibility-policy.md new file mode 100644 index 0000000..b58e6a3 --- /dev/null +++ b/docs/compatibility-policy.md @@ -0,0 +1,85 @@ +# Compatibility Policy + +This page describes how `dpp.cot` handles compatibility when public ATAK/TAK sources do not expose exactly the same protobuf surface. + +## Source Hierarchy + +For protobuf and transport behavior, this repo uses the following hierarchy: + +1. repo-local public protocol artifacts committed with this project +2. public ATAK CIV client artifacts +3. public TAK Server artifacts +4. implementation policy documented here and backed by tests + +Sources: + +- [`dpp.cot/protobuf/`](../dpp.cot/protobuf/) +- +- + +## Baseline + +The compatibility baseline for `dpp.cot` is: + +- decode legacy XML CoT correctly +- encode and decode protobuf v1 payloads using the committed local `.proto` contract +- preserve XML `detail` information even when it exceeds the protobuf typed subset + +This means XML fidelity is treated as a first-class requirement, not as a lossy staging format for protobuf. + +## Public Proto Split + +The public ATAK CIV client repository and the public TAK Server repository do not expose an identical protobuf surface. + +Observed split: + +- ATAK CIV `takmessage.proto` exposes `takControl` and `cotEvent` +- public TAK Server `takmessage.proto` also exposes `submissionTime` and `creationTime` +- ATAK CIV `cotevent.proto` exposes the base event fields and `detail` +- public TAK Server `cotevent.proto` also exposes fields such as `caveat` and `releaseableTo` + +Implication: + +- `dpp.cot` should remain able to interoperate with the narrower public ATAK CIV client surface +- `dpp.cot` may also support public TAK Server extensions when they remain wire-compatible and do not break the client baseline + +## Policy For Additional Fields + +`dpp.cot` may implement public server-side fields beyond the public ATAK CIV client baseline when all of the following hold: + +- the field exists in a public upstream protocol artifact +- protobuf unknown-field behavior keeps omission safe for narrower peers +- local mapping and tests clearly document the behavior + +Current examples: + +- `submissionTime` +- `creationTime` +- `caveat` +- `releaseableTo` + +These are treated as compatible public extensions, not as proof that every public client emits them. + +## Policy For XML Detail + +When XML `detail` content fits one of the typed protobuf messages exactly, `dpp.cot` may map it into that message. + +When XML `detail` content does not fit exactly, `dpp.cot` should prefer preserving the full XML element in residual `xmlDetail`. + +This includes cases such as: + +- additional attributes on an otherwise known typed element +- repeated occurrences of an element intended for a single typed field +- nested structures that the typed protobuf model does not represent + +This policy is derived from the public `detail.proto` rule set and reinforced by public ATAK CIV example payloads. + +## Current Practical Policy + +In practical terms, `dpp.cot` currently aims to be: + +- conservative about protobuf conversion +- lossless about XML preservation +- explicit about public-source-backed extensions + +If a future field or mapping rule is only visible in one public upstream implementation, it should be documented as a compatibility policy choice rather than a universal protocol truth unless a stronger normative source is available. diff --git a/docs/fixture-provenance.md b/docs/fixture-provenance.md new file mode 100644 index 0000000..9bf9778 --- /dev/null +++ b/docs/fixture-provenance.md @@ -0,0 +1,77 @@ +# Fixture Provenance + +This page records the public origin of borrowed or adapted protocol fixtures used in the test suite. + +## Rule + +When a test fixture is adapted from a public upstream source, the local fixture should: + +- name the upstream repository +- point to the upstream file when practical +- explain what protocol behavior the fixture protects + +## ATAK CIV Fixtures + +Source repository: +- + +Imported example files: +- [`takcot/examples/Marker - 2525.cot`](https://github.com/deptofdefense/AndroidTacticalAssaultKit-CIV/blob/main/takcot/examples/Marker%20-%202525.cot) +- [`takcot/examples/Geo Fence.cot`](https://github.com/deptofdefense/AndroidTacticalAssaultKit-CIV/blob/main/takcot/examples/Geo%20Fence.cot) +- [`takcot/examples/Route.cot`](https://github.com/deptofdefense/AndroidTacticalAssaultKit-CIV/blob/main/takcot/examples/Route.cot) + +Local use: +- [`Helpers.cs`](../dpp.cot.Tests/Helpers.cs) +- [`SerializationTests.cs`](../dpp.cot.Tests/SerializationTests.cs) + +Behaviors protected: +- repeated unknown `detail` elements remain ordered +- nested unknown `detail` XML subtrees are preserved +- common ATAK client payloads survive XML parse and semantic round-trip +- typed elements with extra XML attributes are not silently flattened away + +## taky Fixtures + +Source repository: +- + +Imported test files: +- [`tests/test_takuser.py`](https://github.com/tkuester/taky/blob/016f2c3657455617dae3c302cef7ca0c743df8e4/tests/test_takuser.py) +- [`tests/test_geo_chat.py`](https://github.com/tkuester/taky/blob/016f2c3657455617dae3c302cef7ca0c743df8e4/tests/test_geo_chat.py) + +Local use: +- [`Helpers.cs`](../dpp.cot.Tests/Helpers.cs) +- [`SerializationTests.cs`](../dpp.cot.Tests/SerializationTests.cs) +- [`ProtoMappingTests.cs`](../dpp.cot.Tests/ProtoMappingTests.cs) + +Behaviors protected: +- extra attributes on typed `detail` elements +- GeoChat-style residual XML preservation +- protobuf `xmlDetail` fallback for typed elements that no longer fit the proto shape + +## PyTAK References + +Source repository: +- + +Referenced test files: +- [`tests/test_classes.py`](https://github.com/snstac/pytak/blob/fa875a181bd724a7a841be2384677fc363ebbac5/tests/test_classes.py) +- [`tests/test_functions.py`](https://github.com/snstac/pytak/blob/fa875a181bd724a7a841be2384677fc363ebbac5/tests/test_functions.py) + +Current use: +- documentation and behavior reference only + +Behaviors observed: +- generated CoT event defaults +- common event and point attribute expectations +- `_flow-tags_` detail output + +## Provenance Boundary + +Fixtures here are used as public interoperability examples, not as normative specification text. + +The normative hierarchy for this repo remains: + +1. local copies of public protocol artifacts such as `.proto`, `protocol.txt`, and `takcot` XSD material +2. public implementation references such as ATAK CIV and TAK Server +3. public fixture sources used to protect specific tested behaviors diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..0d499e2 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,29 @@ +# dpp.cot + +`dpp.cot` is a library for handling Cursor-on-Target messages. + +This documentation set focuses on: + +- supported protocol behavior +- source attribution +- implementation notes backed by tests + +The goal is to make the supported parts of the library easier to understand and verify. + +## Coverage + +This repo currently documents and tests: + +- CoT XML `event` parsing and serialization +- TAK protobuf payload v1 mapping +- TAK streaming envelope framing +- selected typed `detail` elements +- preservation of unsupported or unknown `detail` elements as residual XML + +## Reading Guide + +- start with [Scope](scope.md) for what these docs are and are not claiming +- use [Protocols](protocols/cot-xml.md) for wire-format behavior +- use [Library Design](library-design.md) for implementation structure +- use [Testing](testing.md) for executable support boundaries +- use [Sources](sources.md) for public source material used in the docs diff --git a/docs/interoperability.md b/docs/interoperability.md new file mode 100644 index 0000000..7a9dc92 --- /dev/null +++ b/docs/interoperability.md @@ -0,0 +1,158 @@ +# Interoperability Notes + +This page records public implementation-reference observations relevant to interoperability. + +## Public Implementation Reference Used Here + +- ATAK CIV client repository: +- TAK Product Center Server repository: +- PyTAK repository: +- taky repository: +- FreeTakServer repository: + +The ATAK CIV client and TAK Server repositories are treated as the primary public implementation references here, not as replacements for normative protocol artifacts. + +## Current Compatibility Notes + +### TAK Protobuf `TakMessage` + +The public TAK Server protobuf definition includes: + +- `takControl = 1` +- `cotEvent = 2` +- `submissionTime = 3` +- `creationTime = 4` + +Source: +- [`takmessage.proto` in TAK Server](https://github.com/TAK-Product-Center/Server/blob/5187abd46d827d37cfc5708805eced197a837e49/src/takserver-protobuf/src/main/proto/takmessage.proto) + +Current `dpp.cot` status: + +- supports `takControl` +- supports `cotEvent` +- supports `submissionTime` +- supports `creationTime` + +Compatibility implication: + +- these top-level protobuf timestamps now round-trip through the current `dpp.cot` object model + +### TAK Protobuf `TakControl` + +The public TAK Server protobuf definition for `TakControl` contains: + +- `minProtoVersion` +- `maxProtoVersion` + +Source: +- [`takcontrol.proto` in TAK Server](https://github.com/TAK-Product-Center/Server/blob/5187abd46d827d37cfc5708805eced197a837e49/src/takserver-protobuf/src/main/proto/takcontrol.proto) + +Current `dpp.cot` status: + +- includes `minProtoVersion` +- includes `maxProtoVersion` +- does not include non-public or extra `TakControl` protobuf fields beyond the public TAK Server shape + +Compatibility implication: + +- `TakControl` is now aligned with the public TAK Server protobuf definition referenced above + +### TAK Protobuf `CotEvent` + +The public TAK Server protobuf definition includes optional `caveat` and `releaseableTo` fields in addition to the event fields already implemented here. + +Source: +- [`cotevent.proto` in TAK Server](https://github.com/TAK-Product-Center/Server/blob/5187abd46d827d37cfc5708805eced197a837e49/src/takserver-protobuf/src/main/proto/cotevent.proto) + +Current `dpp.cot` status: + +- supports `type`, `access`, `qos`, `opex`, `caveat`, `releaseableTo`, `uid`, `sendTime`, `startTime`, `staleTime`, `how`, point fields, and `detail` + +Compatibility implication: + +- these optional fields now round-trip through the current `dpp.cot` implementation + +### Mesh-Style Protobuf Header + +The public TAK Server `SingleProtobufOrCotProtocol` reads the mesh-style protobuf header as: + +- magic byte +- unsigned varint protocol version +- magic byte +- protobuf payload + +Source: +- [`SingleProtobufOrCotProtocol.java` in TAK Server](https://github.com/TAK-Product-Center/Server/blob/5187abd46d827d37cfc5708805eced197a837e49/src/takserver-core/src/main/java/com/bbn/marti/nio/protocol/connections/SingleProtobufOrCotProtocol.java) + +Current `dpp.cot` status: + +- supports the same visible version-1 byte sequence for current use +- parses mesh-style protocol version as an unsigned varint + +Compatibility implication: + +- current version-1 compatibility is aligned and the framing implementation matches the public varint-based version parsing model + +### Streaming Envelope + +The public TAK Server streaming implementation uses: + +- `0xbf` magic byte +- unsigned varint payload length +- payload bytes + +Sources: +- [`StreamingProtoBufProtocol.java` in TAK Server](https://github.com/TAK-Product-Center/Server/blob/5187abd46d827d37cfc5708805eced197a837e49/src/takserver-core/src/main/java/com/bbn/marti/nio/protocol/connections/StreamingProtoBufProtocol.java) +- [`StreamingProtoBufHelper.java` in TAK Server](https://github.com/TAK-Product-Center/Server/blob/5187abd46d827d37cfc5708805eced197a837e49/src/takserver-plugins/src/main/java/tak/server/proto/StreamingProtoBufHelper.java) + +Current `dpp.cot` status: + +- implements the same streaming envelope shape +- supports chained-message parsing from concatenated buffers + +### `detail` Conversion Rule Strictness + +The public TAK Server protobuf helper only promotes supported `detail` child elements into typed protobuf fields when the whole element matches the expected attribute set. Otherwise, the element remains in `xmlDetail`. + +Source: +- [`StreamingProtoBufHelper.java` in TAK Server](https://github.com/TAK-Product-Center/Server/blob/5187abd46d827d37cfc5708805eced197a837e49/src/takserver-plugins/src/main/java/tak/server/proto/StreamingProtoBufHelper.java) + +Current `dpp.cot` status: + +- preserves unknown `detail` child elements +- supports typed mapping for known `detail` child elements +- preserves extra XML attributes on supported typed `detail` elements during XML parse and XML serialization +- falls back to residual `xmlDetail` when a known typed element contains extra attributes that cannot be represented by the protobuf type +- applies `xmlDetail` override semantics on protobuf decode when residual XML contains the same known element names + +Compatibility implication: + +- receiver-side duplicate-name override behavior is aligned more closely with the public TAK Server implementation +- public fixtures such as the `taky` TAK-user payload can now retain extra attributes in XML and degrade to residual `xmlDetail` on protobuf conversion instead of silently dropping them + +### Public Fixture Coverage + +Public tests and examples from other implementations are useful when they expose real CoT payloads. + +Current examples in this repo include: + +- `deptofdefense/AndroidTacticalAssaultKit-CIV` as a primary public client-side interoperability reference +- ATAK CIV example CoT payloads for markers, routes, geofences, and drawing products under `takcot/examples/` +- `tkuester/taky` test fixtures for a TAK-user event with an extra `contact phone=...` attribute and a GeoChat event with `__chat`, `link`, `remarks`, `__serverdestination`, and `marti` +- `snstac/pytak` tests for generated CoT event structure, point defaults, and `_flow-tags_` detail output +- `FreeTakServer` embedded XML samples that show common ATAK-style `detail` combinations + +Implication: + +- `dpp.cot` must preserve repeated unknown elements, nested unknown XML subtrees, and typed elements that carry extra XML attributes because public ATAK client examples use all three patterns + +## Summary + +Based on the public ATAK/TAK implementation references, `dpp.cot` appears broadly aligned on: + +- CoT XML event basics +- typed `detail` handling for supported elements +- protobuf point flattening and Unix millisecond times +- TAK streaming envelope framing + +Known compatibility gaps relative to those public implementations are narrower now and mainly concern unsupported typed detail sub-schemas that are still preserved only as residual XML. diff --git a/docs/library-design.md b/docs/library-design.md new file mode 100644 index 0000000..a596e4f --- /dev/null +++ b/docs/library-design.md @@ -0,0 +1,36 @@ +# Library Design + +## Design Principle + +The library separates three concerns: + +- XML/domain model +- protobuf wire DTOs +- mapping and framing logic + +## Main Components + +- XML/domain model: [`Event.cs`](../dpp.cot/Event.cs), [`Detail.cs`](../dpp.cot/Detail.cs), related typed elements +- protobuf DTOs: [`ProtoContracts.cs`](../dpp.cot/ProtoContracts.cs) +- XML/protobuf mapping: [`CotProtoMapper.cs`](../dpp.cot/CotProtoMapper.cs) +- mesh-style payload framing: [`Message.cs`](../dpp.cot/Message.cs) +- streaming framing: [`TakProtocolStreaming.cs`](../dpp.cot/TakProtocolStreaming.cs) + +## Why This Separation Exists + +The XML wire format and protobuf wire format are both formal protocols, but they are not structurally identical. + +Separating them reduces the risk that: + +- XML object shapes accidentally define protobuf behavior +- protobuf field numbering leaks into XML-domain types +- one wire format silently corrupts the other + +## Target Frameworks + +The library currently multi-targets: + +- `netstandard2.0` for broad .NET ecosystem compatibility +- `net10.0` for a current runtime target verified in this repo + +The test project remains on the newest local runtime so the library can be validated in this environment while still shipping a broadly consumable package surface. diff --git a/docs/protocols/cot-xml.md b/docs/protocols/cot-xml.md new file mode 100644 index 0000000..4a4abcf --- /dev/null +++ b/docs/protocols/cot-xml.md @@ -0,0 +1,43 @@ +# CoT XML Event + +## Scope + +This page describes the CoT XML `event` shape supported by `dpp.cot`. + +## Public Basis + +- the public MITRE CoT overview linked in [Sources](../sources.md) +- repo-local public artifacts such as [`dpp.cot/CoTtypes.xml`](../../dpp.cot/CoTtypes.xml) + +For type-string lineage and other compact taxonomy inferences, see [Type Inference](../type-inference.md). + +## Supported Base Shape + +`dpp.cot` models the XML event as: + +- `event` attributes such as `version`, `uid`, `type`, `time`, `start`, `stale`, and `how` +- a `point` child +- an optional `detail` child + +## Detail Handling + +Known typed `detail` children currently modeled in the library include: + +- `contact` +- `__group` +- `precisionlocation` +- `status` +- `takv` +- `track` + +Unknown `detail` children are preserved as residual XML. + +Implementation Note: +This preservation behavior is backed by tests in [`dpp.cot.Tests/SerializationTests.cs`](../../dpp.cot.Tests/SerializationTests.cs). + +## Time Handling + +The XML serializer emits UTC timestamps. + +Implementation Note: +The serializer behavior is implemented in [`dpp.cot/Event.cs`](../../dpp.cot/Event.cs). diff --git a/docs/protocols/tak-protobuf.md b/docs/protocols/tak-protobuf.md new file mode 100644 index 0000000..cc502bf --- /dev/null +++ b/docs/protocols/tak-protobuf.md @@ -0,0 +1,41 @@ +# TAK Protobuf Payload + +## Normative Source + +The protobuf payload v1 mapping used by this repo is defined by the public `.proto` files committed in this repository: + +- [`takmessage.proto`](../../dpp.cot/protobuf/takmessage.proto) +- [`cotevent.proto`](../../dpp.cot/protobuf/cotevent.proto) +- [`detail.proto`](../../dpp.cot/protobuf/detail.proto) +- [`track.proto`](../../dpp.cot/protobuf/track.proto) + +## Additional Public Implementation Context + +For interoperability-oriented context, this repo may also reference public ATAK CIV and TAK Server artifacts listed in [Sources](../sources.md). + +Those repositories are treated here as public implementation references, not as the normative source for protobuf field definitions. + +When those public sources expose slightly different protobuf surfaces, the rule used by this repo is documented in [Compatibility Policy](../compatibility-policy.md). + +## Implementation Approach + +This repo uses dedicated protobuf DTOs to match the authoritative wire contract. + +The XML/domain model is kept separate from the protobuf DTO layer because: + +- the XML shape is hierarchical +- the protobuf shape is flattened in places +- protobuf timestamps use Unix milliseconds + +## Mapping Behavior + +Current mapping behavior includes: + +- `Point` flattening into `lat`, `lon`, `hae`, `ce`, and `le` +- event time conversion to Unix milliseconds +- typed mapping for supported `detail` elements +- residual unknown `detail` XML stored in `xmlDetail` +- whole-element fallback to `xmlDetail` when a known typed element no longer fits the proto shape exactly + +Implementation Note: +The mapping code lives in [`dpp.cot/CotProtoMapper.cs`](../../dpp.cot/CotProtoMapper.cs) and is tested in [`dpp.cot.Tests/ProtoMappingTests.cs`](../../dpp.cot.Tests/ProtoMappingTests.cs). diff --git a/docs/protocols/tak-streaming.md b/docs/protocols/tak-streaming.md new file mode 100644 index 0000000..5f39824 --- /dev/null +++ b/docs/protocols/tak-streaming.md @@ -0,0 +1,28 @@ +# TAK Streaming Envelope + +## Normative Source + +The streaming framing described here comes from the public protocol text committed in this repo: + +- [`dpp.cot/protobuf/protocol.txt`](../../dpp.cot/protobuf/protocol.txt) + +## Envelope Shape + +The streaming envelope is: + +- one magic byte `0xbf` +- an unsigned protobuf-style varint message length +- the payload bytes + +The payload is version-specific and does not contain the mesh-style version-identification header. + +## Implementation Support + +This repo provides helpers for: + +- encoding streaming messages +- parsing a single streaming message +- incrementally parsing one message from a concatenated byte buffer + +Implementation Note: +The streaming helpers are implemented in [`dpp.cot/TakProtocolStreaming.cs`](../../dpp.cot/TakProtocolStreaming.cs) and covered by [`dpp.cot.Tests/StreamingTests.cs`](../../dpp.cot.Tests/StreamingTests.cs). diff --git a/docs/scope.md b/docs/scope.md new file mode 100644 index 0000000..1be7e87 --- /dev/null +++ b/docs/scope.md @@ -0,0 +1,27 @@ +# Scope + +This documentation describes the behavior implemented by `dpp.cot`. + +It does not attempt to claim complete coverage of the broader Cursor-on-Target ecosystem. + +## Documentation Standard + +A protocol claim in this repo should be grounded in one or more of: + +- a public source committed in this repository +- a public external source linked in [Sources](sources.md) +- a public implementation reference when clearly labeled as such +- a documented compatibility policy when public sources diverge +- a documented implementation note backed by tests + +## Implementation Notes + +Some parts of CoT and TAK behavior are not fully described by a single public source. + +In those cases, this repo may document an `Implementation Note`. + +An implementation note means: + +- the statement describes repo behavior +- the statement is backed by code and tests +- the statement should not be confused with a normative protocol statement unless a public source is cited diff --git a/docs/source-policy.md b/docs/source-policy.md new file mode 100644 index 0000000..00bb132 --- /dev/null +++ b/docs/source-policy.md @@ -0,0 +1,28 @@ +# Source Policy + +This repository should keep its public documentation grounded in public sources and repo-local implementation work. + +## Allowed Source Types + +- public protocol text committed in this repo +- public `.proto` files committed in this repo +- public XML examples used in tests +- public external documents explicitly linked in [Sources](sources.md) +- public implementation repositories explicitly linked in [Sources](sources.md) +- implementation behavior demonstrated by code and tests in this repo + +## Documentation Rules + +- cite a public source for normative protocol claims +- label implementation-repository evidence as implementation context, not as the normative spec +- label repo-specific behavior as an `Implementation Note` +- keep repo-derived evidence separate from external protocol sources + +## Exclusions + +This documentation set should not rely on non-public or access-controlled material. + +If a behavior cannot be supported from public sources, the public docs should either: + +1. omit the claim, or +2. document only the observed implementation behavior as an implementation note diff --git a/docs/source-validation.md b/docs/source-validation.md new file mode 100644 index 0000000..64457f1 --- /dev/null +++ b/docs/source-validation.md @@ -0,0 +1,50 @@ +# Source Validation + +This page records how local protocol artifacts relate to public source material. + +## Exact Public Upstream Matches + +The following local files were validated as exact Git blob matches against public files in `deptofdefense/AndroidTacticalAssaultKit-CIV`. + +Local path -> public upstream + +- [`dpp.cot/protobuf/contact.proto`](../dpp.cot/protobuf/contact.proto) -> +- [`dpp.cot/protobuf/cotevent.proto`](../dpp.cot/protobuf/cotevent.proto) -> +- [`dpp.cot/protobuf/detail.proto`](../dpp.cot/protobuf/detail.proto) -> +- [`dpp.cot/protobuf/group.proto`](../dpp.cot/protobuf/group.proto) -> +- [`dpp.cot/protobuf/precisionlocation.proto`](../dpp.cot/protobuf/precisionlocation.proto) -> +- [`dpp.cot/protobuf/status.proto`](../dpp.cot/protobuf/status.proto) -> +- [`dpp.cot/protobuf/takcontrol.proto`](../dpp.cot/protobuf/takcontrol.proto) -> +- [`dpp.cot/protobuf/takmessage.proto`](../dpp.cot/protobuf/takmessage.proto) -> +- [`dpp.cot/protobuf/takv.proto`](../dpp.cot/protobuf/takv.proto) -> +- [`dpp.cot/protobuf/track.proto`](../dpp.cot/protobuf/track.proto) -> +- [`dpp.cot/protobuf/protocol.txt`](../dpp.cot/protobuf/protocol.txt) -> + +These files are kept here as exact local copies of the corresponding public ATAK CIV artifacts. + +## Public Prior Art With Validated Equivalent Data + +- [`dpp.cot/CoTtypes.xml`](../dpp.cot/CoTtypes.xml) + +Public source material: + +- the local file header identifies it as MITRE CoT type material +- public mirrors of MITRE-derived `CoTtypes.xml` exist, including: + - + - + +Validation notes: + +- public antecedent is clear +- exact byte-for-byte equality was not found +- structural validation shows the local file is semantically identical to the public mirrors after correcting one obvious public typo: `zot="a-.-A-C"` should be `cot="a-.-A-C"` + +This file is best understood as a corrected MITRE-derived public copy with equivalent data to the public mirrors listed above. + +## Documentation Rule + +When noting sources, this repo distinguishes between: + +- exact public upstream copy +- public prior art / validated equivalent data +- local implementation artifact diff --git a/docs/sources.md b/docs/sources.md new file mode 100644 index 0000000..3efd39c --- /dev/null +++ b/docs/sources.md @@ -0,0 +1,56 @@ +# Sources + +This page tracks the public sources currently used in the docs for this repo. + +## Local Copies Of Public Sources + +- TAK protocol text: [`dpp.cot/protobuf/protocol.txt`](../dpp.cot/protobuf/protocol.txt) +- TAK protobuf definitions in [`dpp.cot/protobuf/`](../dpp.cot/protobuf/) +- CoT type data: [`dpp.cot/CoTtypes.xml`](../dpp.cot/CoTtypes.xml) + +Validation status for these files is tracked in [Source Validation](source-validation.md). + +Current summary: + +- the protobuf family and `protocol.txt` are exact public ATAK CIV upstream matches +- `CoTtypes.xml` has validated public prior art and is semantically equivalent to public MITRE-derived mirrors, with one obvious upstream typo corrected locally + +## External Public Sources + +- MITRE CoT overview PDF: +- ATAK CIV client repository: +- TAK Product Center Server repository: +- JMS-2525 repository: +- PyTAK repository: +- taky repository: +- FreeTakServer repository: + +## Public Implementation Reference + +These repositories are useful as public implementation references for interoperability context, surrounding protocol behavior, and public fixture sourcing. + +Primary public implementation references: + +- `deptofdefense/AndroidTacticalAssaultKit-CIV` for the public ATAK client implementation +- `TAK-Product-Center/Server` for the public TAK server implementation + +It should be treated as: + +- a public implementation source +- useful for observing practical client and server behavior and surrounding documentation +- not a substitute for normative protocol artifacts such as the committed `.proto` files and protocol text in this repo + +Public fixture notes: + +- `tkuester/taky` includes CoT payloads in public tests that are useful for round-trip and residual-detail coverage +- `snstac/pytak` includes public tests for generated CoT event structure and common default fields +- `FreeTAKTeam/FreeTakServer` includes public implementation examples and embedded CoT samples that are useful for interoperability context + +## Repo-Derived Support + +These are implementation support sources, not normative specifications: + +- [`dpp.cot.Tests/SerializationTests.cs`](../dpp.cot.Tests/SerializationTests.cs) +- [`dpp.cot.Tests/MessageTests.cs`](../dpp.cot.Tests/MessageTests.cs) +- [`dpp.cot.Tests/ProtoMappingTests.cs`](../dpp.cot.Tests/ProtoMappingTests.cs) +- [`dpp.cot.Tests/StreamingTests.cs`](../dpp.cot.Tests/StreamingTests.cs) diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..1a3169f --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,45 @@ +# Testing + +## Principle + +Claims in the docs should correspond to executable checks in the test suite whenever practical. + +## Current Coverage Areas + +- XML parsing and semantic round-trip +- known and unknown `detail` preservation +- extra attributes on known typed `detail` elements +- protobuf payload mapping +- mesh-style framed payload handling +- streaming envelope handling +- CoT predicate and description helpers + +## Public Fixture Sources + +- [`takcot/examples/Marker - 2525.cot` in `deptofdefense/AndroidTacticalAssaultKit-CIV`](https://github.com/deptofdefense/AndroidTacticalAssaultKit-CIV/blob/main/takcot/examples/Marker%20-%202525.cot) +- [`takcot/examples/Geo Fence.cot` in `deptofdefense/AndroidTacticalAssaultKit-CIV`](https://github.com/deptofdefense/AndroidTacticalAssaultKit-CIV/blob/main/takcot/examples/Geo%20Fence.cot) +- [`takcot/examples/Route.cot` in `deptofdefense/AndroidTacticalAssaultKit-CIV`](https://github.com/deptofdefense/AndroidTacticalAssaultKit-CIV/blob/main/takcot/examples/Route.cot) +- [`tests/test_takuser.py` in `tkuester/taky`](https://github.com/tkuester/taky/blob/016f2c3657455617dae3c302cef7ca0c743df8e4/tests/test_takuser.py) +- [`tests/test_geo_chat.py` in `tkuester/taky`](https://github.com/tkuester/taky/blob/016f2c3657455617dae3c302cef7ca0c743df8e4/tests/test_geo_chat.py) +- [`tests/test_classes.py` in `snstac/pytak`](https://github.com/snstac/pytak/blob/fa875a181bd724a7a841be2384677fc363ebbac5/tests/test_classes.py) +- [`tests/test_functions.py` in `snstac/pytak`](https://github.com/snstac/pytak/blob/fa875a181bd724a7a841be2384677fc363ebbac5/tests/test_functions.py) + +These are public fixture and behavior references, not normative specifications. When a payload is borrowed from a public test, the originating file should be cited in the corresponding local test or documentation note. + +The ATAK CIV example fixtures are especially useful because they exercise real client-authored `detail` trees with repeated elements, nested subtrees, and non-protobuf detail extensions. + +## Test Files + +- [`SerializationTests.cs`](../dpp.cot.Tests/SerializationTests.cs) +- [`MessageTests.cs`](../dpp.cot.Tests/MessageTests.cs) +- [`ProtoMappingTests.cs`](../dpp.cot.Tests/ProtoMappingTests.cs) +- [`StreamingTests.cs`](../dpp.cot.Tests/StreamingTests.cs) +- [`CotTypeTests.cs`](../dpp.cot.Tests/CotTypeTests.cs) + +## Documentation Rule + +If a new protocol behavior is documented as supported, it should normally gain: + +1. a cited source or an explicit implementation note +2. a focused test +3. a code path that can be pointed to directly diff --git a/docs/type-inference.md b/docs/type-inference.md new file mode 100644 index 0000000..f25ab64 --- /dev/null +++ b/docs/type-inference.md @@ -0,0 +1,87 @@ +# Type Inference + +This page describes how `dpp.cot` interprets CoT type strings and other compact taxonomy fields. + +## Source Basis + +The type and predicate material in this repo is grounded in public source material: + +- [`dpp.cot/CoTtypes.xml`](../dpp.cot/CoTtypes.xml) +- the MITRE CoT overview linked in [Sources](sources.md) +- public ATAK CIV example payloads under `takcot/examples/` +- the public JMS-2525 repository: + +The `CoTtypes.xml` header explicitly states that upper-case portions of the CoT type hierarchy are taken from the MIL-STD-2525 type hierarchy, while lower-case characters are CoT extensions. + +## What The Library Infers + +`dpp.cot` currently uses CoT type data in three practical ways: + +- descriptions for known type strings +- regex-style predicates over type strings +- grouping logic for common families such as air, ground, route, point, weather, and tasking + +These behaviors are exposed through: + +- [`CotDescriptions.cs`](../dpp.cot/CotDescriptions.cs) +- [`CotPredicates.cs`](../dpp.cot/CotPredicates.cs) + +The current generated tables in those files are derived from `CoTtypes.xml`. + +## What The Library Does Not Claim + +`dpp.cot` does not claim that: + +- every CoT type string is fully specified by a single public source +- CoT type parsing is equivalent to implementing the full MIL-STD-2525 symbology standard +- the library can render or validate complete 2525 symbol semantics +- every lower-case extension seen in CoT is part of MIL-STD-2525 itself + +In practice, the library treats the CoT type string as a stable taxonomy key and uses public CoT type data to attach descriptions and predicates where available. + +## Inference Categories + +For this repo, type-related statements generally fall into one of these categories: + +### Direct Type Mapping + +Statements that come directly from `CoTtypes.xml`, such as: + +- a known type string description +- a known `is` predicate +- a known `how` mapping + +### Public-Lineage Inference + +Statements that rely on the public lineage between CoT type data and MIL-STD-2525/JMS, such as: + +- upper-case path segments correspond to the 2525 military symbology hierarchy +- lower-case segments often represent CoT-side extensions or refinements + +### Implementation Inference + +Statements about how `dpp.cot` groups or exposes type information in code, such as: + +- convenience predicates +- helper methods +- generated lookup tables + +These should be understood as repo behavior, even when they are built from public source material. + +## Examples + +Public ATAK CIV example payloads reinforce that real CoT products rely on these compact type keys: + +- marker examples use values such as `a-u-G` and `b-m-p-s-m` +- route examples use `b-m-r` +- geofence examples use `u-d-c-c` + +These payloads show that CoT type strings carry operational meaning even when the surrounding `detail` tree is highly product-specific. + +## Documentation Rule + +When documenting type semantics in this repo: + +- cite `CoTtypes.xml` for direct CoT mapping data +- cite JMS-2525 or other public military symbology sources for lineage or taxonomy context +- label repo-specific grouping behavior as implementation behavior rather than universal protocol fact diff --git a/dpp.cot.Tests/Helpers.cs b/dpp.cot.Tests/Helpers.cs index 49504f6..40ef2d9 100644 --- a/dpp.cot.Tests/Helpers.cs +++ b/dpp.cot.Tests/Helpers.cs @@ -10,5 +10,16 @@ public static class Helpers { public const string SimplePayload = @""; public const string EudPayload = @"<__group name='Blue' role='Team Member'/>"; + public const string MixedDetailPayload = @""; + // Public fixture adapted from tkuester/taky tests/test_takuser.py. + public const string TakyTakUserPayload = @"<__group role='Team Member' name='Cyan'/>"; + // Public fixture adapted from tkuester/taky tests/test_geo_chat.py. + public const string TakyGeoChatPayload = @"<__chat parent='RootContactGroup' groupOwner='false' chatroom='JOKER MAN' id='ANDROID-cafebabe' senderCallsign='JENNY'>test<__serverdestination destinations='123.45.67.89:4242:tcp:ANDROID-deadbeef'/>"; + // Public fixture adapted from ATAK CIV takcot/examples/Marker - 2525.cot. + public const string AtakMarker2525Payload = @""; + // Public fixture adapted from ATAK CIV takcot/examples/Geo Fence.cot. + public const string AtakGeoFencePayload = @"<__geofence elevationMonitored='true' minElevation='-33.30720360300985' monitor='All' trigger='Entry' tracking='true' maxElevation='271.4927963969902' boundingSphere='75000.0'/>"; + // Public fixture adapted from ATAK CIV takcot/examples/Route.cot. + public const string AtakRoutePayload = @"<__routeinfo><__navcues/>"; } -} \ No newline at end of file +} diff --git a/dpp.cot.Tests/MessageTests.cs b/dpp.cot.Tests/MessageTests.cs new file mode 100644 index 0000000..d39313d --- /dev/null +++ b/dpp.cot.Tests/MessageTests.cs @@ -0,0 +1,64 @@ +using System; +using System.Linq; +using Xunit; + +namespace dpp.cot.Tests +{ + public class MessageTests + { + [Fact] + public void ParseFramedXmlRespectsOffsetAndLength() + { + var evt = Event.Parse(Helpers.MixedDetailPayload); + var message = new Message + { + Event = evt, + Control = new TakControl + { + minProtoVersion = 0, + maxProtoVersion = 0, + } + }; + + var framed = message.ToXmlBytes(); + var prefix = new byte[] { 0x01, 0x02, 0x03, 0x04 }; + var suffix = new byte[] { 0x05, 0x06 }; + var buffer = prefix.Concat(framed).Concat(suffix).ToArray(); + + var parsed = Message.Parse(buffer, prefix.Length, framed.Length); + + Assert.Equal("MIXED-1", parsed.Event.Uid); + Assert.Equal(123.4, parsed.Event.Detail.Track.Speed); + Assert.Single(parsed.Event.Detail.AdditionalElements); + } + + [Fact] + public void ProtobufRoundTripPreservesResidualDetailAndTrack() + { + var evt = Event.Parse(Helpers.MixedDetailPayload); + var message = new Message + { + Event = evt, + Control = new TakControl + { + minProtoVersion = 1, + maxProtoVersion = 1, + }, + SubmissionTime = new DateTime(2022, 2, 3, 0, 0, 0, DateTimeKind.Utc), + CreationTime = new DateTime(2022, 2, 3, 0, 5, 0, DateTimeKind.Utc), + }; + + var framed = message.ToProtobufBytes(); + var parsed = Message.Parse(framed, 0, framed.Length); + + Assert.Equal("MIXED-1", parsed.Event.Uid); + Assert.Equal(123.4, parsed.Event.Detail.Track.Speed); + Assert.Single(parsed.Event.Detail.AdditionalElements); + Assert.Contains("Droid", parsed.Event.Detail.AdditionalElements[0].OuterXml); + Assert.Equal((uint)1, parsed.Control.minProtoVersion); + Assert.Equal((uint)1, parsed.Control.maxProtoVersion); + Assert.Equal(message.SubmissionTime, parsed.SubmissionTime); + Assert.Equal(message.CreationTime, parsed.CreationTime); + } + } +} diff --git a/dpp.cot.Tests/ProtoMappingTests.cs b/dpp.cot.Tests/ProtoMappingTests.cs new file mode 100644 index 0000000..741611e --- /dev/null +++ b/dpp.cot.Tests/ProtoMappingTests.cs @@ -0,0 +1,179 @@ +using ProtoBuf; +using System; +using System.IO; +using Xunit; + +namespace dpp.cot.Tests +{ + public class ProtoMappingTests + { + [Fact] + public void ProtobufPayloadUsesAuthoritativeWireShape() + { + var evt = Event.Parse(Helpers.MixedDetailPayload); + evt.Caveat = "TEST-CAVEAT"; + evt.ReleaseableTo = "USA"; + var message = new Message + { + Event = evt, + Control = new TakControl + { + minProtoVersion = 1, + maxProtoVersion = 1, + }, + SubmissionTime = new DateTime(2022, 2, 2, 23, 0, 0, DateTimeKind.Utc), + CreationTime = new DateTime(2022, 2, 2, 23, 5, 0, DateTimeKind.Utc), + }; + + var framed = message.ToProtobufBytes(); + + Assert.Equal(0xef, framed[0]); + Assert.Equal(0x01, framed[1]); + Assert.Equal(0xef, framed[2]); + + using var ms = new MemoryStream(framed, 3, framed.Length - 3, writable: false); + var proto = Serializer.Deserialize(ms); + + Assert.NotNull(proto.TakControl); + Assert.Equal((uint)1, proto.TakControl.MinProtoVersion); + Assert.Equal((uint)1, proto.TakControl.MaxProtoVersion); + + Assert.NotNull(proto.CotEvent); + Assert.Equal("MIXED-1", proto.CotEvent.Uid); + Assert.Equal("a-f-G-U-C", proto.CotEvent.Type); + Assert.Equal("h-e", proto.CotEvent.How); + Assert.Equal("TEST-CAVEAT", proto.CotEvent.Caveat); + Assert.Equal("USA", proto.CotEvent.ReleaseableTo); + Assert.Equal(1.0, proto.CotEvent.Lat); + Assert.Equal(2.0, proto.CotEvent.Lon); + Assert.Equal(3.0, proto.CotEvent.Hae); + Assert.Equal(4.0, proto.CotEvent.Ce); + Assert.Equal(5.0, proto.CotEvent.Le); + Assert.Equal((ulong)new DateTimeOffset(evt.Time).ToUnixTimeMilliseconds(), proto.CotEvent.SendTime); + Assert.Equal((ulong)new DateTimeOffset(evt.Start).ToUnixTimeMilliseconds(), proto.CotEvent.StartTime); + Assert.Equal((ulong)new DateTimeOffset(evt.Stale).ToUnixTimeMilliseconds(), proto.CotEvent.StaleTime); + Assert.Equal((ulong)new DateTimeOffset(message.SubmissionTime.Value).ToUnixTimeMilliseconds(), proto.SubmissionTime); + Assert.Equal((ulong)new DateTimeOffset(message.CreationTime.Value).ToUnixTimeMilliseconds(), proto.CreationTime); + + Assert.NotNull(proto.CotEvent.Detail); + Assert.NotNull(proto.CotEvent.Detail.Contact); + Assert.NotNull(proto.CotEvent.Detail.Track); + Assert.Equal("ALPHA", proto.CotEvent.Detail.Contact.Callsign); + Assert.Equal(123.4, proto.CotEvent.Detail.Track.Speed); + Assert.Equal(270.0, proto.CotEvent.Detail.Track.Course); + Assert.Contains("", proto.CotEvent.Detail.XmlDetail); + Assert.DoesNotContain("hello", + Track = new Track + { + Speed = 55.5, + Course = 180.0, + } + } + } + }; + + var hydrated = CotProtoMapper.FromProto(message); + + Assert.NotNull(hydrated.Event); + Assert.NotNull(hydrated.Event.Detail); + Assert.NotNull(hydrated.Event.Detail.Track); + Assert.Equal("CAVEAT", hydrated.Event.Caveat); + Assert.Equal("REL", hydrated.Event.ReleaseableTo); + Assert.Equal(55.5, hydrated.Event.Detail.Track.Speed); + Assert.Equal(180.0, hydrated.Event.Detail.Track.Course); + Assert.Equal(2, hydrated.Event.Detail.AdditionalElements.Length); + Assert.Equal("uid", hydrated.Event.Detail.AdditionalElements[0].Name); + Assert.Equal("remarks", hydrated.Event.Detail.AdditionalElements[1].Name); + Assert.Equal("hello", hydrated.Event.Detail.AdditionalElements[1].InnerText); + } + + [Fact] + public void XmlDetailOverridesTypedDetailElementsOnDecode() + { + var message = new ProtoTakMessage + { + CotEvent = new ProtoCotEvent + { + Uid = "OVERRIDE-1", + Type = "a-f-G-U-C", + How = "h-e", + SendTime = (ulong)new DateTimeOffset(2022, 2, 2, 22, 22, 22, TimeSpan.Zero).ToUnixTimeMilliseconds(), + StartTime = (ulong)new DateTimeOffset(2022, 2, 2, 22, 22, 22, TimeSpan.Zero).ToUnixTimeMilliseconds(), + StaleTime = (ulong)new DateTimeOffset(2022, 2, 2, 22, 32, 22, TimeSpan.Zero).ToUnixTimeMilliseconds(), + Lat = 1, + Lon = 2, + Hae = 3, + Ce = 4, + Le = 5, + Detail = new ProtoDetail + { + Contact = new Contact + { + Callsign = "TYPED", + Endpoint = "1.2.3.4:1111:tcp" + }, + XmlDetail = "" + } + } + }; + + var hydrated = CotProtoMapper.FromProto(message); + + Assert.Null(hydrated.Event.Detail.Contact); + Assert.Single(hydrated.Event.Detail.AdditionalElements); + Assert.Equal("contact", hydrated.Event.Detail.AdditionalElements[0].Name); + Assert.Contains("callsign=\"XML\"", hydrated.Event.Detail.AdditionalElements[0].OuterXml); + } + + [Fact] + public void KnownTypedDetailWithExtraAttributesFallsBackToXmlDetail() + { + var evt = Event.Parse(Helpers.TakyTakUserPayload); + var message = new Message { Event = evt }; + + var framed = message.ToProtobufBytes(); + + using var ms = new MemoryStream(framed, 3, framed.Length - 3, writable: false); + var proto = Serializer.Deserialize(ms); + + Assert.NotNull(proto.CotEvent); + Assert.NotNull(proto.CotEvent.Detail); + Assert.Null(proto.CotEvent.Detail.Contact); + Assert.Contains("(); + var actualAdditional = actual.Detail.AdditionalElements ?? Array.Empty(); + Assert.Equal(expectedAdditional.Length, actualAdditional.Length); + + for (var i = 0; i < expectedAdditional.Length; i++) + { + Assert.Equal(expectedAdditional[i].OuterXml, actualAdditional[i].OuterXml); + } + } + + private static void AssertXmlAttributesEquivalent(System.Xml.XmlAttribute[] expected, System.Xml.XmlAttribute[] actual) + { + var expectedAttributes = expected ?? Array.Empty(); + var actualAttributes = actual ?? Array.Empty(); + + Assert.Equal(expectedAttributes.Length, actualAttributes.Length); - Assert.Equal(corpus, xml); + for (var i = 0; i < expectedAttributes.Length; i++) + { + Assert.Equal(expectedAttributes[i].Name, actualAttributes[i].Name); + Assert.Equal(expectedAttributes[i].Value, actualAttributes[i].Value); + } } } } diff --git a/dpp.cot.Tests/StreamingTests.cs b/dpp.cot.Tests/StreamingTests.cs new file mode 100644 index 0000000..39501e6 --- /dev/null +++ b/dpp.cot.Tests/StreamingTests.cs @@ -0,0 +1,119 @@ +using System.Linq; +using System.Text; +using Xunit; + +namespace dpp.cot.Tests +{ + public class StreamingTests + { + [Fact] + public void XmlStreamingRoundTripPreservesMessage() + { + var evt = Event.Parse(Helpers.MixedDetailPayload); + var message = new Message + { + Event = evt, + Control = new TakControl + { + minProtoVersion = 0, + maxProtoVersion = 0, + } + }; + + var streamed = message.ToStreamingBytes(0x00); + var parsed = Message.ParseStreaming(streamed, 0, streamed.Length, 0x00); + + Assert.Equal("MIXED-1", parsed.Event.Uid); + Assert.Equal(123.4, parsed.Event.Detail.Track.Speed); + Assert.Single(parsed.Event.Detail.AdditionalElements); + } + + [Fact] + public void ProtobufStreamingRoundTripPreservesMessage() + { + var evt = Event.Parse(Helpers.MixedDetailPayload); + var message = new Message + { + Event = evt, + Control = new TakControl + { + minProtoVersion = 1, + maxProtoVersion = 1, + } + }; + + var streamed = message.ToStreamingBytes(0x01); + var parsed = Message.ParseStreaming(streamed, 0, streamed.Length, 0x01); + + Assert.Equal("MIXED-1", parsed.Event.Uid); + Assert.Equal((uint)1, parsed.Control.minProtoVersion); + Assert.Single(parsed.Event.Detail.AdditionalElements); + } + + [Fact] + public void TryParseStreamingConsumesOnlyOneMessageFromConcatenatedBuffer() + { + var first = new Message + { + Event = Event.Parse(Helpers.MixedDetailPayload), + Control = new TakControl + { + minProtoVersion = 1, + maxProtoVersion = 1, + } + }; + var second = new Message + { + Event = Event.Parse(Helpers.EudPayload), + Control = new TakControl + { + minProtoVersion = 1, + maxProtoVersion = 1, + } + }; + + var firstBytes = first.ToStreamingBytes(0x01); + var secondBytes = second.ToStreamingBytes(0x01); + var buffer = firstBytes.Concat(secondBytes).ToArray(); + + Assert.True(Message.TryParseStreaming(buffer, 0, buffer.Length, 0x01, out var parsedFirst, out var firstConsumed)); + Assert.Equal(firstBytes.Length, firstConsumed); + Assert.Equal("MIXED-1", parsedFirst.Event.Uid); + + Assert.True(Message.TryParseStreaming(buffer, firstConsumed, buffer.Length - firstConsumed, 0x01, out var parsedSecond, out var secondConsumed)); + Assert.Equal(secondBytes.Length, secondConsumed); + Assert.Equal("ANDROID-ASDFASDFASDF", parsedSecond.Event.Uid); + } + + [Fact] + public void StreamingHeaderSupportsMultiByteVarintLengths() + { + var largeRemark = new string('x', 300); + var largePayload = + "" + + "" + + $"{largeRemark}"; + + var message = new Message + { + Event = Event.Parse(largePayload), + Control = new TakControl + { + minProtoVersion = 0, + maxProtoVersion = 0, + } + }; + + var streamed = message.ToStreamingBytes(0x00); + + Assert.Equal(0xbf, streamed[0]); + Assert.True((streamed[1] & 0x80) != 0); + + var parsed = Message.ParseStreaming(streamed, 0, streamed.Length, 0x00); + + Assert.Equal("STREAM-LARGE", parsed.Event.Uid); + Assert.Single(parsed.Event.Detail.AdditionalElements); + Assert.Contains(largeRemark, parsed.Event.Detail.AdditionalElements[0].OuterXml); + } + } +} diff --git a/dpp.cot.Tests/dpp.cot.Tests.csproj b/dpp.cot.Tests/dpp.cot.Tests.csproj index 60e1ce4..0588ce4 100644 --- a/dpp.cot.Tests/dpp.cot.Tests.csproj +++ b/dpp.cot.Tests/dpp.cot.Tests.csproj @@ -1,13 +1,13 @@ - net5.0 + net10.0 false - + runtime; build; native; contentfiles; analyzers; buildtransitive diff --git a/dpp.cot/Contact.cs b/dpp.cot/Contact.cs index 61097a8..e9d5126 100644 --- a/dpp.cot/Contact.cs +++ b/dpp.cot/Contact.cs @@ -5,6 +5,7 @@ using System.Linq; using System.Text; using System.Threading.Tasks; +using System.Xml; using System.Xml.Serialization; namespace dpp.cot @@ -26,5 +27,8 @@ IExtension IExtensible.GetExtensionObject(bool createIfMissing) [XmlAttribute(AttributeName = "callsign")] public string Callsign { get; set; } = ""; + [XmlAnyAttribute] + public XmlAttribute[] AdditionalAttributes { get; set; } = System.Array.Empty(); + } } diff --git a/dpp.cot/CotProtoMapper.cs b/dpp.cot/CotProtoMapper.cs new file mode 100644 index 0000000..ea87584 --- /dev/null +++ b/dpp.cot/CotProtoMapper.cs @@ -0,0 +1,341 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Xml; + +namespace dpp.cot +{ + internal static class CotProtoMapper + { + public static ProtoTakMessage ToProto(Message message) + { + return new ProtoTakMessage + { + TakControl = ToProto(message.Control), + CotEvent = ToProto(message.Event), + SubmissionTime = message.SubmissionTime.HasValue ? ToUnixMilliseconds(message.SubmissionTime.Value) : 0, + CreationTime = message.CreationTime.HasValue ? ToUnixMilliseconds(message.CreationTime.Value) : 0, + }; + } + + public static Message FromProto(ProtoTakMessage message) + { + return new Message + { + Control = FromProto(message.TakControl), + Event = FromProto(message.CotEvent), + SubmissionTime = FromUnixMillisecondsOrDefault(message.SubmissionTime), + CreationTime = FromUnixMillisecondsOrDefault(message.CreationTime), + }; + } + + private static ProtoTakControl ToProto(TakControl control) + { + if (control == null) + { + return null; + } + + return new ProtoTakControl + { + MinProtoVersion = control.minProtoVersion, + MaxProtoVersion = control.maxProtoVersion, + }; + } + + private static TakControl FromProto(ProtoTakControl control) + { + if (control == null) + { + return new TakControl(); + } + + return new TakControl + { + minProtoVersion = control.MinProtoVersion, + maxProtoVersion = control.MaxProtoVersion, + }; + } + + private static ProtoCotEvent ToProto(Event evt) + { + if (evt == null) + { + return null; + } + + return new ProtoCotEvent + { + Type = evt.Type ?? "", + Access = evt.Access ?? "", + Qos = evt.Qos ?? "", + Opex = evt.Opex ?? "", + Caveat = evt.Caveat ?? "", + ReleaseableTo = evt.ReleaseableTo ?? "", + Uid = evt.Uid ?? "", + SendTime = ToUnixMilliseconds(evt.Time), + StartTime = ToUnixMilliseconds(evt.Start), + StaleTime = ToUnixMilliseconds(evt.Stale), + How = evt.How ?? "", + Lat = evt.Point?.Lat ?? 0, + Lon = evt.Point?.Lon ?? 0, + Hae = evt.Point?.Hae ?? 0, + Ce = evt.Point?.Ce ?? 0, + Le = evt.Point?.Le ?? 0, + Detail = ToProto(evt.Detail), + }; + } + + private static Event FromProto(ProtoCotEvent evt) + { + if (evt == null) + { + return null; + } + + return new Event + { + Type = evt.Type ?? "", + Access = evt.Access ?? "", + Qos = evt.Qos ?? "", + Opex = evt.Opex ?? "", + Caveat = evt.Caveat ?? "", + ReleaseableTo = evt.ReleaseableTo ?? "", + Uid = evt.Uid ?? "", + Time = FromUnixMilliseconds(evt.SendTime), + Start = FromUnixMilliseconds(evt.StartTime), + Stale = FromUnixMilliseconds(evt.StaleTime), + How = evt.How ?? "", + Point = new Point + { + Lat = evt.Lat, + Lon = evt.Lon, + Hae = evt.Hae, + Ce = evt.Ce, + Le = evt.Le, + }, + Detail = FromProto(evt.Detail), + }; + } + + private static ProtoDetail ToProto(Detail detail) + { + if (detail == null) + { + return null; + } + + var xmlDetail = SerializeResidualDetail(detail, out var overriddenElementNames); + detail.xmlDetail = xmlDetail; + + return new ProtoDetail + { + XmlDetail = xmlDetail, + Contact = overriddenElementNames.Contains("contact") ? null : detail.Contact, + Group = overriddenElementNames.Contains("__group") ? null : detail.Group, + PrecisionLocation = overriddenElementNames.Contains("precisionlocation") ? null : detail.PrecisionLocation, + Status = overriddenElementNames.Contains("status") ? null : detail.Status, + Takv = overriddenElementNames.Contains("takv") ? null : detail.Takv, + Track = overriddenElementNames.Contains("track") ? null : detail.Track, + }; + } + + private static Detail FromProto(ProtoDetail detail) + { + if (detail == null) + { + return null; + } + + var additionalElements = ParseResidualDetail(detail.XmlDetail); + var overriddenElementNames = GetOverrideElementNames(additionalElements); + + return new Detail + { + xmlDetail = detail.XmlDetail ?? "", + Contact = overriddenElementNames.Contains("contact") ? null : detail.Contact, + Group = overriddenElementNames.Contains("__group") ? null : detail.Group, + PrecisionLocation = overriddenElementNames.Contains("precisionlocation") ? null : detail.PrecisionLocation, + Status = overriddenElementNames.Contains("status") ? null : detail.Status, + Takv = overriddenElementNames.Contains("takv") ? null : detail.Takv, + Track = overriddenElementNames.Contains("track") ? null : detail.Track, + AdditionalElements = additionalElements, + }; + } + + private static ulong ToUnixMilliseconds(DateTime value) + { + var utc = value.Kind switch + { + DateTimeKind.Utc => value, + DateTimeKind.Local => value.ToUniversalTime(), + _ => DateTime.SpecifyKind(value, DateTimeKind.Utc), + }; + + return (ulong)new DateTimeOffset(utc).ToUnixTimeMilliseconds(); + } + + private static DateTime FromUnixMilliseconds(ulong value) + { + return DateTimeOffset.FromUnixTimeMilliseconds((long)value).UtcDateTime; + } + + private static DateTime? FromUnixMillisecondsOrDefault(ulong value) + { + return value == 0 ? null : FromUnixMilliseconds(value); + } + + private static string SerializeResidualDetail(Detail detail, out HashSet overriddenElementNames) + { + var residualElements = new List(); + + if (detail.AdditionalElements != null && detail.AdditionalElements.Length > 0) + { + residualElements.AddRange(detail.AdditionalElements.Where(e => e != null)); + } + + AddResidualTypedElement(residualElements, "contact", detail.Contact, HasAdditionalAttributes(detail.Contact?.AdditionalAttributes), WriteContactAttributes); + AddResidualTypedElement(residualElements, "__group", detail.Group, HasAdditionalAttributes(detail.Group?.AdditionalAttributes), WriteGroupAttributes); + AddResidualTypedElement(residualElements, "precisionlocation", detail.PrecisionLocation, HasAdditionalAttributes(detail.PrecisionLocation?.AdditionalAttributes), WritePrecisionLocationAttributes); + AddResidualTypedElement(residualElements, "status", detail.Status, HasAdditionalAttributes(detail.Status?.AdditionalAttributes), WriteStatusAttributes); + AddResidualTypedElement(residualElements, "takv", detail.Takv, HasAdditionalAttributes(detail.Takv?.AdditionalAttributes), WriteTakvAttributes); + AddResidualTypedElement(residualElements, "track", detail.Track, HasAdditionalAttributes(detail.Track?.AdditionalAttributes), WriteTrackAttributes); + + overriddenElementNames = GetOverrideElementNames(residualElements.ToArray()); + + if (residualElements.Count > 0) + { + return string.Concat(residualElements.Select(e => e.OuterXml)); + } + + return detail.xmlDetail ?? ""; + } + + private static bool HasAdditionalAttributes(XmlAttribute[] attributes) + { + return attributes != null && attributes.Length > 0; + } + + private static void AddResidualTypedElement(List residualElements, string elementName, T element, bool keepAsXml, Action writeAttributes) + where T : class + { + if (!keepAsXml || element == null) + { + return; + } + + var document = new XmlDocument(); + var xmlElement = document.CreateElement(elementName); + writeAttributes(xmlElement, element); + residualElements.Add(xmlElement); + } + + private static void WriteContactAttributes(XmlElement element, Contact contact) + { + WriteStringAttribute(element, "endpoint", contact.Endpoint); + WriteStringAttribute(element, "callsign", contact.Callsign); + WriteAdditionalAttributes(element, contact.AdditionalAttributes); + } + + private static void WriteGroupAttributes(XmlElement element, Group group) + { + WriteStringAttribute(element, "name", group.Name); + WriteStringAttribute(element, "role", group.Role); + WriteAdditionalAttributes(element, group.AdditionalAttributes); + } + + private static void WritePrecisionLocationAttributes(XmlElement element, PrecisionLocation precisionLocation) + { + WriteStringAttribute(element, "geopointsrc", precisionLocation.Geopointsrc); + WriteStringAttribute(element, "altsrc", precisionLocation.Altsrc); + WriteAdditionalAttributes(element, precisionLocation.AdditionalAttributes); + } + + private static void WriteStatusAttributes(XmlElement element, Status status) + { + element.SetAttribute("battery", XmlConvert.ToString(status.Battery)); + WriteAdditionalAttributes(element, status.AdditionalAttributes); + } + + private static void WriteTakvAttributes(XmlElement element, Takv takv) + { + WriteStringAttribute(element, "device", takv.Device); + WriteStringAttribute(element, "platform", takv.Platform); + WriteStringAttribute(element, "os", takv.Os); + WriteStringAttribute(element, "version", takv.Version); + WriteAdditionalAttributes(element, takv.AdditionalAttributes); + } + + private static void WriteTrackAttributes(XmlElement element, Track track) + { + element.SetAttribute("speed", XmlConvert.ToString(track.Speed)); + element.SetAttribute("course", XmlConvert.ToString(track.Course)); + WriteAdditionalAttributes(element, track.AdditionalAttributes); + } + + private static void WriteStringAttribute(XmlElement element, string name, string value) + { + if (!string.IsNullOrEmpty(value)) + { + element.SetAttribute(name, value); + } + } + + private static void WriteAdditionalAttributes(XmlElement element, XmlAttribute[] additionalAttributes) + { + if (additionalAttributes == null) + { + return; + } + + foreach (var attribute in additionalAttributes) + { + if (attribute == null) + { + continue; + } + + var imported = (XmlAttribute)element.OwnerDocument.ImportNode(attribute, deep: true); + element.Attributes.Append(imported); + } + } + + private static XmlElement[] ParseResidualDetail(string xmlDetail) + { + if (string.IsNullOrWhiteSpace(xmlDetail)) + { + return Array.Empty(); + } + + var doc = new XmlDocument(); + doc.LoadXml($"{xmlDetail}"); + + return doc.DocumentElement? + .ChildNodes + .OfType() + .Select(e => (XmlElement)e.CloneNode(deep: true)) + .ToArray() ?? Array.Empty(); + } + + private static System.Collections.Generic.HashSet GetOverrideElementNames(XmlElement[] additionalElements) + { + var names = new System.Collections.Generic.HashSet(StringComparer.Ordinal); + + if (additionalElements == null) + { + return names; + } + + foreach (var element in additionalElements) + { + if (element != null) + { + names.Add(element.Name); + } + } + + return names; + } + } +} diff --git a/dpp.cot/Detail.cs b/dpp.cot/Detail.cs index f9ea003..35019e0 100644 --- a/dpp.cot/Detail.cs +++ b/dpp.cot/Detail.cs @@ -1,47 +1,32 @@ -using ProtoBuf; -using System; -using System.Collections.Generic; -using System.ComponentModel; -using System.Linq; -using System.Text; -using System.Threading.Tasks; +using System.Xml; using System.Xml.Serialization; namespace dpp.cot { - [ProtoContract] - public class Detail : IExtensible + public class Detail { - private IExtension __pbn__extensionData; - IExtension IExtensible.GetExtensionObject(bool createIfMissing) - => Extensible.GetExtensionObject(ref __pbn__extensionData, createIfMissing); - - [ProtoMember(1)] - [DefaultValue("")] + // Residual detail XML used by the protobuf mapping layer. public string xmlDetail { get; set; } = ""; - [ProtoMember(2, Name = @"contact")] [XmlElement(ElementName = "contact", IsNullable = true)] public Contact Contact { get; set; } - [ProtoMember(3, Name = @"group")] [XmlElement(ElementName = "__group", IsNullable = true)] public Group Group { get; set; } - [ProtoMember(4)] [XmlElement(ElementName = @"precisionlocation", IsNullable = true)] public PrecisionLocation PrecisionLocation { get; set; } - [ProtoMember(5, Name = @"status")] [XmlElement(ElementName = "status", IsNullable = true)] public Status Status { get; set; } - [ProtoMember(6, Name = @"takv")] [XmlElement(ElementName = "takv", IsNullable = true)] public Takv Takv { get; set; } - [ProtoMember(7, Name = @"track")] [XmlElement(ElementName = "track", IsNullable = true)] public Track Track { get; set; } + + [XmlAnyElement] + public XmlElement[] AdditionalElements { get; set; } = System.Array.Empty(); } } diff --git a/dpp.cot/Event.cs b/dpp.cot/Event.cs index c4a23dd..191340c 100644 --- a/dpp.cot/Event.cs +++ b/dpp.cot/Event.cs @@ -1,82 +1,70 @@ -using ProtoBuf; -using System; +using System; using System.ComponentModel; using System.IO; -using System.Linq; using System.Text; -using System.Text.RegularExpressions; using System.Xml; using System.Xml.Serialization; namespace dpp.cot { - [ProtoContract] [XmlRoot(ElementName = "event")] - public partial class Event : IExtensible + public partial class Event { - private IExtension __pbn__extensionData; - IExtension IExtensible.GetExtensionObject(bool createIfMissing) - => Extensible.GetExtensionObject(ref __pbn__extensionData, createIfMissing); - [XmlAttribute(AttributeName = "version")] public string Version { get; set; } - [ProtoMember(1, Name = @"type")] [DefaultValue("")] [XmlAttribute(AttributeName = "type")] public string Type { get; set; } = ""; - [ProtoMember(2, Name = @"access")] [DefaultValue("")] [XmlAttribute(AttributeName = "access")] public string Access { get; set; } = ""; - [ProtoMember(3, Name = @"qos")] [DefaultValue("")] [XmlAttribute(AttributeName = "qos")] public string Qos { get; set; } = ""; - [ProtoMember(4, Name = @"opex")] [DefaultValue("")] [XmlAttribute(AttributeName = "opex")] public string Opex { get; set; } = ""; - [ProtoMember(5, Name = @"uid")] + [DefaultValue("")] + [XmlAttribute(AttributeName = "caveat")] + public string Caveat { get; set; } = ""; + + [DefaultValue("")] + [XmlAttribute(AttributeName = "releasableTo")] + public string ReleaseableTo { get; set; } = ""; + [DefaultValue("")] [XmlAttribute(AttributeName = "uid")] public string Uid { get; set; } = ""; - [ProtoMember(6)] - private ulong _time; private DateTime time; [XmlAttribute(AttributeName = "time")] public DateTime Time { get { return time; } - set { time = value; _time = (ulong)time.Ticks; } + set { time = value; } } - [ProtoMember(7)] - private ulong _start; private DateTime start; [XmlAttribute(AttributeName = "start")] public DateTime Start { get { return start; } - set { start = value; _start = (ulong)start.Ticks; } + set { start = value; } } - [ProtoMember(8)] - private ulong _stale; private DateTime stale; [XmlAttribute(AttributeName = "stale")] public DateTime Stale { get { return stale; } - set { stale = value; _stale = (ulong)stale.Ticks; } + set { stale = value; } } - [ProtoMember(9, Name = @"how")] [DefaultValue("")] [XmlAttribute(AttributeName = "how")] public string How { get; set; } = ""; @@ -90,12 +78,13 @@ public DateTime Stale public Event() { Point = new Point(); - Time = DateTime.Now; - Start = DateTime.Now; - Stale = DateTime.Now.AddMinutes(5); + var now = DateTime.UtcNow; + Time = now; + Start = now; + Stale = now.AddMinutes(5); } - public static Event Pong(Event? ping) + public static Event Pong(Event ping = null) { var e = ping ?? new Event(); e.Uid ??= Guid.NewGuid().ToString(); @@ -118,34 +107,170 @@ public static Event Parse(byte[] payload, int offset, int length) public String ToXmlString() { - // empty namespaces to force serializer to omit them - var ns = new XmlSerializerNamespaces(new[] { XmlQualifiedName.Empty }); - var settings = new XmlWriterSettings() { Indent = false, OmitXmlDeclaration = true, - ConformanceLevel = ConformanceLevel.Auto, + ConformanceLevel = ConformanceLevel.Document, }; - using MemoryStream ms = new(); - using (XmlWriter writer = XmlWriter.Create(ms, settings)) - { - var serializer = new XmlSerializer(typeof(Event), ""); - serializer.Serialize(writer, this, ns); - } + using var sw = new Utf8StringWriter(); + using var writer = XmlWriter.Create(sw, settings); + + writer.WriteStartElement("event"); + WriteOptionalAttribute(writer, "version", Version); + WriteOptionalAttribute(writer, "uid", Uid); + WriteOptionalAttribute(writer, "type", Type); + WriteOptionalAttribute(writer, "access", Access); + WriteOptionalAttribute(writer, "qos", Qos); + WriteOptionalAttribute(writer, "opex", Opex); + WriteOptionalAttribute(writer, "caveat", Caveat); + WriteOptionalAttribute(writer, "releasableTo", ReleaseableTo); + writer.WriteAttributeString("time", XmlConvert.ToString(ToUtc(Time), XmlDateTimeSerializationMode.Utc)); + writer.WriteAttributeString("start", XmlConvert.ToString(ToUtc(Start), XmlDateTimeSerializationMode.Utc)); + writer.WriteAttributeString("stale", XmlConvert.ToString(ToUtc(Stale), XmlDateTimeSerializationMode.Utc)); + WriteOptionalAttribute(writer, "how", How); - var encoding = new UTF8Encoding(); - var result = encoding.GetString(ms.ToArray()); + WritePoint(writer, Point ?? new Point()); + WriteDetail(writer, Detail); - // fix BOM, self closing tags quirk, and namespace from default values - result = result.Replace("\ufeff", ""); - result = result.Replace("", ""); - result = Regex.Replace(result, @"(xmlns:)?p3(:nil)?="".+?\""", ""); + writer.WriteEndElement(); + writer.Flush(); - return result; + return sw.ToString(); } - } -} + private static void WriteOptionalAttribute(XmlWriter writer, string name, string value) + { + if (!string.IsNullOrEmpty(value)) + { + writer.WriteAttributeString(name, value); + } + } + + private static void WritePoint(XmlWriter writer, Point point) + { + writer.WriteStartElement("point"); + writer.WriteAttributeString("lat", XmlConvert.ToString(point.Lat)); + writer.WriteAttributeString("lon", XmlConvert.ToString(point.Lon)); + writer.WriteAttributeString("hae", XmlConvert.ToString(point.Hae)); + writer.WriteAttributeString("ce", XmlConvert.ToString(point.Ce)); + writer.WriteAttributeString("le", XmlConvert.ToString(point.Le)); + writer.WriteEndElement(); + } + + private static void WriteDetail(XmlWriter writer, Detail detail) + { + if (detail == null) + { + return; + } + + writer.WriteStartElement("detail"); + + if (detail.Takv != null) + { + writer.WriteStartElement("takv"); + WriteOptionalAttribute(writer, "device", detail.Takv.Device); + WriteOptionalAttribute(writer, "platform", detail.Takv.Platform); + WriteOptionalAttribute(writer, "os", detail.Takv.Os); + WriteOptionalAttribute(writer, "version", detail.Takv.Version); + WriteAdditionalAttributes(writer, detail.Takv.AdditionalAttributes); + writer.WriteEndElement(); + } + + if (detail.Contact != null) + { + writer.WriteStartElement("contact"); + WriteOptionalAttribute(writer, "endpoint", detail.Contact.Endpoint); + WriteOptionalAttribute(writer, "callsign", detail.Contact.Callsign); + WriteAdditionalAttributes(writer, detail.Contact.AdditionalAttributes); + writer.WriteEndElement(); + } + + if (detail.PrecisionLocation != null) + { + writer.WriteStartElement("precisionlocation"); + WriteOptionalAttribute(writer, "geopointsrc", detail.PrecisionLocation.Geopointsrc); + WriteOptionalAttribute(writer, "altsrc", detail.PrecisionLocation.Altsrc); + WriteAdditionalAttributes(writer, detail.PrecisionLocation.AdditionalAttributes); + writer.WriteEndElement(); + } + + if (detail.Group != null) + { + writer.WriteStartElement("__group"); + WriteOptionalAttribute(writer, "name", detail.Group.Name); + WriteOptionalAttribute(writer, "role", detail.Group.Role); + WriteAdditionalAttributes(writer, detail.Group.AdditionalAttributes); + writer.WriteEndElement(); + } + if (detail.Status != null) + { + writer.WriteStartElement("status"); + writer.WriteAttributeString("battery", XmlConvert.ToString(detail.Status.Battery)); + WriteAdditionalAttributes(writer, detail.Status.AdditionalAttributes); + writer.WriteEndElement(); + } + + if (detail.Track != null) + { + writer.WriteStartElement("track"); + writer.WriteAttributeString("speed", XmlConvert.ToString(detail.Track.Speed)); + writer.WriteAttributeString("course", XmlConvert.ToString(detail.Track.Course)); + WriteAdditionalAttributes(writer, detail.Track.AdditionalAttributes); + writer.WriteEndElement(); + } + + if (detail.AdditionalElements != null) + { + foreach (var element in detail.AdditionalElements) + { + element.WriteTo(writer); + } + } + + if (!string.IsNullOrWhiteSpace(detail.xmlDetail) && + (detail.AdditionalElements == null || detail.AdditionalElements.Length == 0)) + { + writer.WriteRaw(detail.xmlDetail); + } + + writer.WriteFullEndElement(); + } + + private static void WriteAdditionalAttributes(XmlWriter writer, XmlAttribute[] attributes) + { + if (attributes == null) + { + return; + } + + foreach (var attribute in attributes) + { + if (attribute == null) + { + continue; + } + + writer.WriteAttributeString(attribute.Prefix, attribute.LocalName, attribute.NamespaceURI, attribute.Value); + } + } + + private static DateTime ToUtc(DateTime value) + { + return value.Kind switch + { + DateTimeKind.Utc => value, + DateTimeKind.Local => value.ToUniversalTime(), + _ => DateTime.SpecifyKind(value, DateTimeKind.Utc), + }; + } + + private sealed class Utf8StringWriter : StringWriter + { + public override Encoding Encoding => Encoding.UTF8; + } + } +} diff --git a/dpp.cot/Group.cs b/dpp.cot/Group.cs index 11d3aa3..ca0587f 100644 --- a/dpp.cot/Group.cs +++ b/dpp.cot/Group.cs @@ -5,6 +5,7 @@ using System.Linq; using System.Text; using System.Threading.Tasks; +using System.Xml; using System.Xml.Serialization; namespace dpp.cot @@ -26,5 +27,8 @@ IExtension IExtensible.GetExtensionObject(bool createIfMissing) [XmlAttribute(AttributeName = @"role")] public string Role { get; set; } = ""; + [XmlAnyAttribute] + public XmlAttribute[] AdditionalAttributes { get; set; } = System.Array.Empty(); + } } diff --git a/dpp.cot/Message.cs b/dpp.cot/Message.cs index 309e43b..af3580b 100644 --- a/dpp.cot/Message.cs +++ b/dpp.cot/Message.cs @@ -1,15 +1,12 @@ using ProtoBuf; using System; -using System.Collections.Generic; using System.IO; using System.Linq; using System.Text; -using System.Threading.Tasks; namespace dpp.cot { - [ProtoContract()] - public partial class Message : IExtensible + public partial class Message { // versioning bytes for header // magic|version|magic|data @@ -17,27 +14,54 @@ public partial class Message : IExtensible private const byte v0_xml = 0x00; private const byte v1_protobuf = 0x01; - private IExtension __pbn__extensionData; - IExtension IExtensible.GetExtensionObject(bool createIfMissing) - => Extensible.GetExtensionObject(ref __pbn__extensionData, createIfMissing); - - [ProtoMember(1)] public TakControl Control { get; set; } - [ProtoMember(2)] public Event Event { get; set; } + public DateTime? SubmissionTime { get; set; } + + public DateTime? CreationTime { get; set; } + public static Message Parse(byte[] data, int offset, int length) { + if (data == null) + { + throw new ArgumentNullException(nameof(data)); + } + + if (offset < 0 || length < 0 || offset + length > data.Length) + { + throw new ArgumentOutOfRangeException($"Invalid segment. offset={offset}, length={length}, data.Length={data.Length}"); + } + + if (length == 0) + { + throw new ArgumentException("Message payload cannot be empty.", nameof(length)); + } + string msg; Event e; - if (data[0] == magic && data[2] == magic) + if (length >= 3 && data[offset] == magic) { - switch (data[1]) + if (!TryReadUnsignedVarint(data, offset + 1, length - 1, out var version, out var versionBytes)) + { + throw new ArgumentException("Incomplete TAK mesh protocol header.", nameof(length)); + } + + var trailingMagicOffset = offset + 1 + versionBytes; + if (trailingMagicOffset >= offset + length || data[trailingMagicOffset] != magic) + { + throw new NotImplementedException("Failed to find TAK mesh trailing magic byte."); + } + + var payloadOffset = trailingMagicOffset + 1; + var payloadLength = (offset + length) - payloadOffset; + + switch (version) { case v0_xml: - msg = Encoding.UTF8.GetString(data, (int)offset + 3, (int)length); + msg = Encoding.UTF8.GetString(data, payloadOffset, payloadLength); e = Event.Parse(msg); return new Message @@ -47,18 +71,18 @@ public static Message Parse(byte[] data, int offset, int length) { minProtoVersion = v0_xml, maxProtoVersion = v0_xml, - contactUid = e.Uid, } }; case v1_protobuf: - using (var ms = new MemoryStream(data, offset + 3, length)) + using (var ms = new MemoryStream(data, payloadOffset, payloadLength, writable: false)) { - return ProtoBuf.Serializer.Deserialize(ms); + var proto = Serializer.Deserialize(ms); + return CotProtoMapper.FromProto(proto); } default: - throw new NotImplementedException($"Unknown protocol version. Version={data[1]:X}"); + throw new NotImplementedException($"Unknown protocol version. Version={version:X}"); } } @@ -73,7 +97,6 @@ public static Message Parse(byte[] data, int offset, int length) { minProtoVersion = 0, maxProtoVersion = 0, - contactUid = e.Uid, } }; } @@ -85,14 +108,83 @@ public string ToXmlString() public byte[] ToXmlBytes() { - var header = new byte[]{ - magic, - v0_xml, - magic, - }; + var header = BuildMeshHeader(v0_xml); var data = Encoding.UTF8.GetBytes(this.Event.ToXmlString()); return header.Concat(data).ToArray(); } + + public byte[] ToProtobufBytes() + { + var header = BuildMeshHeader(v1_protobuf); + + using var ms = new MemoryStream(); + Serializer.Serialize(ms, CotProtoMapper.ToProto(this)); + + return header.Concat(ms.ToArray()).ToArray(); + } + + public byte[] ToStreamingBytes(byte protocolVersion) + { + return TakProtocolStreaming.ToStreamBytes(this, protocolVersion); + } + + public static Message ParseStreaming(byte[] data, int offset, int length, byte protocolVersion) + { + return TakProtocolStreaming.ParseStreamMessage(data, offset, length, protocolVersion); + } + + public static bool TryParseStreaming(byte[] data, int offset, int length, byte protocolVersion, out Message message, out int bytesConsumed) + { + return TakProtocolStreaming.TryParseStreamMessage(data, offset, length, protocolVersion, out message, out bytesConsumed); + } + + private static byte[] BuildMeshHeader(byte version) + { + using var ms = new MemoryStream(); + ms.WriteByte(magic); + WriteUnsignedVarint(ms, version); + ms.WriteByte(magic); + return ms.ToArray(); + } + + private static void WriteUnsignedVarint(Stream stream, ulong value) + { + while (value >= 0x80) + { + stream.WriteByte((byte)((value & 0x7f) | 0x80)); + value >>= 7; + } + + stream.WriteByte((byte)value); + } + + private static bool TryReadUnsignedVarint(byte[] data, int offset, int length, out ulong value, out int bytesRead) + { + value = 0; + bytesRead = 0; + const int maxVarintBytes = 10; + + while (bytesRead < length && bytesRead < maxVarintBytes) + { + var current = data[offset + bytesRead]; + value |= (ulong)(current & 0x7f) << (7 * bytesRead); + bytesRead++; + + if ((current & 0x80) == 0) + { + return true; + } + } + + if (bytesRead == maxVarintBytes && length >= maxVarintBytes) + { + throw new FormatException("Invalid TAK mesh varint header."); + } + + value = 0; + bytesRead = 0; + return false; + } } } diff --git a/dpp.cot/Point.cs b/dpp.cot/Point.cs index 87c072c..8df441e 100644 --- a/dpp.cot/Point.cs +++ b/dpp.cot/Point.cs @@ -1,33 +1,22 @@ -using ProtoBuf; -using System; -using System.Collections.Generic; -using System.Linq; -using System.Text; -using System.Threading.Tasks; -using System.Xml.Serialization; +using System.Xml.Serialization; namespace dpp.cot { [XmlRoot(ElementName = "point")] public class Point { - [ProtoMember(10, Name = @"lat")] [XmlAttribute(AttributeName = "lat")] public double Lat { get; set; } = 0; - [ProtoMember(11, Name = @"lon")] [XmlAttribute(AttributeName = "lon")] public double Lon { get; set; } = 0; - [ProtoMember(12, Name = @"ce")] [XmlAttribute(AttributeName = "ce")] public double Ce { get; set; } = 9999999.0; - [ProtoMember(13, Name = @"hae")] [XmlAttribute(AttributeName = "hae")] public double Hae { get; set; } = 9999999.0; - [ProtoMember(14, Name = @"le")] [XmlAttribute(AttributeName = "le")] public double Le { get; set; } = 9999999.0; } diff --git a/dpp.cot/PrecisionLocation.cs b/dpp.cot/PrecisionLocation.cs index 7dbe4d3..1bdb9c5 100644 --- a/dpp.cot/PrecisionLocation.cs +++ b/dpp.cot/PrecisionLocation.cs @@ -5,6 +5,7 @@ using System.Linq; using System.Text; using System.Threading.Tasks; +using System.Xml; using System.Xml.Serialization; namespace dpp.cot @@ -26,5 +27,8 @@ IExtension IExtensible.GetExtensionObject(bool createIfMissing) [XmlAttribute(AttributeName = "altsrc")] public string Altsrc { get; set; } = ""; + [XmlAnyAttribute] + public XmlAttribute[] AdditionalAttributes { get; set; } = System.Array.Empty(); + } } diff --git a/dpp.cot/Properties/AssemblyInfo.cs b/dpp.cot/Properties/AssemblyInfo.cs new file mode 100644 index 0000000..3501831 --- /dev/null +++ b/dpp.cot/Properties/AssemblyInfo.cs @@ -0,0 +1,3 @@ +using System.Runtime.CompilerServices; + +[assembly: InternalsVisibleTo("dpp.cot.Tests")] diff --git a/dpp.cot/ProtoContracts.cs b/dpp.cot/ProtoContracts.cs new file mode 100644 index 0000000..81d048f --- /dev/null +++ b/dpp.cot/ProtoContracts.cs @@ -0,0 +1,111 @@ +using ProtoBuf; + +namespace dpp.cot +{ + [ProtoContract] + internal sealed class ProtoTakMessage + { + [ProtoMember(1, Name = @"takControl")] + public ProtoTakControl TakControl { get; set; } + + [ProtoMember(2, Name = @"cotEvent")] + public ProtoCotEvent CotEvent { get; set; } + + [ProtoMember(3, Name = @"submissionTime")] + public ulong SubmissionTime { get; set; } + + [ProtoMember(4, Name = @"creationTime")] + public ulong CreationTime { get; set; } + } + + [ProtoContract] + internal sealed class ProtoTakControl + { + [ProtoMember(1)] + public uint MinProtoVersion { get; set; } + + [ProtoMember(2)] + public uint MaxProtoVersion { get; set; } + + } + + [ProtoContract] + internal sealed class ProtoCotEvent + { + [ProtoMember(1, Name = @"type")] + public string Type { get; set; } = ""; + + [ProtoMember(2, Name = @"access")] + public string Access { get; set; } = ""; + + [ProtoMember(3, Name = @"qos")] + public string Qos { get; set; } = ""; + + [ProtoMember(4, Name = @"opex")] + public string Opex { get; set; } = ""; + + [ProtoMember(16, Name = @"caveat")] + public string Caveat { get; set; } = ""; + + [ProtoMember(17, Name = @"releaseableTo")] + public string ReleaseableTo { get; set; } = ""; + + [ProtoMember(5, Name = @"uid")] + public string Uid { get; set; } = ""; + + [ProtoMember(6, Name = @"sendTime")] + public ulong SendTime { get; set; } + + [ProtoMember(7, Name = @"startTime")] + public ulong StartTime { get; set; } + + [ProtoMember(8, Name = @"staleTime")] + public ulong StaleTime { get; set; } + + [ProtoMember(9, Name = @"how")] + public string How { get; set; } = ""; + + [ProtoMember(10, Name = @"lat")] + public double Lat { get; set; } + + [ProtoMember(11, Name = @"lon")] + public double Lon { get; set; } + + [ProtoMember(12, Name = @"hae")] + public double Hae { get; set; } + + [ProtoMember(13, Name = @"ce")] + public double Ce { get; set; } + + [ProtoMember(14, Name = @"le")] + public double Le { get; set; } + + [ProtoMember(15, Name = @"detail")] + public ProtoDetail Detail { get; set; } + } + + [ProtoContract] + internal sealed class ProtoDetail + { + [ProtoMember(1, Name = @"xmlDetail")] + public string XmlDetail { get; set; } = ""; + + [ProtoMember(2, Name = @"contact")] + public Contact Contact { get; set; } + + [ProtoMember(3, Name = @"group")] + public Group Group { get; set; } + + [ProtoMember(4, Name = @"precisionLocation")] + public PrecisionLocation PrecisionLocation { get; set; } + + [ProtoMember(5, Name = @"status")] + public Status Status { get; set; } + + [ProtoMember(6, Name = @"takv")] + public Takv Takv { get; set; } + + [ProtoMember(7, Name = @"track")] + public Track Track { get; set; } + } +} diff --git a/dpp.cot/Status.cs b/dpp.cot/Status.cs index 4034cc4..ad47238 100644 --- a/dpp.cot/Status.cs +++ b/dpp.cot/Status.cs @@ -4,6 +4,7 @@ using System.Linq; using System.Text; using System.Threading.Tasks; +using System.Xml; using System.Xml.Serialization; namespace dpp.cot @@ -19,5 +20,8 @@ IExtension IExtensible.GetExtensionObject(bool createIfMissing) [XmlAttribute(AttributeName = "battery")] public uint Battery { get; set; } + [XmlAnyAttribute] + public XmlAttribute[] AdditionalAttributes { get; set; } = System.Array.Empty(); + } } diff --git a/dpp.cot/TakControl.cs b/dpp.cot/TakControl.cs index 8e4ba44..4bafd47 100644 --- a/dpp.cot/TakControl.cs +++ b/dpp.cot/TakControl.cs @@ -1,5 +1,4 @@ using ProtoBuf; -using System.ComponentModel; namespace dpp.cot { @@ -15,9 +14,5 @@ IExtension IExtensible.GetExtensionObject(bool createIfMissing) [ProtoMember(2)] public uint maxProtoVersion { get; set; } - - [ProtoMember(3)] - [DefaultValue("")] - public string contactUid { get; set; } = ""; } -} \ No newline at end of file +} diff --git a/dpp.cot/TakProtocolStreaming.cs b/dpp.cot/TakProtocolStreaming.cs new file mode 100644 index 0000000..cfb5071 --- /dev/null +++ b/dpp.cot/TakProtocolStreaming.cs @@ -0,0 +1,172 @@ +using ProtoBuf; +using System; +using System.IO; +using System.Text; + +namespace dpp.cot +{ + public static class TakProtocolStreaming + { + private const byte StreamingMagic = 0xbf; + private const byte XmlVersion = 0x00; + private const byte ProtobufVersion = 0x01; + + public static byte[] ToStreamBytes(Message message, byte protocolVersion) + { + if (message == null) + { + throw new ArgumentNullException(nameof(message)); + } + + var payload = protocolVersion switch + { + XmlVersion => Encoding.UTF8.GetBytes(message.ToXmlString()), + ProtobufVersion => SerializeProtobufPayload(message), + _ => throw new NotImplementedException($"Unknown protocol version. Version={protocolVersion:X}"), + }; + + using var ms = new MemoryStream(); + ms.WriteByte(StreamingMagic); + WriteUnsignedVarint(ms, (ulong)payload.Length); + ms.Write(payload, 0, payload.Length); + return ms.ToArray(); + } + + public static Message ParseStreamMessage(byte[] data, int offset, int length, byte protocolVersion) + { + if (!TryParseStreamMessage(data, offset, length, protocolVersion, out var message, out var bytesConsumed)) + { + throw new ArgumentException("Incomplete streaming message.", nameof(length)); + } + + if (bytesConsumed != length) + { + throw new ArgumentException("The supplied segment contains extra bytes beyond a single streaming message.", nameof(length)); + } + + return message; + } + + public static bool TryParseStreamMessage(byte[] data, int offset, int length, byte protocolVersion, out Message message, out int bytesConsumed) + { + message = null; + bytesConsumed = 0; + + if (data == null) + { + throw new ArgumentNullException(nameof(data)); + } + + if (offset < 0 || length < 0 || offset + length > data.Length) + { + throw new ArgumentOutOfRangeException($"Invalid segment. offset={offset}, length={length}, data.Length={data.Length}"); + } + + if (length == 0) + { + return false; + } + + if (data[offset] != StreamingMagic) + { + throw new NotImplementedException($"Unknown stream magic byte. Value={data[offset]:X}"); + } + + if (!TryReadUnsignedVarint(data, offset + 1, length - 1, out var payloadLength, out var headerBytes)) + { + return false; + } + + if (payloadLength > int.MaxValue) + { + throw new NotSupportedException($"Streaming payload too large. Length={payloadLength}"); + } + + var totalLength = 1 + headerBytes + (int)payloadLength; + if (totalLength > length) + { + return false; + } + + var payloadOffset = offset + 1 + headerBytes; + message = protocolVersion switch + { + XmlVersion => ParseXmlPayload(data, payloadOffset, (int)payloadLength), + ProtobufVersion => ParseProtobufPayload(data, payloadOffset, (int)payloadLength), + _ => throw new NotImplementedException($"Unknown protocol version. Version={protocolVersion:X}"), + }; + + bytesConsumed = totalLength; + return true; + } + + private static Message ParseXmlPayload(byte[] data, int offset, int length) + { + var evt = Event.Parse(data, offset, length); + return new Message + { + Event = evt, + Control = new TakControl + { + minProtoVersion = XmlVersion, + maxProtoVersion = XmlVersion, + } + }; + } + + private static Message ParseProtobufPayload(byte[] data, int offset, int length) + { + using var ms = new MemoryStream(data, offset, length, writable: false); + var proto = Serializer.Deserialize(ms); + return CotProtoMapper.FromProto(proto); + } + + private static byte[] SerializeProtobufPayload(Message message) + { + using var ms = new MemoryStream(); + Serializer.Serialize(ms, CotProtoMapper.ToProto(message)); + return ms.ToArray(); + } + + private static void WriteUnsignedVarint(Stream stream, ulong value) + { + while (value >= 0x80) + { + stream.WriteByte((byte)(value | 0x80)); + value >>= 7; + } + + stream.WriteByte((byte)value); + } + + private static bool TryReadUnsignedVarint(byte[] data, int offset, int length, out ulong value, out int bytesRead) + { + value = 0; + bytesRead = 0; + + const int maxVarintBytes = 10; + + while (bytesRead < length && bytesRead < maxVarintBytes) + { + var current = data[offset + bytesRead]; + var chunk = (ulong)(current & 0x7f); + value |= chunk << (7 * bytesRead); + bytesRead++; + + if ((current & 0x80) == 0) + { + return true; + } + } + + if (bytesRead == maxVarintBytes && length >= maxVarintBytes) + { + throw new FormatException("Invalid TAK streaming varint header."); + } + + value = 0; + bytesRead = 0; + return false; + } + } +} diff --git a/dpp.cot/Takv.cs b/dpp.cot/Takv.cs index f1a3be6..8da2fe5 100644 --- a/dpp.cot/Takv.cs +++ b/dpp.cot/Takv.cs @@ -1,5 +1,6 @@ using ProtoBuf; using System.ComponentModel; +using System.Xml; using System.Xml.Serialization; namespace dpp.cot @@ -30,5 +31,8 @@ IExtension IExtensible.GetExtensionObject(bool createIfMissing) [DefaultValue("")] [XmlAttribute(AttributeName = "version")] public string Version { get; set; } = ""; + + [XmlAnyAttribute] + public XmlAttribute[] AdditionalAttributes { get; set; } = System.Array.Empty(); } -} \ No newline at end of file +} diff --git a/dpp.cot/Track.cs b/dpp.cot/Track.cs index fdb5abc..e96500b 100644 --- a/dpp.cot/Track.cs +++ b/dpp.cot/Track.cs @@ -1,6 +1,21 @@ -namespace dpp.cot +using ProtoBuf; +using System.Xml; +using System.Xml.Serialization; + +namespace dpp.cot { + [ProtoContract] public class Track { + [ProtoMember(1, Name = @"speed")] + [XmlAttribute(AttributeName = "speed")] + public double Speed { get; set; } + + [ProtoMember(2, Name = @"course")] + [XmlAttribute(AttributeName = "course")] + public double Course { get; set; } + + [XmlAnyAttribute] + public XmlAttribute[] AdditionalAttributes { get; set; } = System.Array.Empty(); } -} \ No newline at end of file +} diff --git a/dpp.cot/dpp.cot.csproj b/dpp.cot/dpp.cot.csproj index e4eb932..f6dabf3 100644 --- a/dpp.cot/dpp.cot.csproj +++ b/dpp.cot/dpp.cot.csproj @@ -1,10 +1,25 @@  - net5.0 - 1.0.5.0 - 1.0.5.0 - 1.0.5 + netstandard2.0;net10.0 + 2.0.0.0 + 2.0.0.0 + 2.0.0 + disable + latest + dpp.cot + dpp.cot + darkplusplus + darkplusplus + .NET library for Cursor-on-Target XML, TAK protobuf v1, and TAK streaming envelopes. + cot;cursor-on-target;tak;protobuf;atak;geospatial + MIT + readme.md + https://github.com/darkplusplus/cot + https://github.com/darkplusplus/cot.git + git + 2.0.0: XML/protobuf model separation, TAK streaming support, public-source-backed docs, stronger compatibility coverage, and broader .NET targeting via netstandard2.0 plus net10.0. + false @@ -43,4 +58,8 @@ + + + + diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..40a47aa --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,21 @@ +site_name: dpp.cot +site_description: Documentation for the dpp.cot Cursor-on-Target library +repo_url: https://github.com/darkplusplus/cot +theme: + name: mkdocs +nav: + - Home: index.md + - Scope: scope.md + - Source Policy: source-policy.md + - Source Validation: source-validation.md + - Type Inference: type-inference.md + - Fixture Provenance: fixture-provenance.md + - Compatibility Policy: compatibility-policy.md + - Protocols: + - CoT XML Event: protocols/cot-xml.md + - TAK Protobuf Payload: protocols/tak-protobuf.md + - TAK Streaming Envelope: protocols/tak-streaming.md + - Interoperability Notes: interoperability.md + - Library Design: library-design.md + - Testing: testing.md + - Sources: sources.md diff --git a/readme.md b/readme.md index b50a7f4..824b4a8 100644 --- a/readme.md +++ b/readme.md @@ -1,3 +1,52 @@ # dpp.cot [![Status](https://github.com/darkplusplus/cot/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/darkplusplus/cot/actions/workflows/ci.yml) -a library for handling cursor-on-target messages. +A library for handling Cursor-on-Target messages. + +## Package + +NuGet package: `dpp.cot` + +```bash +dotnet add package dpp.cot +``` + +## Documentation + +This repo includes a MkDocs scaffold in [`mkdocs.yml`](mkdocs.yml) and [`docs/`](docs/index.md). + +The docs are organized around: + +- supported protocol behavior +- public source attribution +- implementation notes backed by code and tests + +## Current Coverage + +The library currently covers: + +- CoT XML `event` parsing and serialization +- TAK protobuf payload v1 mapping +- TAK streaming envelope framing + +## Target Frameworks + +- `netstandard2.0` +- `net10.0` + +## Release Notes + +See [CHANGELOG.md](CHANGELOG.md). + +## Source Policy + +Public documentation in this repo should stay grounded in: + +- public protocol artifacts committed in the repo +- public external sources explicitly linked in the docs +- implementation behavior demonstrated by tests + +See: + +- [docs/scope.md](docs/scope.md) +- [docs/source-policy.md](docs/source-policy.md) +- [docs/sources.md](docs/sources.md)