KMS-705: Correct publisher keyword-event mapping so published and draft keyword CSVs produce complete, CSV-shaped correction objects for every supported scheme. by cgokey · Pull Request #134 · nasa/kms

cgokey · 2026-08-20T21:36:19Z

Overview

What is the feature?

Correct publisher keyword-event mapping so published and draft keyword CSVs produce complete, CSV-shaped correction objects for every supported scheme.

This work began with the short-name scheme defect described in KMS-705. The shared CSV comparison treated every column before UUID as a hierarchy position. For schemes with auxiliary columns such as Long_Name or Data_Center_URL, the last auxiliary value could be emitted as ShortName while the actual Short_Name was discarded. This potentially affected platforms, instruments, projects, providers, IDN nodes, data formats, and granule data formats.

Since this was IP sprint, my goal was to test the complete workflow from published CSVs through generated native metadata rather than limiting validation to the initial publisher fix. This uncovered a broader contract issue: correction objects sometimes relied on positional assumptions or native metadata field names instead of the actual production CSV columns. The scope therefore expanded to formalize that contract and protect the full workflow with end-to-end regression tests.

For example, a GOSAT short-name update previously produced an unchanged event because Long_Name was incorrectly treated as ShortName:

{
  "Category": "Platforms",
  "Class": "Space-based Platforms",
  "Type": "Earth Observation Satellites",
  "ShortName": "Greenhouse Gases Observing Satellite"
}

The corrected event follows the platform CSV headers and preserves both values independently:

{
  "Basis": "Space-based Platforms",
  "Category": "Earth Observation Satellites",
  "SubCategory": "",
  "ShortName": "GOSAT",
  "LongName": "Greenhouse Gases Observing Satellite"
}

The draft object is identical except that ShortName is correctly set to GOSAT - Test1.

The same positional behavior could lose intermediate hierarchy values. An Amazonia platform update was recorded with a new path similar to Space-based Platforms > > Amazonia-1-Chris-Test, dropping Earth Observation Satellites. Reading the named Category column now preserves that value, while also retaining the intentionally blank Sub_Category slot.

What is the Solution?

Parse exported keyword CSVs by normalized header name instead of treating every pre-UUID column as an indexed hierarchy value.
Define the exact CSV fields and canonical order for each scheme, including production names such as Basis, Sub_Category, Short_Name, Long_Name, and Data_Center_URL.
Preserve blank hierarchy positions so later values do not shift into the wrong semantic field.
Keep auxiliary values such as LongName and DataCenterURL in correction objects without treating them as hierarchy levels.
Use one CSV field contract for publisher events, historical lookups, and correction resolution.
Separate CSV correction-object fields from native metadata fields. The UMM-C, ECHO10, DIF10, and ISO editors now explicitly translate CSV-shaped correction objects into their format-specific JSON or XML fields.
Preserve long-name-only changes even when the canonical hierarchy path itself does not change.
Add production-shaped published.csv and draft.csv fixtures for every supported keyword scheme.
Remove redundant tests while retaining focused unit coverage and the complete end-to-end regression suite.

The end-to-end tests now verify two independent boundaries:

The production-shaped published and draft CSV fixtures are compared, publisher events are created, and the resulting old/new keyword objects are resolved into corrections. Those generated corrections must match the checked-in correction JSON for each native format.
The checked-in correction JSON is independently applied to complete UMM-C, DIF10, ECHO10, ISO19115, and ISO-SMAP records. The full serialized result must exactly match the checked-in after record for both updates and deletions.

The generated corrections are deliberately not passed directly into the native transformation assertion. Keeping the two stages independent prevents a broken publisher from defining its own expected native result. It also gives us a repeatable pattern for future mapping defects: add the production-shaped CSV case, the expected correction object, and the affected native metadata field, then the test locks the reported behavior from publication through transformation.

What areas of the application does this impact?

Published-versus-draft keyword CSV comparison and event generation
Historical and published keyword cache lookups
Keyword path and correction-object construction
UMM-C, DIF10, ECHO10, ISO19115, and ISO-SMAP metadata editors
Native metadata correction fixtures and regression tests
Local publisher mapping smoke coverage

Testing

Run npm test -- --run serverless/src/shared/__tests__/applyNativeMetadataCorrectionMappings.test.js.
Confirm each native format generates the checked-in update and deletion corrections from the published/draft CSV fixtures.
Confirm those checked-in corrections produce an exact match with each complete expected native metadata record.
Run npm test -- --run and confirm the full suite passes. The current result is 226 test files and 2,296 tests passing.
Run npm run lint and confirm ESLint passes.
In VS Code, select a before native metadata fixture for comparison, then compare it with the corresponding after fixture under serverless/src/shared/__mocks__/native_metadata_correction_mappings/end_to_end.
Verify _CHANGED values appear only in intended mapped fields, _SHOULD_DELETE values are absent from deletion results, and unrelated metadata is unchanged.
Review generated/short-name-events.json and confirm short-name schemes preserve the true short name plus auxiliary long-name or provider URL values.

Attachments

The production-shaped CSV inputs, expected publisher events, correction objects, and complete before/after native records are included under serverless/src/shared/__mocks__/native_metadata_correction_mappings/end_to_end.

Checklist

I have added automated tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings

…etadata

…ft keyword CSVs produce complete, CSV-shaped correction objects for every supported scheme.

coderabbitai · 2026-08-20T21:36:29Z

Important

🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: 63009893-0db2-4a77-9f6e-9fbdd9e6e8cb

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

codecov-commenter · 2026-08-20T21:38:49Z

Codecov Report

❌ Patch coverage is 98.88268% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.64%. Comparing base (28d0069) to head (9a38be7).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
serverless/src/shared/Iso19115DomEditor.js	97.36%	1 Missing ⚠️
.../src/shared/redis-path-store/helpers/keywordCsv.js	98.80%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #134      +/-   ##
==========================================
- Coverage   99.73%   99.64%   -0.09%     
==========================================
  Files         234      236       +2     
  Lines        6391     6512     +121     
  Branches     1907     1909       +2     
==========================================
+ Hits         6374     6489     +115     
- Misses         16       21       +5     
- Partials        1        2       +1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

eudoroolivares2016 · 2026-08-28T19:10:33Z

      }
    },
    {
      "scheme": "instruments",


isn't this one missing longname?

https://github.com/nasa/kms/pull/134/changes#diff-fafcaa8b6f2d2555d8bd94f78365cdd9fb6425d9d161bc36fa68c742f51bd109R2

LongName is read from the CSV, but it is not part of the hierarchy-based oldKeywordObject; long names are handled separately as oldLongName and newLongName. This is a deletion, so DIF10 only needs the hierarchy and ShortName to identify and remove the instrument; replacement corrections provide newLongName separately when <Long_Name> must be updated.

eudoroolivares2016 · 2026-08-28T22:09:44Z

+  idnnode: ['ShortName'],
+  dataformat: ['ShortName'],
+  granuledataformat: ['ShortName'],
+  discipline: ['DisciplineName', 'Subdiscipline'],


This doesn't have a csv pairing but, also I don't think its actually used in CMR at all anyways

It does have a CSV definition: Discipline_Name,Subdiscipline,UUID, which normalizes to these object keys. You’re right that CMR does not currently consume discipline corrections; this entry only ensures the publisher would construct a correctly shaped event if that scheme is processed.

eudoroolivares2016 · 2026-08-28T22:11:20Z

+  verticalresolutionrange: ['VerticalResolutionRange'],
+  horizontalresolutionrange: ['HorizontalResolutionRange'],
+  productlevelid: ['ProductLevelId'],
+  measurementname: ['ContextMedium', 'Object', 'Quantity']


This one is only used in umm-var so I don't think it needs the csv either

Although MeasurementName is not currently used in CMR collections, it has the CSV columns Context_Medium,Object,Quantity,UUID. This mapping ensures those CSV fields are represented correctly in generated correction objects if collection support is added later.

eudoroolivares2016 · 2026-08-28T22:12:00Z

+ *
+ * @type {Readonly<Record<string, string|string[]>>}
+ */
+export const UMMC_FIELDS = Object.freeze({


This is fine for now we might wanna figure out a common interface with CMR that we can pull from because if they change this we could silently drift really easily

I agree, a shared interface with CMR would be a good longer-term solution because it would establish one source of truth for native field names and prevent the KMS mappings from silently drifting when CMR changes its schemas.

eudoroolivares2016 · 2026-08-28T22:16:49Z

+    'CategoryKeyword',
+    'TopicKeyword',
+    'TermKeyword',
+    'VariableLevel1Keyword/Value',


I also saw just a regular VariableLevel1Keyword in there without the /value

ECHO10 schema requires it to contain a element. Therefore, we should not support a bare VariableLevel1Keyword; the /Value path is intentional.

eudoroolivares2016 · 2026-08-28T22:21:12Z

+ *
+ * @type {Readonly<Record<string, string|string[]>>}
+ */
+export const ISO19115_FIELDS = Object.freeze({


Is this the same ones for ISO MENDS and for SMAP?

Yes. ISO MENDS and ISO SMAP use the same ISO 19115 field names, so they share ISO19115_FIELDS and the same scheme editors. SMAP’s only relevant structural difference is its outer DS_Series/seriesMetadata/MI_Metadata wrapper, which Iso19115MetadataPathEditor adds to the XPath automatically.

eudoroolivares2016 · 2026-08-28T22:38:11Z

+  .toLowerCase()
+  .replace(/[^a-z0-9]/g, '')
+
+const AUXILIARY_CSV_HEADERS = new Set(['uuid', 'longname', 'datacenterurl'])


Should this maybe just be in the consts

Good suggestion. I moved these into constants.js and clarified that they are normalized CSV column names excluded from keyword hierarchy paths.

cgokey

.

Christopher D. Gokey added 2 commits August 19, 2026 12:35

KMS-705: parse keyword CSVs by named columns and preserve auxiliary m…

1957387

…etadata

KMS-705: Correct publisher keyword-event mapping so published and dra…

0f38c87

…ft keyword CSVs produce complete, CSV-shaped correction objects for every supported scheme.

cgokey requested review from eudoroolivares2016 and htranho August 20, 2026 21:36

cgokey requested a review from william-valencia August 20, 2026 21:36

Christopher D. Gokey added 2 commits August 20, 2026 20:33

KMS-705: Updated test coverage.

42db0bc

KMS-705: Ignore DataFormat corrections when only LongName changes

4adaabb

htranho approved these changes Aug 24, 2026

View reviewed changes

eudoroolivares2016 approved these changes Aug 28, 2026

View reviewed changes

cgokey commented Sep 2, 2026

View reviewed changes

KMS-705: Moved AUXILIARY_CSV_HEADERS to constants

9a38be7

cgokey merged commit bab8a4d into main Sep 2, 2026
7 checks passed

cgokey deleted the KMS-705 branch September 2, 2026 02:33

Uh oh!

Conversation

cgokey commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

What is the feature?

What is the Solution?

What areas of the application does this impact?

Testing

Attachments

Checklist

Uh oh!

coderabbitai Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgokey left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cgokey commented Aug 20, 2026 •

edited

Loading

coderabbitai Bot commented Aug 20, 2026 •

edited

Loading

codecov-commenter commented Aug 20, 2026 •

edited

Loading

cgokey left a comment •

edited

Loading