Company of Heroes 3 Data

Data and icons on all Company of Heroes 3 units / stuff in a developer friendly format.

Goal

Goal of this repository is to gather all the data from the game Company of Heroes 3. For example data for all units. As well as tools and scripts for gathering this data from the game. We beliave that open sourcing all the data and tools can only help the game and community. Once we have the data it's up to anyone to build any web-site or apps and work with this data. But let's put all our heads together

Convention

• All the data should be converted into .json files
• Please provide all the scripts / add manuals how you get the data
  • We can expect a lot of updates from developers. So we will need to run our scripts many times.
• We shall do git tags to mark patch changes
• The tags are in the format v1.0.7-1, where 1.0.7 should match the game patch version, -x (marks our revision). So v1.0.7-1 means game patch 1.0.7, data revision 1.

Downloading the data

If you want to utilize the data in the build / download them periodically. Please use this URL:
https://data.coh3stats.com/cohstats/coh3-data/${dataTag}/data/${dataFile}

• dataTag is the tag in this repo
• dataFile is the file from the data folder in this repo

So for example the url could look like this: https://data.coh3stats.com/cohstats/coh3-data/v1.2.5-1/data/battlegroup.json

Contribution

We are looking for more help! If you would like to contribute in any way. Please rech out ot us here in the issues / on the discord https://discord.gg/jRrnwqMfkr

Also if you utilize the data, please give us shoutout! Thank you

Automated data generation

You can use the GitHub workflow to automatically extract and process game data:

1. Go to the Extract Data workflow
2. Click "Run workflow" button
3. Select the branch you want to run it against (usually master)
4. Click "Run workflow"

The workflow will:

• Download only the game files it needs (about 1 GB) with DepotDownloader, using the file list in .github/workflows/coh3-filelist.txt - not the whole ~34 GB game
• Extract USC lang files from .sga using COH3-SGA-Extraction tool for all languages (en, fr, de, it, ja, ko, pl, pt-br, zh-hans, es, zh-hant, tr, cs)
• Process UCS files into JSON using scripts/ucs-to-json.py which converts the raw localization data into structured JSON files
• Extract and process ReferenceAttributes using AOEMods.Essence tool, which:
  • Unpacks ReferenceAttributes.sga into XML files
  • Converts XML data to JSON using scripts/xml-to-json/main.py
• Unpack ScenariosMP.sga and extract multiplayer map data into data/mp-maps.json using scripts/mp-maps/main.py (see scripts/mp-maps/README.md)
• Unpack Data.sga and extract the Final Stand technology choices into data/fs-technologies.json using scripts/xml-to-json/fs_technologies.py
• Create a new branch with changes
• Submit a Pull Request with the updates

Alternatively, you can still generate the data manually following the steps in the "How to generate the data" section above.

How to generate the data

1. Open Company of Heroes 3 Essence Editor (it's located in your COH3 instalaltion folder).

2. One file ReferenceAttributes.sga ...\Company of Heroes 3\anvil\archives\ReferenceAttributes.sga

3. Click on the attrib field, right click, extract
   

4. Delete content of folder xml in this repo,

5. Extract it into folder xml of this repository. Should look like this:
   

The locstrings must be exported as well, but requires the following steps:

1. Within the Essence Editor, click "Attributes" in the menu panel then click "Open Attributes". Wait for it to completely load.
2. Open any attribute file (ebps is opened by default) then click "View" -> Show LocString Report... in the menu panel.
3. With the opened locstring table, select the first column then scroll to the last one and press "Shift" to select all (alternatively Ctrl + A). Then right click, copy or Ctrl + C then paste the content inside the xml/loc_english/locale.txt file of this repository.

Convert the data

1. Go into folder cd scripts/xml-to-json.
2. Run Python script main.py. You need python3. Like this python main.py, you should see this result:
   
3. Check the folder, you should see exported files.

Hint: python main.py -no_bl disables blacklist filter and generates complete JSON files. However, when possible, the filtered JSON files should be used to reduce file size.

You can verify the changes by running command git diff

Final Stand (HOFF) perks

The Final Stand perk trees are exported by a separate script, because they need a different output shape than the generic XML dump. It reads the same unpacked xml/attrib folder, so run it after the extraction above:

python scripts/xml-to-json/fs_perks.py

It writes data/fs-perks.json - the four faction perk trees, their tiers (with the unlockThreshold, i.e. how many perk points must be spent to unlock the tier) and every perk with its levels, perk point costs and modifiers. All text is kept as locstring IDs, resolve them against data/locales/<lang>-locstring.json.

Final Stand (HOFF) technology choices

Perks are the meta progression between matches; the technology choices are the in-match one. At the start of the game and at the start of every wave the player is shown three technologies and picks one - a new unit for the Barracks, a new player ability, or a passive bonus.

They are exported by scripts/xml-to-json/fs_technologies.py into data/fs-technologies.json. Unlike the other scripts this one needs a second archive: the pool of technologies is in xml/attrib (see above), but the order of the twelve picks is game script that lives in Data.sga, so that archive has to be unpacked first:

tools/AOEMods.Essence/AOEMods.Essence.CLI.exe sga-unpack "<game>/anvil/archives/Data.sga" ./game-data
python scripts/xml-to-json/fs_technologies.py --game-data ./game-data

(./game-data is the default, so --game-data can be left out when you unpack it there. The folder is gitignored.)

The file has two parts:

• meta - how a match plays out. choicesPerPick (3), maxOfferingCount (a technology is only ever offered once) and picks: the twelve picks in order, each with the wave it is handed out at, the category it draws from (unit / ability / passive) and the upgradeTypes (the "bucket") a technology must be tagged with to appear in it. Read from scar/hoff/hoff_technologymenu.scar and statemodel_schema/technologymenu.
• races - the pool per faction. Each technologies entry is one technology, already merged from the faction list and the shared "common" list (source says which list it came from), with its category / buckets / tags (the upgrade_type tags the picks match against), the thresholdMin / thresholdMax pick indices it may be offered between, its ui text and icon, all of its raw custom properties, and the squad / ability / upgrade it unlocks for cross referencing sbps.json and abilities.json.

Which of the matching technologies are actually shown is a weighted random draw, so the file describes what can appear at each pick, not a fixed tree. Two caveats worth handling on the consuming side: a technology with "enabled": false is in a faction list but not reachable in game (its ui_menu is not the technologies menu), and one with "category": null matches no bucket, so it can only ever turn up when the game tops up a menu it could not fill.

All text is kept as locstring IDs, resolve them against data/locales/<lang>-locstring.json.

Changes after the patch

1. Generate the data into folder /data
2. Make an MR with the changes

The folder /data should always have the stable export of the data.

Releases

This repo uses Release Drafter to keep a draft release up to date as PRs are merged into master (config: .github/release-drafter.yml). Merged PRs are auto-labeled (data-update, bug, enhancement, chore) based on title/branch patterns so they show up under the right section of the draft.

When you're ready to cut a release:

1. Go to the Releases page and open the draft.
2. Edit the tag/title to follow this repo's convention: vMAJOR.MINOR.PATCH-REVISION, e.g. v2.5.2-1 means game patch 2.5.2, data revision 1 (see the "Convention" section above) - Release Drafter doesn't know about the game's version number, so this always needs a manual edit.
3. Publish the release, which creates the matching git tag used by the data download URL.

Folder chunked

Folder chunked has the big json files split to a smaller files for better manipulation.

Multiplayer maps (data/mp-maps.json)

Map data comes from a different archive than the attributes (ScenariosMP.sga) and has its own scripts and documentation: see scripts/mp-maps/README.md for how to generate the file and what is in it.

How to add/fix missing data not delivered by the Essence Editor

Some times it is nessecary to add some modifications to the XML as not all informations are delivered by relic. E.g. Stormtroopers are spawned with an lmg but there is no lmg member referenced within the loadout of the sbps. For that we created a CoH3 tuning mod which can be adoptet to overwrite missing information.

1. Open the mod from the projects xml directory with the Essence Editor

2. Clone the attribute file that needs to be modified and assign a name. If the mod file has the same name as the original file, the extensions will overwrite extensions from the orginal otherwise, a new json object will be created for that element.

3. Modify the cloned file which will appear in the corresponding Mod directory. You can also delete some extensions which are not required at all to reduce traffic.

4. Save the mod. When the steps of How to generate data are performed, the modifications will be added or overwrite the original data during the json generation process (see step 6 above).

Hint: Overwriting can be skipped during script execution via -no_mod flag. Be aware after patches, the overwriting mod might need to be adopted to reflect Relics patch changes. Thus, choose the files to modify wisely.

References:

Big thanks to all open source tools focused on Relic games. https://github.com/RobinKa/RGDReader/tree/master/RRTexConverter