keys are not unique across all files

[kellijohnson-NOAA]

A function or process for creating unique keys across files is needed. I am inclined to do this for just a user-defined set of files rather than across all files within journals because there are so many files. I think it would be important to sort the articles present chronologically by bibdate so if/when the grouped file is updated the old keys would not change, i.e., if "happy:2006:ABCa" and "happy:2006:ABCb" are already in the file, then a new addition would become "happy:2006:ABCc". This would not alleviate pains if a new journal was added to a grouped file. We could export information about all the names that changed though.

If the journal abbreviation is added to the key they are all unique. The new problem lies in that when a note to another article appears in one decade and the article is from a different decade then the said article is copied into the other bib file so it is listed two times that way it can be referenced in the note. This causes duplicate keys, exactly two times for 104 entries as of 2026-08-13.

Steps

• Write a function to group reports together by bibdate.
• Make the keys unique by adding the journal abbreviation.
• Use this function for {asar}.

FYI

@Schiano-NOAA @sbreitbart-NOAA

[Schiano-NOAA]

Here are the current journals used in the asar bib file:
Fisheries Research
Canadian Journal of Fisheries and Aquatic Sciences
Fishery Bulletin
Marine and Coastal Fisheries
ICES Journal of Marine Science
North American Journal of Fisheries Management
ICES Journal of Marine Science: Journal du Conseil
Journal of Fisheries Research Board Canada
Journal of computational and graphical statistics
Canadian journal of fisheries and aquatic sciences
Biometrika
Operations Research
Ecology Letters
Biometrics
Marine Biology
Environmental Biology of Fishes
Fisheries Oceanography
Journal of Statistical Software
Journal of Fish Biology
Ecological Complexity
IEEE Transaction on Automatic Control
Pacific Marine Fisheries Commission Bulletin
California Fish and Game Fish Bulletin
Applied Statistics
Wash. Dept. Fish. Res. Pap.
California Department of Fish and Game
Oregon Fish Commission Research Briefs
Fisheries Commission of Oregon, Research Briefs
California Department of Fish and Game Fish Bulletin
Bull. Am. Meteorol Soc.
Abt. Anat. Ontog. Tiere.
Philosophical Transactions of the Royal Society B: Biological Sciences
PLoS One2
Marine and Coastal Fisheries: Dynamics, Management, and Ecosystem Science.
bioRxiv

[kellijohnson-NOAA]

Of that list we currently have the following journals: Canadian Journal of Fisheries and Aquatic Sciences, Fishery Bulletin, Fisheries Oceanography, Fisheries Research, ICES Journal of Marine Science, ICES Journal of Marine Science: Journal du Conseil, Journal of Fish Biology, Journal of Fisheries Research Board Canada, North American Journal of Fisheries Management

We will need to prioritize which other ones are needed based on how many entries there are per journal, for example I do not want to cover the entire journal when it is only used for two articles. I will look at the bib file online in asar to see how many times each is included.

[Schiano-NOAA]

I have a script that automatically extracts these. I can real quick calc the number of entries per journal

Edit: here are the journals with >2 entries.

journal	count
Canadian Journal of Fisheries and Aquatic Sciences	19
Fisheries Research	16
ICES Journal of Marine Science	9
North American Journal of Fisheries Management	7
Fishery Bulletin	7
Fisheries Oceanography	3
Journal of Fisheries Research Board Canada	3
Biometrics	2
California Fish and Game Fish Bulletin	2
ICES Journal of Marine Science: Journal du Conseil	2
Marine and Coastal Fisheries	2
Marine Biology	2

[kellijohnson-NOAA]

@Schiano-NOAA I made quite a bit of headway on this today because I was curious 😉. I think what I am going to do is save all the entries, which currently there are 171,575 entries, in a single .bib file. The keys are

<journal abbreviation>:<last name of first author>:<year>:<first letter of the first three significant words>

e.g., canjfishaquatsci_Armstrong:2010:CEC

for a 2010 article in the Canadian Journal of Fisheries and Aquatic Sciences by Jonathan B. Armstrong titled "Comment on egg consumption in mature Pacific salmon (Oncorhynchus spp.)". Maybe for assessments we can add "sa" instead of the journal as the prefix 🤷‍♀️? I am open to other ideas as well.

Then, users will just need to download a single file that they can overwrite whenever they want because overwriting it won't change any of the old keys, in theory 😜.

Thoughts?

[Schiano-NOAA]

I think that sounds reasonable. I do think the key is quite long. Did you encounter duplicates of the same journal, author, and year? I am wondering if that is enough to make them unique (or maybe remove journal and add title acronym. Alternatively, if we are going to shorten the title like in the example, I think we just set it to the acronym.

I agree about setting a prefix for stock assessment reports. I'd suggest actually setting the prefix as "sar" since that term is commonly used with the marine mammal folks and it would help start the alignment with them.

Lastly, agreed with the process of users downloading and overwriting bib file to update it. Maybe we instruct users to use a function in journals called "update_bib" or something and it will overwrite a specified bib file or just whatever standard named one we have that was initially produced from this.

[Schiano-NOAA]

Also, I will be talking to the SIS devs next week about getting the metadata for assessment reports to turn into a bib file. I will have more information about that soon. Should I open an issue about that or just make a PR when things are being developed?

[kellijohnson-NOAA]

I am not sure what you mean by journal acronym? Yes, there is a standard one for cjfas but that is colloquial and not available for all journals. The files names are currently based on standardized shortening of journal names where some journals use these shorted names instead of the full journal name in their references sections. I used the name of the file as the prefix here. I cannot remember how many were duplicated across journals before doing this but I know it was a problem. Again still open to other ideas on how to solve this but I don't know how to get "acronyms" for each journal beyond cjfas.

Regarding SIS, see #2

[Schiano-NOAA]

Ah I see. I was thinking acronyms by the first letter of each word in the journal name even if there is no typical convention. I did not know about the shortening convention, so that approach is much more clear why.

I think what you have proposed is fine. I think the alternative would be to indicate a, b, c, d, but this approach is much more specific and you would think might be easier to remember if the author knows the reference (which they should lol). I think your suggestion then will work great and I do not have any alternatives to propose that would be a better indicator.

Note: I looked into the use of : in the key and it looks like it usually won't cause issues. There have been reports that there is an error it causes when the document language is in french. There does seem to be a fix so, for documentation purposes, I am linking the issue and solution here.
https://tex.stackexchange.com/questions/555730/colon-in-bibtextkey
https://tex.stackexchange.com/questions/89016/colon-in-bibliography-keys-incompatible-with-babel-biblatex-and-tex4ht

[kellijohnson-NOAA]

@Schiano-NOAA I tried to push all of my changes this afternoon as I thought that I had completed this work and I was stopped by GitHub because the combined file is 159.92 MB and the file size limit is 100.00 MB on GitHub. I am using awk to combine the files and not everyone will have access to awk so I cannot make users do it themselves. There are definitely some journals that I could pull out of the global file but that seems like a waste. Do you have any ideas?

[Schiano-NOAA]

Nothing off the top of my head but I'll do some brainstorming research on Monday. Brain is getting a little fried atp on Friday 😆

[Schiano-NOAA]

I wonder if there is an alternative to awk? I have not heard of it before. It seems to make the most sense to me to have the function which calls the .bib file to bring it all together and save it to the user's id'd folder.

I'm thinking ideally the function would

• combine the files
• set unique identifiers
• save it to the specified directory
• [possibly] include a option for filters where users could select journals (by default it would be all?)

Do you currently have it in a script or action that I can take a look at? I feel like there has to be some R package that could help us here.

[kellijohnson-NOAA]

There are plenty of R packages but they take forever to read the files in and not all of the original file is maintained depending on which package you use. So, using R is not ideal.
What do you think about saving multiple combined files, maybe one per decade. I could see what the size of those files would be? Then, asar would have to reference approximately 5 files rather than a single file.

[Schiano-NOAA]

I think that could work too. I was thinking of having a separate one for assessments anyway. It's easy enough to add multiple bib files to a quarto doc. And I agree, we need something that is faster than R.

So the plan would be to have another action to combine the files by decade and have them in a folder? Then provide a function to call these and copy into a folder of the user's choosing or do you have another thought?

I am thinking about its use in asar and I ideally just want to copy in the files and then list off the names under the bib area in the yaml.

[kellijohnson-NOAA]

I moved all the raw files to tex/bib, the combined files by decade will live in inst/bib and the "download" function has an argument telling it where to look, do you want file in "tex" or in "inst". So for asar you would just point it to "inst" where every article has a unique id and include the five file names. Eventually, we can add the assessment bib to journals in inst/bib and then the download function would also download that file.

[Schiano-NOAA]

inst would be preferred