V0.4.5 - #168
Merged
Merged
Conversation
Exclude Claude Code configuration files from package builds to avoid NOTEs during R CMD check. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Bug Fixes: - H2: Fix parenthesis bug in nrow(failed_coordinates) > 0 - H3: Fix "langauge" typo -> "language" in attribute name - H4: Fix duplicated() to catch ALL duplicate headers, not just second occurrence - H6: Fix "_sqlte_fra$" regex typo -> "_sqlite_fra$" - H7: Fix subjectFr using wrong source (subjectEn -> subjectFr) Typo Corrections: - M5: Fix "Unkown" -> "Unknown" (4 occurrences) - M6: Fix "gplimpse" -> "glimpse" in documentation - M7: Fix "failty" -> "faulty" in error messages (2 occurrences) - L4: Fix "pr" -> "or" in roxygen documentation - L5: Remove redundant (strip_classification_code) in param docs Closes #145, #147 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…M8, M9)
- H9: Fix NULL/empty check order in view_cansim_webpage
- H8: Add data context to pull(date_field) when sample_date is NA
- H10: Pass correct cache_path and language to list_cansim_cached_tables
- M3: Pass warn_only through recursive retry calls in get/post helpers
- M4: Route API calls through retry helpers (get_cansim_table_url,
get_cansim_changed_tables, get_cansim_cube_metadata)
- M8: Handle NULL/empty vector in cache freshness check
- M9: Add tryCatch error handling for cache write operations
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- M1: Fix percent UOM label inconsistency - use grepl() instead of
exact equality to match percentage patterns consistently
- M10: Replace hardcoded column indices with named column access
in legacy column file support
Note: M2 (handle missing attrs in normalize_cansim_values) deferred
to a separate PR as it requires more investigation - the naive fix
caused unintended metadata folding in vector flows.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Performance optimizations for frequently executed code paths: P5: Factor conversion gsub loop optimization - Replace for loop with across() for single-pass processing - Use vapply for field existence check - ~30-50% improvement for factor conversion P2: parse_metadata pre-split optimization - Pre-split meta3 and meta2 by dimension_id before loop - O(1) hash lookup instead of O(n) filter per column - ~60-80% improvement for metadata parsing P13: lapply %>% unlist chain optimizations - Replace with lengths() where computing list lengths - Replace with purrr::map_chr() for list-to-vector extraction - Replace with vectorized gsub() where applicable - Replace with vapply() for class checks - ~20-30% improvement across various functions P1 (partial): fold_in_metadata member ID extraction - Replace lapply %>% unlist with purrr::map_chr() - Full batch join restructuring deferred (complex refactor) Locations optimized: - cansim.R: normalize_cansim_values, fold_in_metadata_for_columns, categories_for_level - cansim_metadata.R: parse_metadata, read_notes, get_cansim_cube_metadata Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
P3: list_cansim_cached_tables single-pass optimization - Consolidate three separate lapply calls into a single iteration - Collects timeCached, rawSize, and title in one pass per cached table - Avoids repeated dir() and file read operations - Use vapply for type-safe extraction from collected metadata - Expected improvement: ~65-85% for list_cansim_cached_tables() P10: Avoid unnecessary tibble conversion - Check tibble::is_tibble() before calling as_tibble() - Skips conversion when data is already a tibble - Expected improvement: ~5-15% for normalize_cansim_values() Note: P6 (field cache utilization) and P7 (csv2sqlite transform copies) were evaluated but not implemented: - P6: Could not identify specific field cache location in current code - P7: Conditional piping would harm readability for minor gains Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
P4/P11: Environment hash table for parent lookup in add_hierarchy - Replace named vector lookup (O(n) per key) with environment hash table (O(1)) - Use mget() for efficient batch lookup with proper NA handling - Significant improvement for large datasets with 10k+ members - Expected improvement: 40-95% for hierarchy building P8: vapply instead of sapply for null check - Replace sapply(x, is.null) with vapply(x, is.null, logical(1)) - Type-safe and faster null detection in vector extraction - Expected improvement: 30-45% for vector data processing Deferred optimizations: - P9: French string constants - intToUtf8() is already very fast - P12: Coordinate metadata loop - requires significant refactoring (moving get_cansim_cube_metadata call outside the map loop) Files modified: - R/cansim_metadata.R: add_hierarchy function - R/cansim_vectors.R: vector extraction null check Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace deprecated dplyr functions in list_cansim_cubes():
- mutate_at(vars(ends_with("Date")), ...) → mutate(across(ends_with("Date"), ...))
- mutate_at(vars(matches("releaseTime")), ...) → mutate(across(matches("releaseTime"), ...))
This addresses deprecation warnings from dplyr 1.0.4+.
Note: Other API consistency items from the audit plan (H11, H12, M12, M13, M14)
are deferred as they require coauthor discussion:
- H11: Standardizing language parameter defaults could break existing code
- H12: Adding parameter validation could reject previously accepted input
- M12: Standardizing refresh parameter semantics needs design discussion
- M13: Documentation appears to be already aligned with code after review
- M14: Extracting language constants is a larger refactoring effort
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Benchmarking showed the environment hash table approach is actually 1.6% slower than the original named vector lookup. In R, named vector indexing is highly optimized (vectorized C code), while environment creation and mget() have overhead that outweighs the O(1) lookup benefit. Benchmark results (5000 members, hierarchy building): - Original (named vector): 354ms median - Optimized (env hash): 360ms median - Improvement: -1.6% (regression) Keeping P8 (vapply instead of sapply) which is a valid type-safety improvement. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Instead of calling as.Date() on every row (millions of rows), build a lookup table for unique date values (typically hundreds or thousands), then use vector lookup for assignment. Benchmark on table 14-10-0287 (5.4M rows): - Original: mean 85.8 sec - Optimized: mean 67.0 sec - Improvement: 22% faster overall Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Key optimizations: - Use base R order() for sorting instead of dplyr arrange() - Combine multiple mutate() calls into single operations - Use base R direct assignment for factor conversion instead of dplyr mutate() - Remove redundant arrange() call after get_deduped_column_level_data (now returns pre-sorted data) Benchmark results on 5 test tables show 2-41% speedup while producing identical output: | Table | Old (s) | New (s) | Speedup | Identical | |------------|---------|---------|---------|-----------| | 36-10-0108 | 4.34 | 3.07 | 1.41x | TRUE | | 36-10-0107 | 3.68 | 4.12 | 0.89x | TRUE | | 36-10-0580 | 18.24 | 17.95 | 1.02x | TRUE | | 98-10-0044 | 0.22 | 0.17 | 1.29x | TRUE | | 38-10-0234 | 0.39 | 0.29 | 1.34x | TRUE | Note: Timing variations are partially due to network latency during table download. Local processing optimizations provide consistent improvements. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Merge GitHub checks from master into dev branch
fix: Critical bug fixes and typo corrections
get_cansim_cube_metadata() built the API request for several product ids but assumed a single table everywhere else, so a vector of table numbers errored on a length > 1 dir.exists() condition and would have kept only the first table. Metadata for all uncached tables is now downloaded in a single API call and the response is split and cached per table. Types other than "overview" carry no table identifier, they now always get a cansimTableNumber column. get_cansim_table_template() builds one template per table and stacks them, and functions that only work on a single table fail with an informative message instead of silently processing the first entry.
…on-optimization
StatCan returns some names containing characters that render as an ordinary space or as nothing at all, most importantly the non-breaking space U+00A0. A column whose name held one could not be reached by typing or copy-pasting what the console displayed, which made the column unusable. A survey of all 8226 cubes found U+00A0 in 137 dimension names and titles and U+000A in 4. Zero width characters are now dropped and everything else that behaves like a space becomes a regular space, followed by squishing and trimming. Strings holding none of these characters are returned untouched, so the squish can never alter a name StatCan spelled ordinarily, and the repair is idempotent. The repair is applied to the CSV header, to the dimension and member names in the table metadata, to cube metadata, and to the cube list. Data column names and metadata dimension names have to agree or metadata stops folding in, so both go through the same function; 27-10-0123 and 46-10-0101 keep their full hierarchy and classification columns after repair. Header repair runs before the duplicate column check, since it can turn two names that differed only by a non-breaking space into the same name. Column names are baked into cached parquet and sqlite files, so get_cansim_connection() warns when it opens a cache that predates this and points at refresh=TRUE. The cube list doubles as an internal lookup for cache staleness checks, so it only reports repairs when called on the user's behalf. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The warning printed the repaired name, so the character it was reporting on was invisible in the report itself: "Repaired "Performance strategy"" gave no way to see what had been wrong. The name is now shown as StatCan sent it, with the offending characters rendered as their code points, "Performance<U+00A0> strategy". The list of repaired names is replaced by a count plus one example. list_cansim_cubes() repairs 120 names and was printing five full table titles on a call users make casually. Long names are windowed around the code point so the marker stays visible. The stale cache warning in get_cansim_connection() had both problems and gets the same treatment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The factor conversion path in normalize_cansim_values() split the COORDINATE of every row to recover member ids, while fold_in_metadata_for_columns() already split only the unique ones. A table repeats each coordinate once per reference period, so the unique set is a small fraction of the rows: 36-10-0580 has 6,882 unique coordinates across 996,978 rows, 0.7%. The split now runs on the unique values and rows are read back through a match index. The step drops from 0.29s to 0.03s and a cached read of 36-10-0580 from 4.5s to 4.0s. Tables whose dimensions have no duplicate member names never enter this path and are unaffected. Output verified identical on 36-10-0580, 18-10-0004, 27-10-0123 and 98-10-0036, the last covering the census geography case where deduplication is deliberately skipped. For the record, factor conversion is not the bottleneck it was taken for: it is 18% of a cached read of 36-10-0580 and 2% of 18-10-0004, against 68% for reading the cached data back off disk. factor() itself is 0.06s of the 0.79s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every request to StatCan now goes through a single failure path. Timeouts,
connection failures, SSL peer verification failures and non-200 responses are
reported with a loud warning and the calling function returns NULL, instead of
aborting with an error. Erroring is what repeatedly got the package pulled from
CRAN, since a check run started while StatCan was down failed on examples that
are not at fault. Set options(cansim.error_on_unavailable=TRUE) for the old
behaviour.
get_cansim_table_last_release_date() and get_cansim_series_info_cube_coord()
previously bypassed the retry helper, they now use it as well. Functions that
make several requests return NULL if any of them failed, a partial result there
is silently wrong rather than obviously missing. StatCan rejecting a specific
product id still errors, that is a bad table number rather than an outage.
With that, 20 example blocks move from \dontrun{} to \donttest{} so they are
checked, and cansim_old_to_new() needs no network at all so its example always
runs. Examples that download a full table or the cube list stay \dontrun{} on
run time grounds.
Also fixes two pre-existing bugs found along the way: get_cansim_changed_tables()
passed "days" to difftime() as a time zone instead of a unit, and the staleness
check in get_cansim_connection() could reach an if() with a zero length or NA
condition.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019a31KaUZ26aRrurFSXNjSu
Update cran-comments.md with the current test environments, the 0.4.5 changes,
and a note explaining why some examples are still \dontrun{}, since this package
has a history of check failures caused by StatCan being unavailable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019a31KaUZ26aRrurFSXNjSu
metadata_for_coordinates() looped over the coordinates and rebuilt the member table of every dimension, along with the duplicate name disambiguation and the factor levels derived from it, once per coordinate. None of that depends on the coordinate, so the work grew linearly at about 24ms per coordinate on a four dimension table. The member tables are now built once per dimension and the coordinates are split once into a character matrix, so each dimension resolves its members for all coordinates in a single match(). This is the same approach fold_in_metadata_for_columns() already uses. Resolving 200 coordinates of 36-10-0580 goes from 5.0s to 0.02s, and all 10,164 coordinates of that table now take 0.03s. Output is unchanged, verified byte for byte against the previous implementation on six table/language combinations including truncated coordinates and members that are absent from the cube metadata. The one behaviour change is that a missing member is now warned about once rather than once per coordinate that uses it. Closes the last open item of #149. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019a31KaUZ26aRrurFSXNjSu
The repair added in 29cd36e covers the member names in the metadata but not the labels in the data, which are the same strings read from the data csv. Factor conversion takes its levels from the repaired metadata and applies them to the unrepaired values, so every row carrying an affected label fell outside the levels and became NA. On 13-10-0920 that was 2,574 of 10,296 rows, on 27-10-0222 in French 3,498 of 10,920, on 46-10-0099 894,432 of 19.7M. Nothing warned, factor() simply returns NA for a value that is not in levels. These characters are more common in member labels than in the dimension names the repair was written for: 1,230 of 159,392 member name fields across 500 sampled tables, against 7 of 3,548 dimension names, with 53 of the 500 tables carrying at least one. get_cansim() and get_cansim_connection() do not share a csv reader, so the repair goes in three places: the in-memory read in get_cansim(), the chunk transform of csv2sqlite(), and csv2arrow() for parquet and feather. Only the dimension columns are repaired, mapped through the same geography rename the factor conversion uses, since the coordinate column holds one distinct value per series. Dimension columns hold a few distinct labels repeated across millions of rows, so only the distinct values are scanned and the rows are read back through an index. On a 19.7M row table that is 2.1s against 184.8s for scanning every row, and adds 6s to a parquet write that takes 5s. Letting arrow evaluate the repair instead is not an option, it has no binding for the string squishing and silently pulls the whole table into R, taking 73s. Labels are now identical whichever way the data is retrieved, so table data can be joined to template, vector or coordinate data on its dimension columns. Previously get_cansim_table_template("13-10-0920") and get_cansim("13-10-0920") disagreed on the same member. Tables already in the persistent cache of get_cansim_connection() keep the original values until downloaded again with refresh=TRUE, detecting that is still to do. The session cache of get_cansim() lives in tempdir() and needs nothing. Refs #169. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019a31KaUZ26aRrurFSXNjSu
The download timestamp says whether StatCan has newer data, it says nothing about whether the package has since changed how it reads that data. The version that did the parsing is now written next to the timestamp, and list_cansim_cached_tables() reports it in a cansimVersion column. get_cansim_connection() uses it to recognize a cache built before non-breaking spaces and control characters were repaired. Rather than guessing at what is on disk, it reads the dimension names and member labels cached alongside the table to see whether they actually carry any, and only then warns, naming the offending label and pointing at refresh=TRUE. Most tables are unaffected and stay quiet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019a31KaUZ26aRrurFSXNjSu
The download timestamp and the version that parsed the files describe the same cached table, and both are consulted together, so they now live in a single .Rda_info file as a named list rather than in a file each. Further entries can join them there. The old .Rda_time file is still read, so existing caches keep their download date, and is removed when a table is refreshed rather than left behind to go stale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019a31KaUZ26aRrurFSXNjSu
R prints a warning as it was assembled, so a message built from several sentences arrives as one unbroken line. Both repair warnings, the one about names StatCan sent and the one about a cache built before they were repaired, are now passed through strwrap so they wrap the way ordinary console output does. Wrapping can put a newline inside a phrase a test matches on, so the two tests that check for multi-word wording now go through a `warning_text()` helper that collapses whitespace before matching. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019a31KaUZ26aRrurFSXNjSu
cleaned_ndm_language() returned NA for anything it did not recognize and passed it on. The NA travelled as far as the name of a cache directory or the tail of a StatCan URL, so an unrecognized language surfaced as a download failure or as a column that could not be found, neither of which points at the argument that caused it. get_cansim_table_url(language="de") returned NULL with "StatCan returned status code 406". It now errors, and the message names the value that was passed rather than the NA it normalized to. Either language can be named in either language. Input is trimmed, folded to lower case and stripped of accents before matching, so "english", "en", "eng", "anglais" and "ang" all select English, and "french", "fr", "fra", "francais" and "fran" all select French, accented or not. Vectors stay allowed, remove_cansim_cached_tables() asks for both languages at once. get_cansim_table_url() and get_cansim_table_notes() were the only two of the 17 functions taking a language that defaulted to "en" rather than "english". Both normalize to the same value, so this changes no behaviour. The two removal functions keep their NULL default, which means every language. Also drops the now redundant guard in file_path_for_table_language(), which reported the normalized NA and misspelled "Language", and standardizes the @PARAM wording that had drifted into five variants. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019a31KaUZ26aRrurFSXNjSu
Every other two letter form the package accepts is an ISO 639-1 code. "an" is not one, and it is not a form anyone would plausibly type for "anglais", so accepting it only risked swallowing a typo. The longer short forms "ang" and "angl" stay. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019a31KaUZ26aRrurFSXNjSu
read_meta() switched from readr::read_delim() to utils::read.delim() in 92639fd (Feb 2025), for the smart quote workaround table 17-10-0016 needs and for exact control over quoting and NA strings. The readr version was kept behind an `if (TRUE) ... else ...` rather than deleted, and has been unreachable since. It also fell behind: a6c1539 added the header row handling to the live branch only, so the two no longer return the same columns and the else branch was not a working fallback either. Also rewrites `if (nrow(d>1))` as `if (nrow(d)>0)`, which is what it evaluates to. `d>1` compares the whole tibble and returns a matrix of the same shape, so nrow() of it is just nrow(d), and the condition was true whenever the section had any lines. That is the wanted behaviour, a section consisting only of its header line yields no rows, but the expression read as if it meant nrow(d)>1, which would have been wrong. read_meta() verified to return identical output before and after on all 42 metadata sections of 17-10-0016, 13-10-0920, 46-10-0072, 36-10-0580 and 98-10-0036 in both languages, and on header-only and padded sections. readr stays an import, it is used in eight other places. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019a31KaUZ26aRrurFSXNjSu
The warning already said the characters came from StatCan, but not that they will go away on their own once StatCan stops sending them. Someone reading it had no way to tell whether this was a permanent fact of the package or a condition being tracked, so it pointed at nothing to watch. It now names the issue where the upstream state is tracked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019a31KaUZ26aRrurFSXNjSu
The package repairs non-breaking spaces and control characters on the way in, which by design hides them, so there was no way to tell whether the upstream problem was shrinking without hand-rolling a reader of the raw API response. scan_statcan_character_problems() is that reader, and summarize_statcan_character_problems() aggregates it by level, language and survey. Neither is exported. Both, and the repair they measure, come out again once a scan comes back empty. Reproduces the baseline posted on the issue: 13 titles, 120 dimension names and 7245 member names across 803 of 8226 tables, 128 of 535 surveys. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019a31KaUZ26aRrurFSXNjSu
…nection It was the last function still named for sqlite that was not itself deprecated, in a package whose other entry points moved to format-agnostic names some time ago. Its own example demonstrated get_cansim_sqlite(), which is deprecated and scheduled for removal, so the one example of a current function showed users the call they are being told to stop making and would have broken when the removal landed. Deprecating it needed a successor to point at, since closing a connection had no other name. disconnect_cansim_connection() is the same function under a name that does not claim a format: it closes a sqlite connection and leaves parquet and feather ones alone, so a caller that does not know which format it holds can close it either way. collect_and_normalize(disconnect=TRUE), the tests and the vignette call the new name, so nobody sees a deprecation warning for a call they did not make. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019a31KaUZ26aRrurFSXNjSu
Pre-existing bugs, documented in NEWS: - percentage normalization now relabels the unit of measure to Rate/Taux as documented, the comparison doing the relabelling could never match - vector and coordinate data normalizes percentages the same way as full table downloads now that it carries UOM columns, with an offline test covering the /100 normalization, the relabelling in both languages and the normalize_percent opt-out - add_cansim_vectors_to_template no longer trims trailing zeros out of member ids when stripping unused coordinate positions - French connections no longer warn "Unknown table type" on collect_and_normalize, an internal language comparison never matched - failed refreshes of cached tables and cube metadata fall back to the previously cached version with a warning instead of returning NULL - the duplicated-column error names the offending columns and no longer blames SQLite for parquet and feather connections Also fixes regressions from the 0.4.5 work: dimension ordering for cubes with ten or more dimensions, duplicate dimension names in templates, member-less dimensions, short coordinates in factor conversion and metadata folding, NULL guards on the StatCan-unavailable pathways, a duplicated metadata fetch, tidyselect deprecation warnings and message typos. The availability test now isolates CANSIM_CACHE_PATH so a real cache cannot satisfy the fallback pathways. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019a31KaUZ26aRrurFSXNjSu
StatCan refuses a request carrying more than 300 items with an HTTP 416. Most of the calls that send a list already batched, but get_cansim_vector_info() and the cube metadata download did not, so asking either for more than 300 items failed rather than returning data. Both now go through a shared batch_items(). The API signals a bad item two different ways depending on the method, either marking the record FAILED or answering SUCCESS and putting the reason in responseStatusCode, and the package only checked the first. That let an invalid vector through get_cansim_vector_info() as a row of NAs indistinguishable from real metadata. Both are now checked everywhere, and dropped items are reported by reason with the vectors or coordinates named. Vector calls that come back empty now warn and return an empty table. The empty answer used to travel on to the metadata join and surface there as a missing cansimTableNumber column, which said nothing about what had happened. When StatCan refuses a request it explains why in the response body, which was being discarded along with the rest of the response. That explanation now accompanies the status code, so a 409 says whether the product is simply not released yet and a 416 names the limit the request went past. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HqWZMTRqhbYA7UexPnZDPu
…ence Requests now go through httr2, which brings retries with exponential backoff and jitter, honouring a Retry-After header, and throttling to the 25 requests per second StatCan documents as its per IP limit. Statuses that will not improve on a retry are excluded: 409 is the nightly update window, 416 is a request carrying more items than StatCan accepts, and 503 is an outage that lasts far longer than any retry budget worth spending. Retrying is bounded to thirty seconds of wall clock time so a failing call cannot sit for the better part of quarter of an hour. The timeout argument now bounds how long StatCan may go without sending anything rather than how long the whole transfer may take. As a cap on the total it could not tell a connection StatCan had stopped answering on from a large table still arriving, and cut both off alike. StatCan works out a whole response before sending any of it, about a tenth of a second per vector, so a full batch of 300 is silent for some thirty five seconds before the first byte; the default of two hundred seconds leaves room for that. Connecting is bounded separately at ten seconds so an unreachable host fails fast. Three new functions expose the changed series methods, which report what changed at a finer grain than get_cansim_changed_tables() does. Series that did not change contribute no rows, and a day on which none of the ones asked about changed is an empty table rather than an error. get_cansim() and get_cansim_connection() ask StatCan where a table lives instead of assembling the address from the table number, which was guessing at a layout StatCan is free to change. get_cansim_changed_tables() takes both the current date and the cutoff in Eastern time. It compared against 9am, half an hour after StatCan closes its nightly window, and took today from the local clock, so a machine set west of Eastern could ask about a day that had not started there yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HqWZMTRqhbYA7UexPnZDPu
StatCan works out a whole response before sending any of it, and the series changing on a given day number in the hundreds of thousands, so getChangedSeriesList outlives StatCan's own gateway and comes back as an HTTP 504 after some nine minutes of silence. A raw curl with no timeout of our own confirms it: 504 at ttfb=529s, the body being StatCan's gateway timeout page. The method takes no parameters, so there is no smaller request to make instead. Exporting it would hand users something that mostly does not work, so it stays internal until it is clear whether this is a fault worth working around or simply how the method behaves. The two changed series data methods are unaffected and stay exported. The timeout default of 600 seconds stays, so the 504 surfaces as StatCan's own answer rather than as a vaguer local abort at 200, and the 504 translation now says that asking for less at once is what gets past a gateway timeout, where retrying unchanged will not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HqWZMTRqhbYA7UexPnZDPu
…lled get_cansim_changed_series_data_for_vectors() handed the caller's vectors on to rename_vectors() with the "v" prefix still attached, which looked labels up under "vv..." and matched nothing, so a named vector such as c(foo="v41690973") came back labelled with its id rather than with "foo". It now passes the naked vectors, whose names survive the stripping, the way get_cansim_vector() always has. The new tests pin the whole naming contract for both functions: either spelling of a named vector yields the caller's label, unnamed vectors get no label column, and the VECTOR column always carries the standardized "v" prefix whatever the caller typed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HqWZMTRqhbYA7UexPnZDPu
Asking the latestN methods for all periods sent an arbitrary 1000000, which worked only because it happened to exceed the longest series. StatCan takes latestN as a signed 32-bit integer: zero or less comes back as "vector id or latest N is negative or zero", anything past 2147483647 as a JSON syntax error, and a count longer than the series is clamped to the whole series at no extra cost. The WDS user guide states none of this, it only requires latestN to be > 0, so the bound is recorded in a comment the way MAX_BATCH_SIZE already records the batching limit. MAX_PERIODS is now that bound. Being .Machine$integer.max it is also the largest value R can hold in an integer, so the coercion downstream can no longer overshoot what the endpoint accepts. The same call sites coerced with as.integer(), which turns anything above the 32-bit range, Inf included, into NA with only a warning and pasted "latestN":NA into the request body. clean_periods() now caps those instead, and refuses a count below one outright rather than sending a request that is certain to earn an HTTP 406. It is vectorized, so it also covers the per-coordinate periods column of a table template, which previously got a coalesce() on NA and no range check at all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HqWZMTRqhbYA7UexPnZDPu
The 504 that held this back is a function of how much StatCan released that morning rather than a fault in the method. Today it answered in 0.35 seconds with 58 series across three tables, and everything about that answer checks out: the three tables are exactly the ones get_cansim_changed_tables() reports for the same release time, all 58 vectors come back from getChangedSeriesDataFromVector, and the 27 coordinates listed for 33-10-0036 come back from the coordinate method as the same set of vectors. Today's payload also arrives in the unwrapped shape, so the branch reading series straight out of `object` is confirmed against real data and not only against the recorded one. So the method works, it is just at the mercy of the day. That is worth documenting rather than worth withholding a function over, and the alternative on a heavy day is the coarser question get_cansim_changed_tables() answers, which the docs now point at. The `timeout` parameter stays and is documented: it defaults high because the method is silent while it works, so a genuine gateway timeout arrives as StatCan's own 504 rather than as a vaguer local abort, but the limit being hit is at StatCan's end and raising it further does nothing. The method takes no parameters, so a busy day cannot be asked about in smaller pieces. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
updated can version