StatCan API returns column names with non-standard characters that make working with data difficult. These characters include em-dash, closing quotation %E2%80%99 instead of apostrophe, carriage return %0A, and most importantly, non-breaking whitespace %C2%A0. In partular that last one causes problems because it gets translated to regular white space in the console and can't be accessed via copy-paste, making that column inaccessible in an R session.
The package should repair the non-breaking whitespace under the hood and emit a warning to let users know that the StatCan data was modified and why. This repair should happen on first parse from the CSV or from vector or coordinate data calls as well as in metadata calls and before caching the data in a local database.
Code to systematically find non-standard characters
whitelist <- "[a-zA-Z0-9 àâäèéêëîïôœùûüÿçÀÂÄÈÉÊËÎÏÔŒÙÛÜŸÇ,\\.:\\(\\)=\\/'\"\\&\\#;-\\+-_]"
d <- list_cansim_cubes()
non_standard_whitespace="%C2%A0"
right_single_quote="%E2%80%99"
line_feed = "%0A"
en_dash="%E2%80%93"
d |>
mutate(special_character=gsub(whitelist,"",dimensionNameEn),.before="cubeTitleEn") |>
mutate(url_encoded=URLencode(special_character),.before="special_character") |>
filter(nchar(special_character) >0) |>
select(cansim_table_number,url_encoded,special_character,dimensionNameEn) |>
filter(url_encoded!=en_dash)
StatCan API returns column names with non-standard characters that make working with data difficult. These characters include em-dash, closing quotation %E2%80%99 instead of apostrophe, carriage return %0A, and most importantly, non-breaking whitespace %C2%A0. In partular that last one causes problems because it gets translated to regular white space in the console and can't be accessed via copy-paste, making that column inaccessible in an R session.
The package should repair the non-breaking whitespace under the hood and emit a warning to let users know that the StatCan data was modified and why. This repair should happen on first parse from the CSV or from vector or coordinate data calls as well as in metadata calls and before caching the data in a local database.
Code to systematically find non-standard characters