diff --git a/LICENSE b/LICENSE index 8b13789..a693704 100644 --- a/LICENSE +++ b/LICENSE @@ -1 +1,21 @@ +# MIT License +Copyright (c) 2024 Daniel Rakotomalala + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/LICENSE.md b/LICENSE.md deleted file mode 100644 index a693704..0000000 --- a/LICENSE.md +++ /dev/null @@ -1,21 +0,0 @@ -# MIT License - -Copyright (c) 2024 Daniel Rakotomalala - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. diff --git a/README.Rmd b/README.Rmd index cc1c4f4..05ff9c8 100644 --- a/README.Rmd +++ b/README.Rmd @@ -2,112 +2,210 @@ output: github_document --- - - -```{r, include = FALSE} -knitr::opts_chunk$set( - collapse = TRUE, - comment = "#>", - fig.path = "man/figures/README-", - out.width = "100%", - warning = FALSE -) -``` - -```{r} -library(magrittr) +```{r, echo = FALSE} +knitr::opts_chunk$set(eval = TRUE, echo = TRUE) ``` + +# industtry - -# The industtry package - a toolkit for structured datasets exploitation - +```{=html} -[![Codecov test coverage](https://codecov.io/gh/danielrak/industtry/branch/master/graph/badge.svg)](https://app.codecov.io/gh/danielrak/industtry?branch=master) +``` + +[![Codecov test +coverage](https://codecov.io/gh/danielrak/industtry/branch/master/graph/badge.svg)](https://app.codecov.io/gh/danielrak/industtry?branch=master) [![R-CMD-check](https://github.com/danielrak/industtry/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/danielrak/industtry/actions/workflows/R-CMD-check.yaml) - +``{=html} -This package proposes a set of functions that helps exploiting structured datasets (mostly data frames) with an industrialization approach. Industrialization here means applying as efficiently as possible the same procedure to any number of inputs as long as these inputs have an identified common structure. This idea would have been very difficult to implement without the `purrr::` and `rio::` packages: - -```{r} -purrr:::map(c("purrr", "rio"), citation) %>% print(style = "text") -``` +`industtry` is a toolkit for **industrial-style exploitation of +structured datasets** (primarily data frames), with an emphasis on +**set-level operations**: applying the *same* procedure to *many* inputs +as long as they share an identified structure. +A large part of the package is built around two pragmatic ideas: -**It's best to use it with RStudio.** +1. **"Collections first"**: many workflows fail to scale because they + are designed for a single dataset; `industtry` pushes common tasks + to the *collection* level (importing, inspecting, detecting schema + differences, batch conversion, etc.).\ +2. **"Operational tooling"**: lightweight helpers for day-to-day work + (paths, duplicates, joins checks, string replacements, etc.) that + tend to recur in production data pipelines. -Its contribution is probably in the idea of applying transformations to the set level (of any number of data frames, for e.g), given that numerous existing package help the user exploit one dataset at a time. The functions of this package that are the most in line with this philosophy are: `convert_r()`, `inspect_vars()`, `serial_import()` and `parallel_import()`. +> Some features integrate tightly with **RStudio** (background jobs / +> interactive workflows). Those parts degrade gracefully when RStudio is +> not available (examples are marked accordingly). -This package also provides a set of micro-tools for dealing with usual data transformation tasks, particularly with R/RStudio. +------------------------------------------------------------------------ ## Installation -You can install the development version of industtry from [GitHub](https://github.com/) with: +You can install the development version from GitHub: ``` r # install.packages("devtools") devtools::install_github("danielrak/industtry") ``` -## Example - importations +------------------------------------------------------------------------ + +## API map (what the package covers) + +The exported surface is intentionally broad; the table below is a +*functional map* of the main user-facing tools. -```{r} +``` {r, echo = FALSE} library(industtry) +api_map <- tibble::tribble( + ~ "Area", + ~ "Main intent", + ~ "Key functions", + "Import collections", + "Load multiple datasets into the Global Environment (serialized or parallelized).", + "serial_import(), parallel_import()", + "Batch conversion / renaming", + "Operate through Excel masks to convert file formats or rename files at scale.", + "mask_convert_r(), convert_r(), mask_rename_r(), rename_r()", + "Inspection & profiling", + "Inspect one dataset or a whole folder of datasets; export diagnostics to Excel.", + "inspect(), inspect_write(), inspect_vars()", + "Schema detection / consistency", + "Detect variables across datasets and compare classes/structures.", + "vars_detect*(), vars_compclasses*(), chars_structure*(), detect_chars_structure*()", + "Data hygiene helpers", + "Duplicate diagnostics, join checks, proportions, etc.", + "dupl_show(), dupl_sources(), ljoin_checks(), table_prop()", + "Paths & filesystem", + "Replicate folder structures / move files.", + "folder_structure_replicate(), path_move()", + "Utilities", + "String replacement, global assignment, script location, etc.", + "replace_multiple(), assign_to_global(), current_script_location()" +) + +knitr::kable(api_map) ``` - -Most of (but not all) use cases of data exploitation begins with datasets importation. When you have to work in some way with several datasets simultaneously, it may be useful to be able to import these with a simple code. That is the purpose of the two functions `serial_import()` and `parallel_import()`. - -Suppose you begin with an empty working session: - -```{r} -ls() -``` -Say you want to import two data frames: cars.rds and mtcars.rds stored somewhere accessible to you: +------------------------------------------------------------------------ + +## Core workflow 1 --- import datasets as a collection + +A lot of analysis pipelines begin with import. When you have **many** +datasets, importing them "one by one" (and keeping names consistent) +becomes error-prone. + +`industtry` provides: + +- `serial_import()` to import a set of datasets *sequentially*.\ +- `parallel_import()` to import a set of datasets *in parallel* + (RStudio only). + +``` {r, eval = FALSE} +library(industtry) -```{r} yourdir <- system.file("permadir_examples_and_tests/importations", package = "industtry") -list.files(yourdir) %>% purrr::keep(stringr::str_detect(., "\\.rds$")) +lfiles <- list.files(yourdir, full.names = TRUE) %>% + purrr::keep(stringr::str_detect(., "\\.rds$")) + +lfiles ``` -Note that as long as you have the resources (storage and memory), the procedure is the same for 2, 20, 200, ... data frames. - -Prepare a vector of paths of data frames you want to import: +### Sequential import -```{r} -lfiles <- list.files(yourdir, full.names = TRUE) %>% - purrr::keep(stringr::str_detect(., "\\.rds")) +``` {r, eval = FALSE} +serial_import(lfiles) ``` -One by one importation: - -```{r} -serial_import(lfiles) +### Parallel import (RStudio) -ls() -``` +`{r, eval = FALSE} # RStudio only: parallel_import(lfiles)` -You should have correctly imported the data: - -```{r} -list("cars" = head(cars.rds), - "mtcars" = head(mtcars.rds)) +------------------------------------------------------------------------ + +## Core workflow 2 --- inspect datasets (single + collection) + +When datasets come from heterogeneous sources (different producers, +different time windows, different exports), a fast "schema + variable" +diagnostic helps you converge quickly. + +### Inspect one data frame to Excel + +``` {r, eval = FALSE} +# built-in dataset as a simple example +data(cars) + +out_dir <- tempdir() +inspect_write( + data_frame_name = "cars", + output_path = out_dir, + output_label = "cars" +) + +list.files(out_dir) ``` - -If you want to be able to still use the Console while importing or to avoid interrupting all of the process after one failure: - -```{r} -# Remove from working session to illustrate parallel_import(): -rm(cars.rds, mtcars.rds) + +### Inspect a whole folder of datasets to Excel + +``` {r, eval = FALSE} +mydir <- file.path(tempdir(), "inspect_vars_readme_example") +dir.create(mydir, showWarnings = FALSE) + +saveRDS(cars, file.path(mydir, "cars1.rds")) +saveRDS(mtcars, file.path(mydir, "cars2.rds")) + +inspect_vars( + input_path = mydir, + output_path = mydir, + output_label = "cardata", + considered_extensions = "rds" +) + +list.files(mydir) ``` - + +------------------------------------------------------------------------ + +## Core workflow 3 --- industrialized conversion (mask-driven) + +`convert_r()` is designed for **batch conversion of dataset file +formats** using an Excel mask: a deterministic, auditable interface to +define *what to convert* and *how to name outputs*. + +High-level pattern: + +1. Create a mask with `mask_convert_r()`.\ +2. Fill the mask columns.\ +3. Run `convert_r(mask_filepath, output_path)` (RStudio only). + ```{r, eval = FALSE} -parallel_import(lfiles) +mydir <- system.file("permadir_examples_and_tests/convert_r", package = + "industtry") + +mask_convert_r(output_path = mydir) + +convert_r(mask_filepath = + file.path(mydir, "mask_convert_r.xlsx"), + output_path = mydir) ``` - -This is what you should observe: -![Parallel import jobs](./inst/images/parallel_import_jobs.png) -NB: `parallel_import()` may be slower than `serial_impport()` as the former copies the imported datasets from parallel sessions to the working session and the later does not have this additional step. +------------------------------------------------------------------------ + +## Core workflow 4 — industrialized file renaming (mask-driven) +Similarly, `rename_r()` performs **batch renaming** based on an Excel mask created by `mask_rename_r()`. + +```{r, eval = FALSE} +mydir <- tempfile() +dir.create(mydir) + +saveRDS(cars, file.path(mydir, "cars.rds")) +saveRDS(mtcars, file.path(mydir, "mtcars.rds")) + +mask_rename_r(input_path = mydir) + +list.files(mydir) + +rename_r(mask_filepath = file.path(mydir, "mask_rename_r.xlsx")) +``` diff --git a/README.md b/README.md index 35bac93..8cebdc5 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,5 @@ - - -``` r -library(magrittr) -``` - -# The industtry package - a toolkit for structured datasets exploitation +# industtry @@ -14,146 +8,173 @@ coverage](https://codecov.io/gh/danielrak/industtry/branch/master/graph/badge.sv [![R-CMD-check](https://github.com/danielrak/industtry/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/danielrak/industtry/actions/workflows/R-CMD-check.yaml) -This package proposes a set of functions that helps exploiting -structured datasets (mostly data frames) with an industrialization -approach. Industrialization here means applying as efficiently as -possible the same procedure to any number of inputs as long as these -inputs have an identified common structure. This idea would have been -very difficult to implement without the `purrr::` and `rio::` packages: +`industtry` is a toolkit for **industrial-style exploitation of +structured datasets** (primarily data frames), with an emphasis on +**set-level operations**: applying the *same* procedure to *many* inputs +as long as they share an identified structure. -``` r -purrr:::map(c("purrr", "rio"), citation) %>% print(style = "text") -#> [[1]] -#> Wickham H, Henry L (2023). _purrr: Functional Programming Tools_. R -#> package version 1.0.2, . -#> -#> [[2]] -#> Chan C, Leeper T, Becker J, Schoch D (2023). _rio: A Swiss-army knife -#> for data file I/O_. . -``` +A large part of the package is built around two pragmatic ideas: -**It’s best to use it with RStudio.** +1. **“Collections first”**: many workflows fail to scale because they + are designed for a single dataset; `industtry` pushes common tasks + to the *collection* level (importing, inspecting, detecting schema + differences, batch conversion, etc.). +2. **“Operational tooling”**: lightweight helpers for day-to-day work + (paths, duplicates, joins checks, string replacements, etc.) that + tend to recur in production data pipelines. -Its contribution is probably in the idea of applying transformations to -the set level (of any number of data frames, for e.g), given that -numerous existing package help the user exploit one dataset at a time. -The functions of this package that are the most in line with this -philosophy are: `convert_r()`, `inspect_vars()`, `serial_import()` and -`parallel_import()`. +> Some features integrate tightly with **RStudio** (background jobs / +> interactive workflows). Those parts degrade gracefully when RStudio is +> not available (examples are marked accordingly). -This package also provides a set of micro-tools for dealing with usual -data transformation tasks, particularly with R/RStudio. +------------------------------------------------------------------------ ## Installation -You can install the development version of industtry from -[GitHub](https://github.com/) with: +You can install the development version from GitHub: ``` r # install.packages("devtools") devtools::install_github("danielrak/industtry") ``` -## Example - importations +------------------------------------------------------------------------ -``` r -library(industtry) -``` +## API map (what the package covers) -Most of (but not all) use cases of data exploitation begins with -datasets importation. When you have to work in some way with several -datasets simultaneously, it may be useful to be able to import these -with a simple code. That is the purpose of the two functions -`serial_import()` and `parallel_import()`. +The exported surface is intentionally broad; the table below is a +*functional map* of the main user-facing tools. -Suppose you begin with an empty working session: +| Area | Main intent | Key functions | +|:---|:---|:---| +| Import collections | Load multiple datasets into the Global Environment (serialized or parallelized). | serial_import(), parallel_import() | +| Batch conversion / renaming | Operate through Excel masks to convert file formats or rename files at scale. | mask_convert_r(), convert_r(), mask_rename_r(), rename_r() | +| Inspection & profiling | Inspect one dataset or a whole folder of datasets; export diagnostics to Excel. | inspect(), inspect_write(), inspect_vars() | +| Schema detection / consistency | Detect variables across datasets and compare classes/structures. | vars_detect*(), vars_compclasses*(), chars_structure*(), detect_chars_structure*() | +| Data hygiene helpers | Duplicate diagnostics, join checks, proportions, etc. | dupl_show(), dupl_sources(), ljoin_checks(), table_prop() | +| Paths & filesystem | Replicate folder structures / move files. | folder_structure_replicate(), path_move() | +| Utilities | String replacement, global assignment, script location, etc. | replace_multiple(), assign_to_global(), current_script_location() | -``` r -ls() -#> character(0) -``` +------------------------------------------------------------------------ -Say you want to import two data frames: cars.rds and mtcars.rds stored -somewhere accessible to you: +## Core workflow 1 — import datasets as a collection -``` r -yourdir <- system.file("permadir_examples_and_tests/importations", package = "industtry") +A lot of analysis pipelines begin with import. When you have **many** +datasets, importing them “one by one” (and keeping names consistent) +becomes error-prone. -list.files(yourdir) %>% purrr::keep(stringr::str_detect(., "\\.rds$")) -#> [1] "cars.rds" "mtcars.rds" -``` +`industtry` provides: -Note that as long as you have the resources (storage and memory), the -procedure is the same for 2, 20, 200, … data frames. - -Prepare a vector of paths of data frames you want to import: +- `serial_import()` to import a set of datasets *sequentially*. +- `parallel_import()` to import a set of datasets *in parallel* (RStudio + only). ``` r -lfiles <- list.files(yourdir, full.names = TRUE) %>% - purrr::keep(stringr::str_detect(., "\\.rds")) +library(industtry) + +yourdir <- system.file("permadir_examples_and_tests/importations", package = "industtry") + +lfiles <- list.files(yourdir, full.names = TRUE) %>% + purrr::keep(stringr::str_detect(., "\\.rds$")) + +lfiles ``` -One by one importation: +### Sequential import ``` r serial_import(lfiles) -#> [[1]] -#> NULL -#> -#> [[2]] -#> NULL ``` +### Parallel import (RStudio) + +`{r, eval = FALSE} # RStudio only: parallel_import(lfiles)` + +------------------------------------------------------------------------ + +## Core workflow 2 — inspect datasets (single + collection) + +When datasets come from heterogeneous sources (different producers, +different time windows, different exports), a fast “schema + variable” +diagnostic helps you converge quickly. + +### Inspect one data frame to Excel + ``` r +# built-in dataset as a simple example +data(cars) + +out_dir <- tempdir() +inspect_write( + data_frame_name = "cars", + output_path = out_dir, + output_label = "cars" +) -ls() -#> [1] "cars.rds" "lfiles" "mtcars.rds" "yourdir" +list.files(out_dir) ``` -You should have correctly imported the data: +### Inspect a whole folder of datasets to Excel ``` r -list("cars" = head(cars.rds), - "mtcars" = head(mtcars.rds)) -#> $cars -#> speed dist -#> 1 4 2 -#> 2 4 10 -#> 3 7 4 -#> 4 7 22 -#> 5 8 16 -#> 6 9 10 -#> -#> $mtcars -#> mpg cyl disp hp drat wt qsec vs am gear carb -#> Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4 -#> Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4 -#> Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1 -#> Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1 -#> Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2 -#> Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1 +mydir <- file.path(tempdir(), "inspect_vars_readme_example") +dir.create(mydir, showWarnings = FALSE) + +saveRDS(cars, file.path(mydir, "cars1.rds")) +saveRDS(mtcars, file.path(mydir, "cars2.rds")) + +inspect_vars( + input_path = mydir, + output_path = mydir, + output_label = "cardata", + considered_extensions = "rds" +) + +list.files(mydir) ``` -If you want to be able to still use the Console while importing or to -avoid interrupting all of the process after one failure: +------------------------------------------------------------------------ + +## Core workflow 3 — industrialized conversion (mask-driven) + +`convert_r()` is designed for **batch conversion of dataset file +formats** using an Excel mask: a deterministic, auditable interface to +define *what to convert* and *how to name outputs*. + +High-level pattern: + +1. Create a mask with `mask_convert_r()`. +2. Fill the mask columns. +3. Run `convert_r(mask_filepath, output_path)` (RStudio only). ``` r -# Remove from working session to illustrate parallel_import(): -rm(cars.rds, mtcars.rds) +mydir <- system.file("permadir_examples_and_tests/convert_r", package = + "industtry") + +mask_convert_r(output_path = mydir) + +convert_r(mask_filepath = + file.path(mydir, "mask_convert_r.xlsx"), + output_path = mydir) ``` +------------------------------------------------------------------------ + +## Core workflow 4 — industrialized file renaming (mask-driven) + +Similarly, `rename_r()` performs **batch renaming** based on an Excel +mask created by `mask_rename_r()`. + ``` r -parallel_import(lfiles) -``` +mydir <- tempfile() +dir.create(mydir) -This is what you should observe: +saveRDS(cars, file.path(mydir, "cars.rds")) +saveRDS(mtcars, file.path(mydir, "mtcars.rds")) -
- - -
+mask_rename_r(input_path = mydir) -NB: `parallel_import()` may be slower than `serial_impport()` as the -former copies the imported datasets from parallel sessions to the -working session and the later does not have this additional step. +list.files(mydir) + +rename_r(mask_filepath = file.path(mydir, "mask_rename_r.xlsx")) +```