As it stands, flightsbr uses data.table::fread() to read the csv files from ANAC. This is relatively fast, but it requires that the entire data set is loaded to memory. This can be a problem when reading data for a whole year or multiple months.
One alternative would be to use instead duckplyr::read_csv_duckdb(), which returns a "duckplyr_df" "tbl_df" "tbl" "data.frame" with lazy evaluation. We would need to add a new argument that let's users decide whether the output should be loaded to memory (current behavior, default) or as lazy table.
A critical concern here is that oftentimes, the .csv files from ANAC have problems with column separadors. We know data.table does a pretty good job and dealing with this issue. We would need to check whether duckplyr would be able to address this problem as well to amke sure duckplyr is a suitable option here.
library(duckplyr)
# Create simple CSV file
path <- tempfile("duckplyr_test_", fileext = ".csv")
write.csv(data.frame(a = 1:3, b = letters[4:6]), path, row.names = FALSE)
# Reading is immediate
df <- read_csv_duckdb(path)
# Names are always available
names(df)
# Materialization upon access is turned off by default
try(print(df$a))
# Materialize explicitly
collect(df)$a
df <- read_csv_duckdb(path, prudence = "lavish")
head(df)
As it stands, flightsbr uses
data.table::fread()to read the csv files from ANAC. This is relatively fast, but it requires that the entire data set is loaded to memory. This can be a problem when reading data for a whole year or multiple months.One alternative would be to use instead
duckplyr::read_csv_duckdb(), which returns a"duckplyr_df" "tbl_df" "tbl" "data.frame"with lazy evaluation. We would need to add a new argument that let's users decide whether the output should be loaded to memory (current behavior, default) or as lazy table.A critical concern here is that oftentimes, the .csv files from ANAC have problems with column separadors. We know data.table does a pretty good job and dealing with this issue. We would need to check whether duckplyr would be able to address this problem as well to amke sure duckplyr is a suitable option here.