Veet import efficiency

[ThomasKraft]

This PR stems from issue #85 and generally attempts to speed up and manage memory efficiency for VEET import, which can cause particular problems due to the large nature of data types such as TOF. Speed is improved here using the data.table package for import. There are also some relevant changes for auto.plot defaults when TOF data are imported to avoid unnecessary processing time. The basic testing code pasted below shows a general outcome I found of these changes cutting VEET import processing time by ~50%:

Ultimately I discovered that VEET files are large enough that the main memory issue occurs simply due to using map:::purr() on files that are of sufficient size to overwhelm a normal laptop when they are all loaded into memory. Packages such as GGIR manage this same type of issue by batch loading files and processing them into epochs in chunks before discarding the raw files during the process. Although something similar could be done here, the LightLogR infrastructure is sufficient for doing this manually with a short custom scripts (which is currently working well for me). So, I favor not introducing any changes for now.

Basic benchmarking code:

setwd(".")

cran_lib <- tempfile("cran-lib-")
local_lib <- tempfile("local-lib-")

dir.create(cran_lib)
dir.create(local_lib)

pkg <- "LightLogR"

# Install CRAN version into its own library
install.packages(
  pkg,
  lib = cran_lib,
  repos = "https://cloud.r-project.org",
  dependencies = TRUE
)

# Install your current local branch into its own library
install.packages(
  c("remotes", "pak"),
  repos = "https://cloud.r-project.org"
)

remotes::install_local(
  path = ".",
  lib = local_lib,
  dependencies = TRUE,
  upgrade = "never",
  force = TRUE
)

unload_lightlogr <- function() {
  if (paste0("package:", pkg) %in% search()) {
    detach(paste0("package:", pkg), unload = TRUE, character.only = TRUE)
  }
  
  if (pkg %in% loadedNamespaces()) {
    unloadNamespace(pkg)
  }
}

run_with_version <- function(lib, expr) {
  unload_lightlogr()
  library(pkg, lib.loc = lib, character.only = TRUE)
  force(expr)
}

filenames <- list.files(path = "PATH_TO_VEET_DATA/",
                        recursive = T, pattern = "Sensor_Data|sensor_data", full.names=TRUE)

microbenchmark::microbenchmark(
cran_time = {
  cran_result <- run_with_version(
    cran_lib,
    {
      LightLogR::import$VEET(
        filename = filenames[1:15],
        modality="TOF",
        auto.plot=F,
        silent=T
      )
    }
  )
},

local_time = {
  local_result <- run_with_version(
    local_lib,
    {
      LightLogR::import$VEET(
        filename = filenames[c(1:15)],
        modality="TOF",
        auto.plot=F,
        silent=T
      )
    }
  )
},
times = 2)

[Copilot]

Pull request overview

Improves performance and memory characteristics of VEET imports (especially TOF) by switching to data.table-based parsing and by reducing costly work during duplicate detection and auto-plotting.

Changes:

• Refactor VEET import expression to use data.table::fread() and rbindlist() with explicit column typing.
• Optimize duplicate detection to avoid full-width scans unless duplicates are plausible, and avoid expensive gap detection during TOF auto-plot.
• Add VEET-focused tests, update NEWS.md, and add data.table to package imports.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
R/import_expressions.R	Reworks VEET import parsing to use data.table and sets Datetime via data.table::set().
R/import_LL.R	Optimizes duplicate detection prefiltering and special-cases VEET/TOF auto-plot to skip gap finding.
tests/testthat/test-import_LL.R	Adds VEET/TOF import and auto-plot tests.
DESCRIPTION	Adds data.table as an imported dependency.
NEWS.md	Documents VEET import efficiency improvements and TOF-specific optimizations.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.