Skip to contents

Run the full survey data cleaning pipeline

Source: R/pipeline.R
run_pipeline.Rd

Processes every file in file_manifest through the complete pipeline (read -> match -> log -> clean -> validate) and returns the results either merged into one data.table or as a named list of separate data.tables.

Usage

run_pipeline(
  file_manifest,
  config_file = "config/variable_map.yml",
  log_dir = "logs/matching",
  output_path = NULL,
  apply_missing = FALSE,
  merge = FALSE
)

Arguments

file_manifest

A data.table or data.frame with columns:

path

Path to each raw input file (CSV, TSV, or XLSX).

year_tag

Integer year to attach, or NA if already in the file.

config_file

Path to the YAML variable map. Default "config/variable_map.yml".

log_dir

Directory for audit logs. Default "logs/matching".

output_path

Where to write the output. A file path when merge = TRUE; a directory path when merge = FALSE. Set NULL to skip writing entirely.

apply_missing

Passed to clean_dataset(). Default FALSE.

merge

Logical. If TRUE, stack all cleaned datasets into one master data.table. If FALSE (default), return a named list of separate data.tables, one per input file.

Value

  • When merge = TRUE: a single data.table with year as the first column.

  • When merge = FALSE: a named list of data.tables, one per file, keyed by the file's basename.

Output modes

  • merge = TRUE (default FALSE): stacks all cleaned datasets row-wise into a single data.table using data.table::rbindlist(). Missing columns across datasets are padded with NA. output_path should be a full file path (e.g. "data/clean/master.csv").

  • merge = FALSE: returns a named list where each element is the cleaned data.table for one file, keyed by the file's basename. output_path should be a directory (e.g. "data/clean"); one CSV per dataset is written there.

Examples

if (FALSE) { # \dontrun{
library(data.table)
manifest <- data.table(
  path     = c(
    system.file("extdata", "survey_2022.csv", package = "cuci"),
    system.file("extdata", "survey_2023.csv", package = "cuci")
  ),
  year_tag = c(2022L, 2023L)
)
yml <- system.file("extdata", "variable_map.yml", package = "cuci")

# Return separate datasets (default)
result_list <- run_pipeline(manifest, config_file = yml, merge = FALSE)
result_list[["survey_2022.csv"]]

# Return one merged dataset
result_merged <- run_pipeline(manifest, config_file = yml, merge = TRUE,
                              output_path = "data/clean/master.csv")
} # }