Clean and standardize a raw survey dataset

Orchestrates the full cleaning pipeline for a single dataset. Each step is delegated to a focused internal helper (see clean_dataset.R source) so individual steps can be tested and extended independently.

Usage

clean_dataset(
  dt,
  config,
  year_tag = NULL,
  dataset_label = "",
  apply_missing = FALSE
)

Arguments

dt

A data.table (or coercible object).

config

Config object returned by load_config().

year_tag

Optional integer year added as a year column.

dataset_label

Human-readable label used in warnings and logs.

apply_missing

If TRUE, recode numeric missing codes to NA. Default FALSE (explicit opt-in to avoid silent data loss).

Value

A list:

data

The cleaned data.table.

issues

data.table with columns variable, issue_type, detail - one row per coercion failure or value violation.

Details

Steps in order:

1. Normalise column names (.normalise_colnames())

2. Match columns to canonical names (match_columns())

3. Apply renames (.apply_renames())

4. Keep only canonical columns (.select_canonical_cols())

5. Recode values (.apply_recodes())

6. Optionally replace missing codes with NA (.apply_missing_codes())

7. Safe type coercion (.coerce_all_columns())

8. Value conformance validation (.validate_all_values())

9. Inject year column (.inject_year())

10. Remove duplicate/empty rows (.drop_empty_rows())

Examples

if (FALSE) { # \dontrun{
yml <- system.file("extdata", "variable_map.yml", package = "cuci")
csv <- system.file("extdata", "test-data.csv",    package = "cuci")
cfg <- load_config(yml)
raw <- data.table::fread(csv)
out <- clean_dataset(raw, cfg, year_tag = 2023, dataset_label = "test")
out$data
out$issues
} # }