Quick answer
Data cleaning in health research requires standardized coding, missing-data strategy, outlier review, and reproducible audit trails. A structured pipeline improves model stability, prevents analytical errors, and ensures that published findings reflect true patterns instead of preventable data-quality artifacts.
Key takeaways
- Preserve raw data as immutable and create analysis data through versioned, repeatable code.
- Write validation rules from the protocol and data dictionary before reviewing unusual values.
- Investigate outliers as possible clinical signals or data errors; do not delete them automatically.
- Document every correction, derivation, exclusion, and unresolved query in an audit trail.
Build a Data Dictionary First
Define variable names, units, allowed values, and transformations before analysis. This reduces ambiguity across collaborators.
Missingness and Outliers
Profile missingness mechanisms and identify implausible observations using domain-aware thresholds rather than automatic deletion.
Auditability and Reproducibility
Maintain versioned scripts and cleaning logs. Reproducible pipelines are easier to validate during peer review and regulatory checks.
Practical method
Step-by-Step Workflow
- 1
Profile the raw extract
Confirm row counts, identifiers, data types, ranges, category levels, date formats, missingness, and duplicate patterns without overwriting source data.
- 2
Run validation rules
Flag impossible and implausible values, inconsistent units, broken skip logic, date-order violations, and cross-field contradictions.
- 3
Resolve and derive
Trace anomalies to their source, document approved corrections, define analysis variables in code, and distinguish structural from unexpected missingness.
- 4
Review and freeze
Independently review high-impact transformations, reconcile the final cohort, produce a cleaning report, and tag the locked analysis dataset.
Worked example
Cleaning a multicenter health registry
- Scenario
- A registry combines records from five hospitals with different date formats, weight units, missing-value codes, and patient identifiers.
- Approach
- Standardize source formats in a staging layer, map all missing codes to documented categories, convert units while retaining originals, resolve duplicate patients with approved rules, and generate site-level discrepancy summaries.
- Interpretation
- The final dataset should be reproducible from the preserved extracts. Site differences in missingness or implausible values remain findings to investigate, not nuisances to conceal.
Common Mistakes to Avoid
- Editing the only copy of a spreadsheet
- Converting missing values to zero
- Deleting all statistical outliers
- Applying undocumented manual fixes
Frequently Asked Questions
Is deleting missing records acceptable?
Only when missingness is minimal and plausibly random; otherwise consider imputation or model-based handling.
Why keep a cleaning log?
Logs preserve transparency, support QA review, and allow exact replication of analytical datasets.