Small Details In Evaluation

1. Data nuances and coding discrepancies

• Normalize variable definitions across systems so eligibility logic is consistent.
• Document coding exceptions (for example, assistive-device logic) before final cohort lock.
• Maintain an up-to-date participating clinician ID file to avoid treatment-group leakage.

2. Participant switching and behavior

• Track crossover: treatment-to-usual-care and usual-care-to-similar-service movement.
• Choose estimand up front: intent-to-treat vs. re-eligibility-based follow-up.
• Quantify dilution risk when switching rates are high.

3. Outliers and site-level influence

• Run leave-one-site-out sensitivity tests to assess concentration risk.
• Handle extreme-cost outliers with a prespecified rule (for example, 99th percentile trimming/winsorizing).
• Report both trimmed and untrimmed estimates for transparency.

4. Unmeasured environmental factors

• Care setting differences (home vs. congregate setting) can shift observed ED/hospital use.
• Family and caregiver capacity can materially affect acute utilization independent of program quality.
• Capture these risks in limitations and, when feasible, stratified analyses.

5. Interaction with other programs

• Check overlap with ACOs and other concurrent initiatives targeting the same population.
• Flag shared-attribution risk in interpretation of savings and utilization changes.

6. Baseline rigor checks

• Verify pre-trends over at least two years, not just one baseline year.
• Evaluate geographic expansion effects that may change cohort case-mix year over year.
• Re-run results on stable-coverage subcohorts as a robustness check.

Related methods

Previous: Index date in program evaluation  |  Next: How to account for outliers in spending data

• Do an evaluation on a napkin
• How to evaluate a healthcare program

For more granular data, more recent data, or scientific analysis support, please email us.

Back to Evidence & Performance