Stephanie Eckman

AI models are only as good as their training data, but most ML researchers aren't trained in data collection. This paper shows how 50+ years of survey science can help build better datasets—from writing clearer labeling instructions to recruiting more representative annotators.

Small changes in how you ask annotators to label data can lead to big differences in model performance. We tested this with hate speech detection and found that tweaking instructions or scales changes what the model learns—sometimes for better, sometimes for worse.

Crowdsourced labels often don't match what your actual users think. PAIR fixes this by reweighting training data so your model learns from the perspectives that matter most for your specific application.

When humans rate AI suggestions, their own biases sneak into the feedback. This creates a hidden loop where models learn to amplify human prejudices rather than correct them—a critical blind spot for AI alignment.

Do survey respondents learn to game the system over time? We tested whether people report fewer purchases in later interviews to shorten the questionnaire. Surprisingly, the opposite happens—experienced respondents actually give higher quality data.

High response rates are supposed to mean better data, but that's not always true. When interviewers help pick who gets surveyed, high response rates can actually signal worse data quality—because interviewers cherry-pick easy respondents.

Government records and surveys both contain errors, but which source is more accurate? This statistical framework lets you compare data quality across different sources without assuming either one is perfect.