Crowd-sourced labels don’t always reflect your target users. PAIR reweights training data so your model learns from the perspectives that matter most for your application.

AI models are only as good as their training data. This paper shows how 50+ years of survey science can help ML researchers collect better data - leading to fairer, more accurate models.

The order you show items to annotators matters. We found that changing the sequence of examples can shift labeling decisions - another reason to carefully design your annotation pipeline.

Small changes in how you ask annotators to label data can dramatically change your model’s behavior. We tested 5 versions of a hate speech labeling task and found significant differences in model performance.

People consistently overreport how much news they consume. We compared survey responses to passively collected behavioral data and found large gaps, highlighting the need to combine data sources for accurate measurement.

Do survey respondents learn to game the questionnaire over time? We tested this in the U.S. Consumer Expenditure Survey and found the opposite. Responses actually improve in later waves.

High response rates don’t always mean good data. Using seven rounds of the European Social Survey, we show that when interviewers help select who gets surveyed, high response rates can actually signal worse data quality.

How you structure filter questions in a survey affects the answers you get. We linked survey responses to administrative records and found that respondents answer strategically to shorten the interview.

Web panels miss people without internet access. The LISS panel provided devices to non-internet households. We tested whether this extra effort actually reduced bias in research findings.