2015 • No. 15–7
Research Data Reports
Identifying and Evaluating Sample Selection Bias in Consumer Payment Surveys 
Making meaningful inferences based on survey data depends on the ability to recognize and adjust for discrepancies between the survey respondents and the target population; this partly involves understanding how survey samples differ with respect to heterogeneous clusters of the population. Ex post adjustments for unbiased population parameter estimates are usually based on easily measured variables with known distributions in the target population, like age, gender, or income. This paper focuses on identifying and assessing the effect of an overlooked source of heterogeneity and potential selection bias related to household structure and dynamics. In household economies, tasks are often concentrated among a subset of the household members, so individual differences in behavior are related to performing different roles within the household. When the sampling involves selecting individuals from within households, a tendency to choose certain types of members for participation in a survey can result in unrepresentative samples and biased estimates for any variable relating to the respondent's household duties. The Boston Fed's Survey of Consumer Payment Choice (SCPC) seeks to estimate parameters, such as the average number of monthly payments, for the entire U.S. population. This data report exploits the fact that in the 2012 SCPC some respondents come from the same household, a unique feature that enables a study of the presence and ramifications of this type of selection bias when asking about household financial decisionmaking and payment choice. Using a two-stage statistical analysis, the survey answers are adjusted for a response error to estimate a latent variable that represents each respondent's share of financial responsibility for the household.
Key Findings
-
It appears that the SCPC is oversampling household members who have more financial responsibility within the household.
-
A higher level of financial responsibility corresponds to a higher share of payments using checks and electronic payment methods, but not cash.
- Selection bias affects the population mean estimates for the number of monthly payments made by U.S. consumers. Electronic payments tend to be the most overestimated, with estimated deviations near 12 percent. Checks and card payments show errors around 8 and 7 percent, respectively, but for checks the range of estimated errors extends to larger values. Cash estimates are not affected by the selection bias.
Implications
Researchers who rely on data from surveys based on answers provided by an individual member should be aware that intra-household variability in behavior may introduce potential bias by over-sampling or under-sampling individuals with certain types of roles. Overcoming such bias depends on better understanding of how household roles relate to variables of interest and the ability to quantify respondents' roles through survey questions. In certain cases, using households as the sampling unit might also improve the quality of inference.