DATA & DECISION MAKING ASSIGNMENT

Who Gets Counted?

Learn how the people included in a sample shape what survey data can—and cannot—tell us.

Learning Outcome

Evaluate whether conclusions drawn from survey data are supported by the population sampled and the method used to collect the data.

Why This Matters

Surveys influence decisions in public health, marketing, government, education, economics, business, and the social sciences. But a large number of responses does not automatically mean that the results represent everyone we want to understand.

When evaluating survey data, ask not only “How many people responded?” but also “Who had the opportunity and motivation to respond?”

Evaluation Process

1. QUESTION 2. POPULATION 3. SAMPLE 4. MISSING VOICES 5. CLAIM

Five Questions to Ask

Questions for evaluating whether survey evidence supports a conclusion
Step	Ask
Question	What does the researcher actually want to know?
Population	Who is the full group the researcher wants to understand?
Sample	Who actually had the opportunity to participate, and who responded?
Missing Voices	Which members of the population may be less likely to appear in the data?
Claim	What conclusion is actually supported by this sample, and what would be an overreach?

Worked Example: A Coffee Company's Instagram Poll

A coffee company wants to know which new seasonal flavor consumers prefer. It posts a poll to its Instagram followers. Of 6,200 respondents, 68% choose caramel.

Question: Which flavor do consumers prefer?
Population: Potential consumers of the company's products.
Sample: People who follow the company on Instagram and chose to respond.
Missing voices: Potential customers who do not follow the account, do not use Instagram, or chose not to respond.
Supported claim: Caramel was preferred by 68% of respondents to this Instagram poll. The survey alone does not establish that 68% of all potential customers prefer caramel.

Notice: The poll can still provide useful information. The problem occurs when the conclusion becomes broader than the evidence supports.

Guided Practice: Who Might Be Missing?

Consider each situation before revealing the suggested analysis.

Public Health: Vaccination Clinic Survey

A health department surveys people attending a free vaccination clinic about attitudes toward vaccination.

Suggested analysis: People who attend a vaccination clinic may already be more willing to receive vaccines than members of the larger community. The responses are useful for understanding clinic participants but may not represent community-wide attitudes.

Higher Education: Student Engagement Survey

A college emails students a voluntary survey about participation in clubs, events, and other campus activities.

Suggested analysis: Students who are already highly engaged with the college may be more likely to notice or complete an engagement survey. Students who feel disconnected may be less represented.

Workplace: Wellness Program Survey

A company asks employees attending an optional wellness seminar whether workplace wellness programs are valuable.

Suggested analysis: Employees who voluntarily attend a wellness seminar may have more positive attitudes toward wellness programs than employees who chose not to attend.

Apply the Framework: City Transit Survey

A city wants to understand how residents use public transportation and what would make them more likely to use it.

The city posts a survey link on the public transit agency's website and promotes it through the agency's social media accounts. After four weeks, 2,846 people respond.

82% of respondents report using public transportation at least once per week.

The city concludes:

Most city residents regularly use public transportation.

Task

In the next assignment, you will be expected to write a 400–600 word analysis evaluating whether the city's conclusion is supported by the survey.

In your response:

Identify the population the city wants to understand.
Describe who had the opportunity to become part of the sample.
Identify at least one group that may be underrepresented.
Explain how the sampling method may affect the results.
State a conclusion that the survey evidence can reasonably support.

Criteria for Success

A successful analysis will:

Correctly distinguish the intended population from the sample.
Identify a plausible source of sampling or self-selection bias.
Explain why missing or underrepresented groups matter.
Distinguish what the survey results show from claims the data cannot support.
Use evidence from the scenario to support the analysis.

A complete grading rubric will be available on the assignment page.