Educational Data Analysis

This week's theme in my workshops (and, by that extension, my posts to you here) is – assessing data collection tools (like surveys) for inclusion and access. Most of my workshops start at the same place – where most have designed at least one survey in the current/past job/education. And then it takes three hours and some meaningful collective learning to realize that planning a survey is much more than just a list of questions. It is an opportunity to connect with your community directly, hear their stories, and understand their experiences and expressions of engagement. In this post, I want to share 5 "red flag" behaviors I often see during a survey design phase: ● When the only questions included are of positive feedback. We all love hearing good things, but only asking for positive feedback disables some real growth opportunities. Example: A question like, "What did you love most about our event?" assumes your respondent only loves the event, and then it offers no room for any different experience. ● When questions are overloaded with complicated words or jargon that only a few will know. You know your mission inside and out, but your community might not understand the same terms you do. Speak in their language. Think of your survey as a conversation. Example: A question like, "How would you rate the efficacy of our donor stewardship activities?" assumes everyone understands the details of "stewardship". ● When every possible question about every possible aspect of the mission is asked – because "why not". Designing surveys – without context – that go on for more than 10-12 minutes - can feel like asking for too much. Be mindful of the respondents and the needs of the data collection. Every question should have a purpose. ● When questions contradict anonymity. Our communities are diverse, and our surveys should hold a neat, safe space for those communities. Ensuring accessibility – balanced with truly useful demographic questions means not harming someone's anonymity – thus making the experience of collecting data easier and meaningful. Example: A survey asking about racial and ethnic diversity in a group of 99% homogenous population (thus making the 1% racially diverse population nervous about the possible breach of anonymity). ● When questions do not offer an 'Opt-Out' option by making everything required. Some questions may feel too personal or uncomfortable for individuals to respond to, and our surveys must create space for that. Give respondents the space to skip a question if they need to. Example: A survey that requires donors to disclose their income range without offering a way to skip the question if they're uncomfortable sharing that information. Stay tuned for a soon-to-be post on what we can do differently then. Have any other such behaviors? Share them here. In the meantime, try some of these resources (all designed to do good with data): https://lnkd.in/gUK-6M_Y #nonprofits #community

Designing effective surveys is not just about asking questions. It is about understanding how people think, remember, decide, and respond. Cognitive science offers powerful models that help researchers structure surveys in ways that align with mental processes. The foundational work by Tourangeau and colleagues provides a four-stage model of the survey response process: comprehension, retrieval, judgment, and response selection. Each step introduces potential for cognitive error, especially when questions are ambiguous or memory is taxed. The CASM model -Cognitive Aspects of Survey Methodology- builds on this by treating survey responses as cognitive tasks. It incorporates working memory limits, motivational factors, and heuristics, emphasizing that poorly designed surveys increase error due to cognitive overload. Designers must recognize that the brain is a limited system and build accordingly Dual-process theory adds another important layer. People shift between fast, automatic responses (System 1) and slower, more effortful reasoning (System 2). Whether a user relies on one or the other depends heavily on question complexity, scale design, and contextual framing. Higher cognitive load often pushes users into heuristic-driven responses, undermining validity. The Elaboration Likelihood Model explains how people process survey content: either centrally (focused on argument quality) or peripherally (relying on surface cues). Users may answer based on the wording of the question, the branding of the survey, or even the visual aesthetics rather than the actual content unless design intentionally promotes central processing. Cognitive Load Theory offers tools for managing effort during survey completion. It distinguishes intrinsic load (task difficulty), extraneous load (poor design), and germane load (productive effort). Reducing the unnecessary load enhances both data quality and engagement. Attention models and eye-tracking reveal how layout and visual hierarchy shape where users focus or disengage. Surveys must guide attention without overwhelming it. Similarly, the models of satisficing vs. optimizing explain when people give thoughtful responses and when they default to good-enough answers because of fatigue, time pressure, or poor UX. Satisficing increases sharply in long, cognitively demanding surveys. The heuristics and biases framework from cognitive psychology rounds out this picture. Respondents fall prey to anchoring effects, recency bias, confirmation bias, and more. These are not user errors, but expected outcomes of how cognition operates. Addressing them through randomized response order and balanced framing reduces systematic error. Finally, modeling approaches like like cognitive interviewing, drift diffusion models, and item response theory allow researchers to identify hesitation points, weak items, and response biases. These tools refine and validate surveys far beyond surface-level fixes.

Survey methodology is not a checklist—it is a science rooted in decades of research and designed to ensure that the numbers we report actually mean something. This landmark volume, authored by pioneers in the field, reframes surveys as a carefully engineered process where sampling, measurement, error reduction, and data quality all converge to produce credible statistics. Blending theory and real-world practice, it introduces the Total Survey Error framework while drawing on a range of large-scale surveys that shape policy, influence markets, and inform global decisions. – It presents the full lifecycle of a survey from design to inference, including key error types: measurement, coverage, sampling, and nonresponse – It explains probability sampling, questionnaire design, interviewing protocols, and mode effects across diverse data collection methods – It introduces cognitive models of response behavior and highlights best practices in question testing, fieldwork, and post-survey processing – It dedicates full chapters to ethical considerations, interviewer bias, weighting, variance estimation, and the scientific underpinnings of quality assurance This is not a book for casual reading—it is a cornerstone reference for researchers, evaluators, and M&E professionals who build systems that depend on getting the data right. If your decisions, policies, or programs rely on the credibility of survey results, then the principles and practices in this text are not optional—they are indispensable.

Are you lost in measurement wonderland? "If you don't know where you're going, any road will take you there." Stop wandering aimlessly in your measurement approach. Like Alice in Wonderland receiving cryptic advice from the Cheshire Cat, too many learning professionals embark on measurement journeys without clear destinations. We collect data after the fact, hoping to stumble upon evidence of impact rather than designing for it from the beginning. The secret to escaping this measurement maze? Make measurement a forethought, not an afterthought. This means: 1: Defining specific micro-behaviors you want to change BEFORE designing your program. 2: Creating tools that capture behavior change at baseline and throughout the learning journey. 3. Using branching logic in your surveys to uncover what truly helps or hinders behavior change. The magic happens in the branching! In your assessment tools, build in branching logic that captures meaningful qualitative data. If someone indicates they "frequently apply" a behavior, ask what positive changes they've seen. If they indicate "sometimes," ask what enables this behavior. If they indicate "never," ask what obstacles they face. This approach gives you actionable intelligence about both the headwinds and tailwinds affecting your behavior change targets - not just whether behaviors changed, but WHY. What's one program where you could apply this micro-behavior approach to measurement? ➡️ Lost in Measurement Wonderland? Here's a playbook that might help you find your way out: https://lnkd.in/giqB8ejQ #learninganddevelopment #measurementstrategy #assessment

On good survey research design (or 28 tips for how to improve your study). Every so often someone will tell me that survey research is dead, it could not be further from the truth. Using surveys to collect data is an essential part of many different research traditions. So what is dead? Bad research design coupled with single-shot surveys and more. So, how do you avoid making mistakes? Me? I religiously adhere to a mix of Dillman's advice and hard-won experience. Dillman's work on Internet and mail-based surveys is canon. If you did not read it in graduate school, then shame on your instructors. You can find a reference to his work here: Dillman, D., Smyth, J. & Christian, L. (2009). Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method, New York: Wiley. You can find a link to 28 tips based on Dillman for how to improve your study here: https://lnkd.in/eEaE2jJs Once you have processed Dillman's work, you are ready to start designing studies. To do so though, you must (1) understand the strengths and limits of your online survey platform, (2) have a clear understanding of how to structure the logical flow of questions, and (3) pilot test, pilot test, and pilot test again. Only once you have really thought through the focus of your work, the platform, the flow, and piloted, should you collect data. If you have done all this well, you should be able to comment on the phenomenon of interest to you in a study! Best of luck! #academicresearch