Matched Question Analysis

Assessment Disaggregation

Search…

Assessment Disaggregation

Quick Start Series

Why Do This?

Exam Files and Assessment Maps

Interpreting Results

Input Files

Exam Files

Assessment Map

Output Files

Matched Question Analysis

Matched Student Analysis

Unmatched Results

Privacy and Security

Software Download

Matched Question Analysis

The matched question analysis provides an estimate of the amount of learning when adjusted for guessing.

Matched question analysis uses data from students who took both the pretest and the posttest to calculate learning values grouped by question; students who took only one of the two exams are removed from the analysis. This is the most important analysis file for assessment and pedagogical improvement as it provides estimates of the percent of students who learned a given question.  This data can be compared over time as instructors make changes to their course or program. 
If you are new to this analysis, focus your attention on γ^\hat \gammaγ^​
 (gamma) and γ^/(1−μ^)\hat \gamma/(1-\hat\mu)γ^​/(1−μ^​)
(gamma gain).  In simple terms, gamma is the proportion of students who learned the material (as opposed to correctly answered the question).   Higher is better but comparing different questions can be problematic as they can be at different levels of difficulty.  The gamma gain ( γ^/(1−μ^)\hat \gamma/(1-\hat\mu)γ^​/(1−μ^​)
)  estimate is the proportion of students who learned the material that didn't already know the material.    In addition to these measures of learning, 'raw' learning values are included in the output file; if you are new to this analysis these can be ignored.  
Formally, γ^\hat \gammaγ^​
(gamma), α^\hat \alphaα^
(alpha), and μ^\hat \muμ^​
(mu) correspond to 'corrected' measurements of the learning types when factoring in the number of students guessing; these adjustments assume that the probability of correctly guessing can be estimated, which is more reasonable in higher-stakes testing environments. γ^\hat \gammaγ^​
 is corrected positive learning, α^\hat \alphaα^
 is corrected negative learning, μ^\hat \muμ^​
 is corrected pretest stock knowledge (corrected retained plus corrected negative learning), and flow is the corrected pretest/posttest delta (γ^−α^\hat \gamma-\hat\alphaγ^​−α^
). The following equations are used to find the corrected values:
μ^=nl^+rl^−1n−1+nl^+rl^γ^=n(nl^+pl^n+rl^−1)(n−1)2α^=n(nl^n+pl^+rl^−1)(n−1)2\begin{aligned}
		\hat \mu &= \frac{\hat {\text{nl}}+\hat {\text{rl}}-1}{n-1}+\hat {\text{nl}}+\hat {\text{rl}} \\
		\hat \gamma &= \frac{n (\hat {\text{nl}}+\hat {\text{pl}} n+\hat {\text{rl}}-1)}{(n-1)^2} \\
		\hat \alpha &= \frac{n (\hat {\text{nl}} n+\hat {\text{pl}}+\hat {\text{rl}}-1)}{(n-1)^2}
	\end{aligned}μ^​γ^​α^​=n−1nl^+rl^−1​+nl^+rl^=(n−1)2n(nl^+pl^​n+rl^−1)​=(n−1)2n(nl^n+pl^​+rl^−1)​​
where pl^\hat{\text{pl}}pl^​
 (positive learning), rl^\hat{\text{rl}}rl^
 (retained learning), and nl^\hat{\text{nl}}nl^
 (negative learning) refer to the raw learning type values and nnn
 is the number of answer options.  It is important to use these corrected values as the raw scores can be sensitive to the percent of the class guessing.  Smith and Wagner 2018 details this adjustment.
R=nl^+pl^+rl^−12pl^+(nl^+rl^−1)(1/n+1)R = \frac{\hat {\text{nl}}+\hat{\text{pl}}+\hat{\text{rl}}-1}{2 \hat{\text{pl}}+(\hat{\text{nl}}+\hat{\text{rl}}-1) (1/n+1)}R=2pl^​+(nl^+rl^−1)(1/n+1)nl^+pl^​+rl^−1​
Gamma gain ( γ^/(1−μ^)\hat \gamma/(1-\hat\mu)γ^​/(1−μ^​)
 ) and RRR
 (the column R) were introduced by Smith and White 2021.   RRR
compares the sensitivity of the gamma and gamma gain estimators to probability misspecification. An   RRR
 value between -1 and 1 indicates the gamma gain estimator is less sensitive to probability misspecification.  AnRRR
value greater than 1 or less than -1 indicates the gamma estimator is less sensitive. The column RMinSensitivity present the less sensitive estimator based on the value in the column R.
Columns ending in 'Zero' indicate that the probability of guessing is determined by assuming that true negative learning is zero instead of using the supplied value.  With these columns, α^\hat \alphaα^
 is assumed to be zero in the equation above.  This assumption allows the system to solve for the implied probability of correctly guessing.  This implied probability is then used to calculate γ^\hat \gammaγ^​
 (column GammaZero), γ^/(1−μ^)\hat \gamma / (1-\hat \mu)γ^​/(1−μ^​)
 (column GammaGainZero), and RRR
 (column RZero).  These columns are useful when the probability of correctly guessing could be substantially incorrect.  This includes situations like low-stakes exams and incentives that manipulate the propensity of a student to guess.
If "Include Summary Rows" under the "Options" menu is checked then means (averages), standard deviations and observation counts will be provided in the analysis file.  
​

Input Files - Previous

Assessment Map

Next - Output Files

Matched Student Analysis

Last modified 6mo ago

Copy link