Matched Question Analysis

The matched question analysis provides an estimate of the amount of learning when adjusted for guessing.
This report is grouped at the question level so the practitioner can make informed adjustments to their class. This is the most import analysis file for assessment and pedagogical improvement. 
In all matched analysis files you will find the raw disaggregated learning types as well as columns labeled 'Gamma', 'Alpha', 'Mu', 'Flow', 'GammaGain', 'R', 'GammaZero', 'GammaGainZero', and 'RZero'.  If you are new to this analysis, focus your attention on γ^\hat \gammaγ^​ (gamma) and γ^/(1−μ^)\hat \gamma/(1-\hat\mu)γ^​/(1−μ^​)(gamma gain).  In simple terms, gamma is the proportion of students who learned the material (as opposed to answered the question correct).   Higher is better, but comparing different questions can be problematic as they can be at different levels of difficulty.  The gamma gain ( γ^/(1−μ^)\hat \gamma/(1-\hat\mu)γ^​/(1−μ^​))  estimate is the proportion of students who learned the material that didn't already know the material.  
Formally, γ^\hat \gammaγ^​(gamma), α^\hat \alphaα^(alpha), and μ^\hat \muμ^​(mu) correspond to "corrected" measurements of the learning types when factoring in the number of students guessing. γ^\hat \gammaγ^​ is corrected positive learning, α^\hat \alphaα^ is corrected negative learning, μ^\hat \muμ^​ is corrected pretest stock knowledge (corrected retained plus corrected negative learning), and flow is the corrected pretest/posttest delta (γ^−α^\hat \gamma-\hat\alphaγ^​−α^). The following equations are used to find the corrected values:
μ^=nl^+rl^−1n−1+nl^+rl^γ^=n(nl^+pl^n+rl^−1)(n−1)2α^=n(nl^n+pl^+rl^−1)(n−1)2\begin{aligned}
		\hat \mu &= \frac{\hat {\text{nl}}+\hat {\text{rl}}-1}{n-1}+\hat {\text{nl}}+\hat {\text{rl}} \\
		\hat \gamma &= \frac{n (\hat {\text{nl}}+\hat {\text{pl}} n+\hat {\text{rl}}-1)}{(n-1)^2} \\
		\hat \alpha &= \frac{n (\hat {\text{nl}} n+\hat {\text{pl}}+\hat {\text{rl}}-1)}{(n-1)^2}
	\end{aligned}μ^​γ^​α^​=n−1nl^+rl^−1​+nl^+rl^=(n−1)2n(nl^+pl^​n+rl^−1)​=(n−1)2n(nl^n+pl^​+rl^−1)​​
where pl^\hat{\text{pl}}pl^​ (positive learning), rl^\hat{\text{rl}}rl^ (retained learning), and nl^\hat{\text{nl}}nl^ (negative learning) refer to the raw learning type values and nnn is the number of answer options.  It is important to use these corrected values as the raw scores can be sensitive to the percent of the class guessing.  Smith and Wagner 2018 details this adjustment.
R=nl^+pl^+rl^−12pl^+(nl^+rl^−1)(1/n+1)R = \frac{\hat {\text{nl}}+\hat{\text{pl}}+\hat{\text{rl}}-1}{2 \hat{\text{pl}}+(\hat{\text{nl}}+\hat{\text{rl}}-1) (1/n+1)}R=2pl^​+(nl^+rl^−1)(1/n+1)nl^+pl^​+rl^−1​
Gamma gain ( γ^/(1−μ^)\hat \gamma/(1-\hat\mu)γ^​/(1−μ^​) ) and RRR (columns R and RZero) were introduced by Smith and White 2021.   RRRcompares the sensitivity of the gamma and gamma gain estimators to probability misspecification. A value in the [−1,1][-1,1][−1,1] range indicates the gain estimator is less sensitive.  Outside of that range, the gamma estimator is less sensitive.  Columns ending in "Zero" indicate that the probability of guessing is determined by assuming that true negative learning is zero instead of using the supplied value.
​
Input Files - Previous
Assessment Map
Next - Output Files
Matched Student Analysis
Last updated 4 months ago