Matched Question Analysis

The matched question analysis provides an estimate of the amount of learning when adjusted for guessing.

Matched question analysis uses data from students who took both the pretest and the posttest to calculate learning values grouped by question; students who took only one of the two exams are removed from the analysis. This is the most important analysis file for assessment and pedagogical improvement as it provides estimates of the percent of students who learned a given question. This data can be compared over time as instructors make changes to their course or program.

If you are new to this analysis, focus your attention on $\hat \gamma$ (gamma) and $\hat \gamma/(1-\hat\mu)$ (gamma gain). In simple terms, gamma is the proportion of students who learned the material (as opposed to correctly answered the question). Higher is better but comparing different questions can be problematic as they can be at different levels of difficulty. The gamma gain ( $\hat \gamma/(1-\hat\mu)$) estimate is the proportion of students who learned the material that didn't already know the material. In addition to these measures of learning, 'raw' learning values are included in the output file; if you are new to this analysis these can be ignored.

Formally, $\hat \gamma$ (gamma), $\hat \alpha$ (alpha), and $\hat \mu$ (mu) correspond to 'corrected' measurements of the learning types when factoring in the number of students guessing; these adjustments assume that the probability of correctly guessing can be estimated, which is more reasonable in higher-stakes testing environments. $\hat \gamma$ is corrected positive learning, $\hat \alpha$ is corrected negative learning, $\hat \mu$ is corrected pretest stock knowledge (corrected retained plus corrected negative learning), and flow is the corrected pretest/posttest delta ($\hat \gamma-\hat\alpha$). The following equations are used to find the corrected values:

$$\begin{aligned} \hat \mu &= \frac{\hat {\text{nl}}+\hat {\text{rl}}-1}{n-1}+\hat {\text{nl}}+\hat {\text{rl}} \\ \hat \gamma &= \frac{n (\hat {\text{nl}}+\hat {\text{pl}} n+\hat {\text{rl}}-1)}{(n-1)^2} \\ \hat \alpha &= \frac{n (\hat {\text{nl}} n+\hat {\text{pl}}+\hat {\text{rl}}-1)}{(n-1)^2} \end{aligned}$$

where $\hat{\text{pl}}$ (positive learning), $\hat{\text{rl}}$ (retained learning), and $\hat{\text{nl}}$ (negative learning) refer to the raw learning type values and $n$ is the number of answer options. It is important to use these corrected values as the raw scores can be sensitive to the percent of the class guessing. Smith and Wagner 2018 details this adjustment.

$$R = \frac{\hat {\text{nl}}+\hat{\text{pl}}+\hat{\text{rl}}-1}{2 \hat{\text{pl}}+(\hat{\text{nl}}+\hat{\text{rl}}-1) (1/n+1)}$$

Gamma gain ( $\hat \gamma/(1-\hat\mu)$ ) and $R$ (the column R) were introduced by Smith and White 2021. $R$ compares the sensitivity of the gamma and gamma gain estimators to probability misspecification. An $R$ value between -1 and 1 indicates the gamma gain estimator is less sensitive to probability misspecification. An $R$ value greater than 1 or less than -1 indicates the gamma estimator is less sensitive. The column RMinSensitivity present the less sensitive estimator based on the value in the column R.

Columns ending in 'Zero' indicate that the probability of guessing is determined by assuming that true negative learning is zero instead of using the supplied value. With these columns, $\hat \alpha$ is assumed to be zero in the equation above. This assumption allows the system to solve for the implied probability of correctly guessing. This implied probability is then used to calculate $\hat \gamma$ (column GammaZero), $\hat \gamma / (1-\hat \mu)$ (column GammaGainZero), and $R$ (column RZero). These columns are useful when the probability of correctly guessing could be substantially incorrect. This includes situations like low-stakes exams and incentives that manipulate the propensity of a student to guess.

If "Include Summary Rows" under the "Options" menu is checked then means (averages), standard deviations and observation counts will be provided in the analysis file.

Comparing to Critical Values

You can lookup critical values of both gamma ( $\hat \gamma$ ) and gamma gain ( $\hat \gamma/(1-\hat\mu)$ ) under the help menu.

These critical values are generated through simulations where there was no learning. Thus the true gamma ( $\gamma$ ) and gamma gain ( $\gamma/(1-\mu)$ ) values are zero in these simulations. However, a distribution of outcomes is produced from randomness (due to guessing). If your observed gamma ( $\hat \gamma$ ) or gamma gain ( $\hat \gamma/(1-\hat\mu)$ ) is greater than what is displayed in the table, you can say that the value is greater than what you would expect from randomness alone. The tables are available for 3-6 question options and classes from 15 to 300 students.