# Assessment Disaggregation
Disaggregates pre- and post-test answers into value-added learning scores (Walstad and Wagner 2016), adjusts for guessing (Smith and Wagner 2018), and computes gain scores (Smith and White 2021).
## What this Does
In the Spring of 2016, [Walstad and Wagner](https://doi.org/10.1080/00220485.2016.1146104) released a paper suggesting that the pretest/posttest delta is insufficient in assessing learning outcomes. In 2018, [Smith and Wagner](https://bensresearch.com/smithwagner) showed that this disaggregation should be adjusted using the probability of correctly guessing. In 2021, [Smith and White](https://doi.org/10.1177/01466216211013905) expanded the analysis with adjusted gain estimators. As this analysis of learning is extremely useful to both research and assessment, this analysis is common. However, performing such a disaggregation and adjustment is time intensive, especially if the questions appear in a different location (or order) on the pretest and posttest.
This software automatically performs this analysis from raw exam files (currently supporting Akindi, Scantron, Canvas Quizzes, Quick Key, Moodle, Google Forms quizzes, ZipGrade, and some Blackboard systems). Questions can be in any order as the optional map file can describe how the same question can appear in different locations.
Main Software Interface
The user of this software simply clicks pretest button and selects the pretest file, and clicks on the posttest button and selects the posttest file. The user can then (optionally) select an assessment map file if the questions are not in the same order.
# Why Do This?
In this video we discuss why a researcher, instructor, or department might want to use the assessment disaggregation software to improve their measures of student learning.
{% embed url="" %}
# Exam Files and Assessment Maps
In this video we show how to load a pretest, posttest, and assessment map into the Assessment Disaggregation software. We then save a matched question analysis file for later review.
{% embed url="" %}
# Interpreting Results
In the third video of the Assessment Disaggregation quick start series, we interpret the results generated at the end of video two.
{% embed url="" %}
# Exam Files
The software reads most common exam formats without modification. Just make sure your students fill in their student id.
With rare exception, providing the software the exam output file will 'just work.' In all cases, you need to provide the exam results in CSV format - which is the default in most cases. The following describes how the software determines the exam format.
#### **Scantron Format**
The software assumes Scantron formatted exam files have the following characteristics:
1. The first row is the answer key
2. The second column is the student id
3. The third column of the first row specifies the number of questions
4. The fourth column and higher are the student answers
The software will compare the answer key (first row) to each student's answer thereby determining if the student correctly answered the question.
#### **Canvas Format**
If the file is in Canvas Student Analysis format (under Quiz Statistics), the program assumes it has the following characteristics.
1. The first row of the file specifies the column names.
2. There is a column named 'attempt'.
3. The column 'id' or 'sis\_id' contain the student ids. If both exist, 'sis\_id' will be used.
4. Students receive some positive number of points when they answer correct. If they receive fewer than these points, the software will assume they answered incorrect.
#### **ZipGrade, Quick Key, Moodle, and Akindi Format**
If the exam files are ZipGrade/Quick Key/Moodle/Akindi formatted, the software assumes the files have the following characteristics:
1. The first row of the file specifies the column names
2. There is a column named 'id', 'student id', 'external id', 'id number', or 'zipgrade id' specifying a unique id number for each student
3. Each exam question column is named 'Q' followed by the question number (e.g. 'Q67' for question 67). Additional information in the column name is ignored.
4. In each question column cell, a positive number indicates the student answered it correct and {'0', ' ', 'a', 'b', 'c', 'd', 'e'} indicates the student got the question wrong. Both ZipGrade and Quick Key place a '1' in any cell where the student correctly answered the question. ZipGrade places a '0' in any cell where the student got it wrong while Quick Key puts the (incorrect) student response ({' ', 'a', 'b', 'c', 'd', 'e'}). Akindi's columns are the same but the values differ.
5. If the column 'Grade' exists in the input file, the program will assume the 'Q' columns are in Akindi's format. Each cell containing {'a', 'b', 'c', 'd', 'e'} will be marked correct while all other values will be counted as incorrect. Akindi places the response in each cell of the column; if the response is incorrect, the response is wrapped in parentheses.
Note that ZipGrade's *standard* output format complies with characteristics above. ZipGrade's *full* output format does not and is incompatible with the program.
#### Google Forms Quizzes
Google Forms quizzes are slightly different than the other formats highlighted here as Forms do not have question numbers and student ids aren't automatically collected. This results in two important points to keep in mind:
1. The instructor must place a student id question on the quiz. This column can be named 'id', 'student id', 'external id', or 'id number'. The question doesn't need to be worth a positive number of points.
2. The question number is implied by the order of the questions on the Form.
The responses can be downloaded from the responses tab, more section (three dots), or download responses (.csv). The software will assume any question worth a positive number of points is a quiz question. Other questions will be ignored.
#### Blackboard Format
In some Blackboard installs, the instructor can download comma separated by question and user. This support is considered experimental as the format seems to vary by institution. The program assumes the following:
1. There are columns named 'Question ID', 'Possible Points', 'Auto Score', and 'Manual Score'.
2. Either a column for id ('id', 'student id', 'external id', or 'id number') or username ('username') appears in the file.
3. The values in the 'Question ID' column end with a numerical value representing the question number (e.g. 'Question ID 14').
#### Custom Format
If your exam software is not currently supported, you can create a custom exam spreadsheet with a column for student id followed by columns for each question. An example spreadsheet might look like the following:
| id | Q1 | Q2 | Q3 |
| ----- | -- | -- | -- |
| 10001 | 1 | 0 | 1 |
| 10002 | 0 | 1 | 0 |
| 10003 | 1 | 1 | 1 |
Where "1" indicates the student answered the question correct and "0" indicates the student did not answer the question correct.
# Assessment Map
By default, the software assumes that questions have the selected answer options (e.g. 4) and appeared in the same order on the pretest and posttest. The assessment map can override these settings.
An assessment map file is any four column CSV file with the following columns: Q (question number), Exam1 (where the question appeared on the pretest), Exam2 (where the question appeared on the posttest), and Options (number of answer options for a given question). The following table is an example assessment map where the questions appeared in the opposite order on posttest as they did on the pretest. The output files will be in pretest order as 'Q' and 'Exam1' are in the same order.
| Q | Exam1 | Exam2 | Options |
| - | ----- | ----- | ------- |
| 1 | 1 | 5 | 4 |
| 2 | 2 | 4 | 4 |
| 3 | 3 | 3 | 5 |
| 4 | 4 | 2 | 4 |
| 5 | 5 | 1 | 5 |
Note that alternative column names include 'question' and 'item' for 'Q', 'pretest' and 'pre-test' for 'Exam1', 'posttest' and 'post-test' for 'Exam2', and 'answers' and 'p' for 'Options'.
The 'Options' column supports non-whole numbers. This might be used if the probability of guessing has been determined empirically (e.g., 3PL with a nationally-normed exam). If the value in 'Options' is less than one, it will be converted to one over the value specified.
# Matched Question Analysis
The matched question analysis provides an estimate of the amount of learning when adjusted for guessing.
Matched question analysis uses data from students who took both the pretest and the posttest to calculate learning values grouped by question; students who took only one of the two exams are removed from the analysis. This is the most important analysis file for assessment and pedagogical improvement as it provides estimates of the percent of students who *learned* a given question. This data can be compared over time as instructors make changes to their course or program.
If you are new to this analysis, focus your attention on $$\hat \gamma$$ (gamma) and $$\hat \gamma/(1-\hat\mu)$$ (gamma gain). In simple terms, gamma is the proportion of students who learned the material (as opposed to correctly answered the question). Higher is better but comparing different questions can be problematic as they can be at different levels of difficulty. The gamma gain ( $$\hat \gamma/(1-\hat\mu)$$) estimate is the proportion of students who learned the material that didn't already know the material. In addition to these measures of learning, 'raw' learning values are included in the output file; if you are new to this analysis these can be ignored.
Formally, $$\hat \gamma$$ (gamma), $$\hat \alpha$$ (alpha), and $$\hat \mu$$ (mu) correspond to 'corrected' measurements of the learning types when factoring in the number of students guessing; these adjustments assume that the probability of correctly guessing can be estimated, which is more reasonable in higher-stakes testing environments. $$\hat \gamma$$ is corrected positive learning, $$\hat \alpha$$ is corrected negative learning, $$\hat \mu$$ is corrected pretest stock knowledge (corrected retained plus corrected negative learning), and flow is the corrected pretest/posttest delta ($$\hat \gamma-\hat\alpha$$). The following equations are used to find the corrected values:
$$
\begin{aligned}
\hat \mu &= \frac{\hat {\text{nl}}+\hat {\text{rl}}-1}{n-1}+\hat {\text{nl}}+\hat {\text{rl}} \\
\hat \gamma &= \frac{n (\hat {\text{nl}}+\hat {\text{pl}} n+\hat {\text{rl}}-1)}{(n-1)^2} \\
\hat \alpha &= \frac{n (\hat {\text{nl}} n+\hat {\text{pl}}+\hat {\text{rl}}-1)}{(n-1)^2}
\end{aligned}
$$
where $$\hat{\text{pl}}$$ (positive learning), $$\hat{\text{rl}}$$ (retained learning), and $$\hat{\text{nl}}$$ (negative learning) refer to the raw learning type values and $$n$$ is the number of answer options. It is important to use these corrected values as the raw scores can be sensitive to the percent of the class guessing. [Smith and Wagner 2018](https://doi.org/10.1080/00220485.2018.1500959) details this adjustment.
$$
R = \frac{\hat {\text{nl}}+\hat{\text{pl}}+\hat{\text{rl}}-1}{2 \hat{\text{pl}}+(\hat{\text{nl}}+\hat{\text{rl}}-1) (1/n+1)}
$$
Gamma gain ( $$\hat \gamma/(1-\hat\mu)$$ ) and $$R$$ (the column R) were introduced by [Smith and White 2021](https://doi.org/10.1177/01466216211013905). $$R$$ compares the sensitivity of the gamma and gamma gain estimators to probability misspecification. An $$R$$ value between -1 and 1 indicates the gamma gain estimator is less sensitive to probability misspecification. An $$R$$ value greater than 1 or less than -1 indicates the gamma estimator is less sensitive. The column RMinSensitivity present the less sensitive estimator based on the value in the column R.
Columns ending in 'Zero' indicate that the probability of guessing is determined by assuming that true negative learning is zero instead of using the supplied value. With these columns, $$\hat \alpha$$ is assumed to be zero in the equation above. This assumption allows the system to solve for the implied probability of correctly guessing. This implied probability is then used to calculate $$\hat \gamma$$ (column GammaZero), $$\hat \gamma / (1-\hat \mu)$$ (column GammaGainZero), and $$R$$ (column RZero). These columns are useful when the probability of correctly guessing could be substantially incorrect. This includes situations like low-stakes exams and incentives that manipulate the propensity of a student to guess.
If "Include Summary Rows" under the "Options" menu is checked then means (averages), standard deviations and observation counts will be provided in the analysis file.
### Comparing to Critical Values
You can lookup critical values of both gamma ( $$\hat \gamma$$ ) and gamma gain ( $$\hat \gamma/(1-\hat\mu)$$ ) under the help menu.
These critical values are generated through simulations where there was no learning. Thus the true gamma ( $$\gamma$$ ) and gamma gain ( $$\gamma/(1-\mu)$$ ) values are zero in these simulations. However, a distribution of outcomes is produced from randomness (due to guessing). If your observed gamma ( $$\hat \gamma$$ ) or gamma gain ( $$\hat \gamma/(1-\hat\mu)$$ ) is greater than what is displayed in the table, you can say that the value is greater than what you would expect from randomness alone. The tables are available for 3-6 question options and classes from 15 to 300 students.
# Matched Student Analysis
The matched student analysis provides an estimate of the amount of learning when adjusted for guessing grouped at the student level.
Matched student analysis uses data from students who took both the pretest and the posttest to calculate learning values grouped by student; students who took only one of the two exams are removed from the analysis. These files are more often used for research than assessment or pedagogical improvement as usually some subset of the students were exposed to a treatment while others were not.
The matched student analysis has identical output columns to [matched question analysis](/output-files/matched-question-analysis). Because this report is grouped by student,$$\hat \gamma$$ (gamma) represents the proportion of the material learned by student *i* and $$\hat \gamma / (1-\hat \mu)$$ (gamma gain) represents the proportion of the material learned by student *i* as a percentage of the material that they did not know when taking the pretest.
Matched Student Analysis
There are two versions of this report, one grouped at the student level and one grouped at the student/options level. The later produces multiple rows for each student grouped by the number of answer options. For instance, suppose there is a 10 question multiple choice paired pretest posttest where half of the questions have four options and the other half have five. This file would produce two rows per student, one for the four option questions and one for the five option questions.
If "Include Summary Rows" under the "Options" menu is checked then means (averages), standard deviations and observation counts will be provided in the analysis file.
# Unmatched Results
Unmatched results are available from the menu bar under analyze. We recommend against these results as they aren't true measures of learning.
The software produces two unmatched results: exam and student. In both cases, the file does not require matched pairs: a student or question could be included in one exam but not the other. Exam1 refers to the pretest and Exam2 refers to the posttest.
{% tabs %}
{% tab title="Exam Results" %}
Exam results
{% endtab %}
{% tab title="Student Results" %}
Student Results
{% endtab %}
{% endtabs %}
# Privacy and Security
Due to FERPA, privacy and security are critical when handling identifiable grade information. Assessment Disaggregation uses a number of technologies to keep your data safe.
Assessment Disaggregation is designed to minimize the necessary permissions to perform the calculations and produce the output files. There are a few key concepts this section covers.
1. Code signing and notarization
2. Limited permission install
3. In-memory file generation
4. Auto updating system
On both Windows and macOS, the application is [code signed](https://en.wikipedia.org/wiki/Code_signing). This prevents the application from running if it has been altered by a third party. The macOS version takes this a step further with two technologies: the [hardened runtime](https://developer.apple.com/documentation/security/hardened_runtime) and [application notarization](https://developer.apple.com/documentation/xcode/notarizing_macos_software_before_distribution). The hardened runtime provides limited access to the system through a system of entitlements. In the case of this application, the system only grants access to files that the user explicitly selected through the system open/save dialog box; the application does not have access to other files on your system. Notarization verifies the application code signing and scans the application for malicious components.
Because the application is designed to run with limited permissions, it can be installed on both Windows and macOS by non-admin users. Standard Windows users can simply download the application and double click on the installer. This will install the application in their own user space (no system-level privileges needed). Similarly, macOS users need not have admin access. Once the application is downloaded, the user can move the application to any area of the file system that they have write access.
The application is also designed to never write temporary files to the system. When a file is selected, the relevant data is loaded into an in-memory database. All calculations are performed using this database. Because the database is in-memory, it no longer exists once the application is closed. Finally, because it is always possible there is a bug in the software, the application automatically updates itself in the background (similar to Google Chrome).
#### What is up with the Microsoft Defender SmartScreen Warning?
On some Microsoft Windows systems, you might receive the following warning from Microsoft Defender SmartScreen. The below screenshots are from before (left) and after(right) you click the more info button.
Defender SmartScreen Warning before and after clicking the more info
This warning appears when an application hasn't established enough of a 'reputation' with Microsoft through many downloads (believed to be about 3000) of the application (some of this is described in this [Microsoft document](https://docs.microsoft.com/en-us/archive/blogs/vsnetsetup/windows-smartscreen-prevented-an-unrecognized-app-from-running-running-this-app-might-put-your-pc-at-risk)). As this is a niche application, it is unlikely to ever establish this reputation through downloads. Alternatively, an application can gain immediate reputation by obtaining an "Extended Validation" code signing certificate; unfortunately, these certificates are only available to organizations, not individuals. Nonetheless, the dialog box can be bypassed by clicking 'more info' and 'run anyway.'