SAT reading score analysis

Development of an open-source software to analyze eye-tracking data and detect correlations between students’ eye movements and their performance on various standardized tests

Tokyo Techies Student Research Project:  

Open source software for analysis and correlation of reading patterns with superior SAT scores using  eye gaze tracking device [Completed]

by a Tokyo Techies student

Research Adviser: Phong Nguyen


Abstract

An open-source application using a gaze tracking device for investigating a correlation between reading patterns and superior Scholastic Aptitude Test (SAT) reading scores is presented. An eye gaze tracking device tracks where the reader is looking across the screen and provides the coordinates of the gaze. The collected data enables us to determine reading patterns and times spent looking at specific sections of the screen and analyze them. The statistical analysis shows that the ratio of time spent on reading the passage and time spent on reading the questions has the highest correlation with number of correct answers. The software is released under the GNU General Public License and is freely available for all educators.

Keywords

gaze-tracking device; eye gaze; reading patterns; superior Scholastic Aptitude Test (SAT) ; 

Acknowledgement

We would like to thank the Tokyo Academics staff, and Students who assisted in this project.


  1. Introduction
    The SAT is a standardized test that many American students choose to take in order to enroll in University. Students want to know what methods are the most effective in order to achieve their highest potential score. Previous analysis of the SAT’s reading section has been through post-test surveys, which are highly subjective and depend completely on the students’ biased answers [1]. A more objective analysis of SAT reading sections should be conducted in order to provide students with the most reliable techniques for reading various passages.Eye gaze tracking technology has been developed recently and made readily available to average consumer.  The device tracks the eye gaze with a camera and high-resolution infrared sensor. The data collection method is very simple, nonintrusive, and objective. By using this non-intrusive method we were able to collect data but still have the test-taking scenario as accurate as possible. Fig. 1 shows the demonstration of the technology.Our method uses an eye gaze tracking device in order to collect and store data on reading patterns when students are taking the SAT reading test. Unbiased data collected from the eye-tracking device allows us to analyze and find different reading patterns. Correlation analysis between reading patterns and raw scores can uncover the most effective and efficient way to read a SAT reading passage. Our findings reveal that some reading patterns (e.g. ratio of time spent on reading questions and the ratio of time spent on reading the passage) have a significant impact on SAT scores while other patterns (e.g. total amount of switches between the passage and the questions) have little to no impact on performance for the SAT reading section. Our application is open-source and is available to all educators who are interested in a more objective SAT analysis. The source code is available on Github: https://github.com/19howea/EyeTribeDataCollection.git.
    Fig 1. Eye gaze tracking technology [3]
    [dt_gap height=”10″ /]
  2. Related Work
    Traditionally the way College Board would collect data about the test is through biased and qualitative post test surveys. This has created a very unreliable and hard to analyze data set. Prior to our research there was no unbiased, nonintrusive and statistical method to evaluate reading patterns on a SAT. Our method aims to give students a more comprehensive and detailed way to better improve their test scores. Moreover, our open-source code gives teachers around the world with varying backgrounds a chance to analyze his or her students relative to their experience.[dt_gap height=”10″ /]
  3. Approach and methodology
    For our research we used an EyeTribe eye-tracking device in order to detect gaze patterns from the test subjects. The biggest screen an EyeTribe eye-tracker can accurately work on is a 24inch monitor. Our code is optimized for a 24inch monitor because with the biggest screen the points of where the subject is looking on the screen can be as clear as possible.Data Collection Method To collect our data we use Khan Academy’s practice SAT reading passages as the environment [4], where the passage is on the left side and the questions are on the right side. Students are required to take the test online while the Eye-Tribe device is tracking their eyes. The coordinates of where the students are looking are stored in a CSV file for us to analyze. With the 24inch monitor, the coordinate goes from (0,0) to (600,1024). The Eye-Tribe device is capable of recording 30 coordinates in a second.Figure two shows the visualization of a test takers eye gaze on the Khan Academy interface.
    Fig 2. Heat map of eye gazes on Khan Academy’s SAT passage test interfaceIn addition, we take note of number of correct answers, incorrect answers, unanswered questions, and other various features of the test subjects like ethnicity, gender, age, etc. Table 1. shows an example of our ground truth data which will be used for our statistical models.
    Table 1. Example of ground truth data
     Analysis method From the collected data, we extract 12 reading pattern features in order to then compare them with the number of questions answered correctly. The list of features and their explanation is presented in Table 2.Table 2. List of features and their explanation

    No. Feature Explanation
    1 Total Time Total time to complete one passage
    2 Percentage of total time looking at the passage Percentage of total time looking specifically at the passage
    3 Percentage of total time looking at the questions Percentage of total time looking specifically at the questions
    4 Total amount of times the subject switches from looking at the passage to looking at the questions or vice versa Total amount of times the test taker switches from looking at the passage half to the questions half or vice versa
    5 Time spent solely on reading the passage at the beginning Amount of time spent solely on reading the passage at the beginning of the test
    6 Amount of ten second intervals spent solely on reading the passage Amount of ten second intervals which have 80% or more of the ten seconds spent on reading the passage
    7 Amount of ten second intervals spent solely on reading the questions Amount of ten second intervals which have 80% or more of the ten seconds spent on questions the passage
    8 Percentage of first four minutes spent only on reading the passage Percentage of first four minutes spent only on reading the passage
    9 Percentage of first four minutes spent only on reading the question Percentage of first four minutes spent only on reading the question
    10 Percentage of last four minutes spent only on reading the passage Percentage of last four minutes spent only on reading the passage
    11 Percentage of last four minutes spent only on reading the questions Percentage of last four minutes spent only on reading the questions
    12 Speed of reading the passage The speed at which the test taker reads the passage

     

    After extracting the 12 features, we then calculate the correlation of each above feature with the number of correct answer in order to find which features have the biggest impact. The correlation coefficient is ranges from -1 to 1. The closer the correlation coefficient gets to 1 or -1 the stronger the correlation between said feature and the number of correct answers is. We want to find which features have the most impact to see how we can maximize potential SAT reading passage scores. The correlation coefficient between features and number of correct answers is calculated by Pearson correlation formula [2].

    (1)

     

    • r is the correlation coefficient
    • n is total number of data
    • x is the feature vector, x̅ is the average of x
    • y is the correct answers vector, ȳ is the average of y
    • Sx is the summation of x
    • Sy is the summation of y

     

    EXPIREMENT SETINGS

    We chose high school students ranging from 14 years old to 18 years old as our experimenters because they are the main targets of this analysis. They were asked to read a passage and answer the questions with no restrictions. After we collected out sample set we decided to have each participant take the test two different ways: reading the passage then the questions and reading the questions then the passage.

    We have collected data from 33 students coming from 10 different countries. In total, we have collected over 1000 minutes raw reading pattern data. Their performances were also recorded as we have explained above and in the example in Table 1.

     

  4. Results Overview
    We found that if a student spends more time reading questions then reading the passage, the performance was better (Fig. 3); however, if a student switch a lot between reading passage and reading questions, it does not improve his performance. Currently, a lot of data is skewed because of a small data set but we are working on collecting more to have the most accurate
    results for what features matter most.

    Fig 3. Graphs of correlation against the 12 featuresThe correlation coefficients of all features and the correct answers are listed in Table 3.Table 3. Correlation coefficient between a reading pattern feature and number of correct answer

    No. Feature Correlation Coefficient
    1 Total Time 0.3798
    2 Percentage of total time looking at the passage 0.0008
    3 Percentage of total time looking at the questions -0.0008
    4 Total amount of times the subject switches from looking at the passage to looking at the questions or vice versa -0.3327
    5 Time spent solely on reading the passage at the beginning 0.3978
    6 Amount of ten second intervals spent solely on reading the passage 0.2381
    7 Amount of ten second intervals spent solely on reading the questions 0.1517
    8 Percentage of first four minutes spent only on reading the passage 0.2557
    9 Percentage of first four minutes spent only on reading the question -0. 2557
    10 Percentage of last four minutes spent only on reading the passage -0.3015
    11 Percentage of last four minutes spent only on reading the questions 0.3015
    12 Speed of reading the passage -0.3302

    [dt_gap height=”10″ /]

  5. Discussion and conclusions
    The purpose of the research was to provide a more quantifiable data set for reading standardized tests, specifically the SAT, and use this data to then help improve test takers efficacies to improve scores. The benefit of our approach is that it is easy, quantifiable and can be used on type of student. Our program is an open source program that is available to all who are interested in conducting objective SAT reading data. From our research we have found that the switch count and speed of reading have the lowest negative correlation and the total time and time spent reading the passage at the beginning have the highest positive correlation. Meaning that reading the passage first and the questions while minimizing switches and maximizing time given is the most effective way to read a SAT passage to maximize your score. To better improve our research we want to collect more data from a more diverse population to verify our analysis more objectively and use a statistical test for those with high score and those with low score in order to show strong evidence on the impact of SAT reading patterns.

References

[1] “New SAT Survey Press Release.” New SAT Survey Press Release, Public Relations The Princeton Review, The Princeton Review, Princeton Review

[2] Daren S. Starnes, Josh Tabor, Dan Yates, David S. Moore, “The Practice of Statistics”, pp. 154.

[3] Roberts, Hannah. “Facebook’s Oculus Purchased a Company That Specializes in Eye-Tracking Technology.”       Business Insider, Business Insider,   http://www.businessinsider.com/oculus-acquires-danish-startup-the-eye-tribe-2016-12

[4] Khan Academy, https://www.khanacademy.org/mission/sat/exams

 

 

 

© 2020 Tokyo Techies. All rights reserved.