Tokyo Techies Student Research Project:
by a group of Tokyo Techies student
Research Adviser: Phong Nguyen – Director of data science & AI technologies.
We propose to use a facial recognition technology in a smart attendance application in order to make the attendance taking process more efficient. The conventional method, such as manual roll call, is inefficient and time consuming. However, with the application of facial recognition technology, the attendance taking process becomes automated and unobtrusive, thereby making it a smarter alternative to traditional methods. In this paper, we introduce an open source attendance taking program using OpenFace for the facial recognition module. We use a machine learning approach to build a classification model to identify a person’s name from the unique facial features of an individual subject. In an experiment, we have verified the accuracy of the facial recognition module at 96.2% given our test using 50 subjects. The result is tolerable for a school/organization with a small number of users. This could be implemented within a classroom to assist administration through automating roll call.
smart attendance application ; facial recognition technology ; automating roll call
We would like to thank the Tokyo Academics staff, and Students who assisted in this project.
Attendance is a tedious but necessary part of lives, from calling roll at school or clocking in at work. The main problem with the process of taking attendance is that although it is cost-effective, it is extremely inefficient with time. While other methods, such as key cards and fingerprints, allow for more time-efficient methods of clocking in, fingerprinting machines are expensive and key cards can easily get lost or misplaced. This creates the need for an alternative to traditional methods of attendance taking that will not get lost and is inexpensive. Facial recognition technology has been developed rapidly for the past few years. The computing power of machine, as well as the advance in the algorithm of artificial neural network, has made facial recognition a reliable technology which can be used as one of the alternative solutions for an attendance taking system. By using facial recognition we created a new application for cost-effective, efficient attendance taking that will serve as an upgrade to conventional methods of attendance taking.In this paper, we introduce the way to utilize facial recognition technology in a smart attendance application. Our open-source code is available at https://github.com/xphongvn/smart-attendance-system-ta, for other developers to reuse and improve for their own use. The rest of the paper is organized as follows: First, we give an overview of the related work to facial recognition technology and smart attendance system. Second, we present our method to apply facial recognition module in the whole system. Third, we demonstrate how we set up experiments to verify the accuracy of the facial recognition module. And we give the results and discussion about the results. Finally, we give our conclusions on the system.
- Related Work
There are other facial recognition softwares, such as Facefirst  and OpenFace . Facefirst has facial recognition software. However, the focus is mainly on surveillance and picture capturing, which is an application of facial recognition but not our desired effect. It does not have possible attendance taking capabilities . There is also programs such as OpenFace. OpenFace can detect a face and into feature representations into usable data . Therefore, it won’t classify the name and use this information in an attendance system. By leveraging the open source code from OpenFace were able to constructed a smart attendance application on that source code that can utilize the resources from OpenFace and be a facial recognition program that successfully takes attendance. Another work has used Bluetooth Low Emissions (BLE) as a smartphone system for checking attendance. By implementing this within a classroom the problems arise with having obstacles within the class that would dampen the signal of the BLE emitter, making potential reading inaccurate. Also, this system requires the student to own a bluetooth compatible smartphone, which is expensive. 
- Approach and methodology
a) Data Collection
In order to prepare the training data, a sample of pictures of various subject’s faces should be collected. These sample pictures were taken by the user on a laptop computer webcamera before the experiment began. These sample pictures are then put into OpenFace API and are stored in folders under the name of each subject. These pictures are used for building a SVM classification model, with names of each person as the label used to classify them. b) Building the Facial Recognition ModelWe built a classification model using a machine learning framework. The goal with this model was to reach an accuracy of 97.53%, based upon the number given for human level accuracy in by Cornell University on Surpassing Human-Level Facial recognition capabilities . From raw image data of a person’s face, we convert it into feature vector using neural network provided by OpenFace. Using the feature vectors, a Support Vector Machine (SVM) classification model was built to identify a person’s name . The flow is described in Figure 1. By using the implementation portion of our program, the system will take in all inputs within the camera’s frame and try to identify a face. If the correct range is met for the two inner eye landmarks and the bottom of the lip landmark, a face is identified within the camera. Then our system will match all other landmarks to the face and compare that to the data previously stored (the sample of pictures taken earlier). Our system will then make 128 strings and compare those to the previously stored data and the percent of the matches between the new set of 128 strings and the old set is returned as the percent confidence and a name.
Fig. 1: Flowchart of training and implementation of Smart Attendance Applicationc) ApplicationTo know whether a person enters a building or leaves a building, we use 2 cameras: one facing the direction to enter, and the other one facing the direction to leave (Figure 2). The system will identify the person entering or exiting, and will then send their name to a processing program to mark their attendance. This system will be used by students to be identified for attendance, and management staff to monitor attendance. Also once a student is identified, the time of arrival and departure are recorded as well as their attendance marked (Figure 3). Simultaneously, an email will be sent to the student’s parents to notify them if the student is or isn’t present and the time of arrival and departure.
Fig. 2: Setup of smart attendance system, with cameras identifying people entering and exiting an area
Fig. 3: User interface of the Smart Attendance System d) EXPERIMENTAL SETUPThe experiment took place at Tokyo Academics, a tutoring school in Japan, and spanned over the course of 2 weeks (2 weekends). This experiment was to verify the accuracy of the facial recognition module. We settled on an experiment that included a total of 50 subjects. Before the start of the two weeks, each subject had 10 photos taken of them with the Facial Data Collection program, thus making profiles for each user which contained data on multiple characteristics of the user’s face. The time to take 10 photos of the experimenters was not too long, therefore it was acceptable in the experiments. We believed the more training photos we had, the better the models can be built.We used these collected photos and their names to build a SVM classification model. And the same SVM classification model was tested for accuracy. Over the course of the 2 weeks, each subject walked through the camera 10 times. We kept the lighting, camera level, and general walking speed of the test subjects as constants in order to truly test the accuracy of our system under fair conditions.We setup the facial recognition module so that it shows the name of the person output by the system on top of their images. Fig 4. shows an example of how the module shows the image and the names on top of the image. We count the number of successful recognition (as the name of the person and the name as the system output is the same) as the main metric to verify the accuracy of the facial recognition module. The accuracy of the facial recognition module determine of reliability of our Smart Attendance System.
Fig. 4:Facial Recognition Module outputs a person name based on new input image
- Results Overview
Of the 500 trials done 481 were accurately identified while 19 were incorrectly identified. This leaves us with approximately a 96.2% accuracy and a 3.8% error. Meaning that about 4 of every 100 students that come into TA will be incorrectly marked in one way or another. In the case of TA, this is a tolerable margin of error considering how few people this will affect. Extrapolating these results to a given company of or above 100,000 employees, there error appears to be at least 3,800 incorrectly marked people, also our SVM classifier can only work well with a small number of labels, or corporations of this scale, these two may potentially cause big problems for said corporations with large numbers of employees.
- Discussion and conclusions
We have achieved an accuracy of 96.2% with our facial recognition module in our smart attendance system. However our experiment was not conducted on a larger scale because our SVM classifier can only work well with a small number of labels. Our future work is to improve the number of labels and improve the accuracy of the facial recognition module to be practical on a larger scale. This means that for this to be effective for real world applications, it would have to be used in a relatively small environment, examples including a small middle or high school, where the number of people needed to be identified is not as large, and where possible errors could easily be corrected by teachers. To employ this system in larger environment improvements would need to be made. These improvements could include the program generating a confidence percentage indicating how confident it is of its guess. This could be used to make it so that if the confidence is too low, the program would not return a name, but would instead alert of a stranger. This is as currently if an entirely new person were introduced to the program, it would identify it and give it a name, even if it can barely match the face, which could hurt our accuracy ratings. Other optimizations would be possibly changing the type of classifier used as because as was previously mentioned, our SVM classifier can only work with a small number of labels, limiting its ability to work on an even larger scale. In addition, our percentage of accuracy was slightly less that what our goal was, as we were trying for an accuracy of 97.53% and ended up with 96.2%. However, even if our model is slightly less accurate than human level, it is still very effective and can be more efficient than conventional methods of attendance. Finally, with improvements, such as returning a confidence number, or changing what we used for development of the model and the recognition, we feel that we could surpass our goal of 97.53% accuracy.
 Face Recognition Software: Best-in-Class Enterprise Facial Recognition Security Platform. (n.d.). Retrieved September 20, 2017, from https://www.facefirst.com/
 Amos, B., Ludwiczuk, B., & Satyanarayanan, M. (2016). Openface: A general-purpose face recognition library with mobile applications. CMU School of Computer Science.
Steinwart, Ingo. Support Vector Machines. S.l.: Springer., n.d. Print.
 Lu, C., & Tang, X. (2015, January). Surpassing Human-Level Face Verification Performance on LFW with GaussianFace. In AAAI (pp. 3811-3819).
Chun, S., Kwak, M., Park, M., & Kwon, T. (2017, September 19). An Automatic Attendance Checking System using Smartphones: An Infrastructureless Approach. Retrieved September 20, 2017.