Final Results and Handoff
Table of Contents
It is probably most convenient to use Nvidia’s Tegra Android Development Pack for development. This is simply a variant of Eclipse with all dependencies included: Android stuff, NDK, OpenCV, etc. It can be found here: https://developer.nvidia.com/content/tegra-android-development-pack-20-available
Code Download: Zip File
This zip file contains our gaze tracker eclipse project. You will also need the opencv sdk eclipse project as a dependency. (Download at: http://opencv.org/downloads.html)
The original Gaze library was created by Flurin Rindisbacher and Chris Feuz. It can be found on github here: https://github.com/Krigu/Gaze
This could be useful if you want to see how things were accomplished in the original library before we hacked away at it to make it work with Android / JNI and to only use two corneal reflections (among many other things). Furthermore, they provide a NetBeans project file that you should just be able to import into NetBeans and start playing with. This can be useful for doing some initial familiarization with the algorithms without the added annoyance of JNI. It’s much easier to acquire debug images when you can just call OpenCV’s imshow() on your PC. In our code, getting debug images is quite difficult due to the requirement for cross-language mutexes.
It could be very useful to make the webcam code more robust. Right now, the system will crash (never start) if, for example, the webcam happens to switch from /dev/video0 to /dev/video1. We were working on a fix but it turned out to be harder than we anticipated and we needed to spend more time on gaze tracking itself.
The webcam is accessed using the Video4Linux2 API. The stock kernel on the nexus 10 doesn't support usb webcams. The kernel modules for the nexus 10 are stored in the /sdcard directory and loaded using the insmod command when the program starts.
We’re using JNI, so if you’ve never used it before, you should familiarize yourself with it at least a little bit before proceeding. This is necessary to be able to use C/C++ code. We have an Android thread that is able to print log messages and general print statements from the C side. This is absolutely critical for debugging; without it, you will have even less visibility into what’s happening on the C side.
There’s some configuration options in GazeConfig that you can play with in order to get the algorithms working better for at least one person while you’re testing. For example, you can change the expected pupil width (within some range) and the thresholds used for finding the glints.
Flurin and Chris had the luxury of having four LED clusters to provide four glints. We only have two glints due to power constraints. To resolve this, we’re pretending that we have four glints in software by reflecting the two real glints downward across a horizontal axis to create two additional, imaginary glints. This method seems to work quite well, and you probably won’t need to change it, but you should be aware of it.
The original starburst algorithm had a remove_glints() helper function that blurred out the glints in the image so it could more easily find the pupil. Interestingly, we found that this call was not needed (actually, it crashed so we removed it) when only two glints were used. It may be worth putting it back in and figuring out why it was crashing - it might make pupil detection slightly more robust. It’s probably much more useful to try to figure out how to accomplish dynamic configuration options (see ideas section) instead to make the overall system more robust.
After starting the app, wait for the webcam LED to turn on. There are 3 modes of operation on the main menu.
The first mode is gaze tracker mode. This mode displays the rear camera image on the screen, and plots the user's gaze point on the screen with a red dot. The red dot will only be plotted if calibration has been completed.
The second mode of operation is debug mode. As soon as you click the debug button, the program will attempt to find your eye, and will not continue until it does. Make sure the webcam is sufficiently close to your eye. After finding your eye the debug activity will show up. Click the update button to take a picture of your face with the webcam. If your eye is found, the pupil and glints will be highlighted.
The third mode of operation is calibration. During this mode you must follow the red dot on the screen with your eye. If it completes successfully, the main menu will show up, and gaze tracking mode can be entered.
If at any point during the program, the webcam LED turns off, the gaze tracking will no longer work. Exit the program by pressing the android app button and swiping. Unplug the webcam and plug it back in, then restart the app. In some cases, rebooting the tablet may be necessary.
Future & Ideas
It would be useful to implement an additional debug screen where the user can modify some config settings on the fly while looking at how the detection algorithms are affected. For example, there could be sliders to change the expected pupil width or the threshold used for finding glints.
Mechanical Handoff Information
Any plastic Mount replacement parts can be purchased online or wherever GoPro parts are sold. The Flat, 3M Adhesive Mount can be purchased in packs of six in case one should fail. Currently, the ABS plastic case is fully operational and does not require any additional modifications. However, if a future team recognizes potential areas of improvement regarding size and weight than modifications can be made to the part in the machine shop. Also, our current design required the IR LED's to be a certain distance apart in order to create two distinguishable glints on the users eye. If a future team can reduce this distance and still yield the same results, a new case should be manufactured and the old one should be recycled or donated to the shop for scrap.
Infrared CircuitThe infrared red circuit currently uses two sets of three LEDs to illuminate the face and create the glints. This number was chosen based on the power constraints placed on the circuit by using USB to power the circuit. This number is not set in stone. If future teams could use less LEDs the power consumed by the circuit would go done and increase battery life. The use of more LEDs is not something that would be recommended if the circuit would still be powered by USB. Adding more LEDs also brings into concern the safety of the intensity of the light be emitted. Testing would need to be done to ensure that it would pose no risk to the user. The LEDs used, Osram SFH-4550, needed to be sanded to diffuse the light emitted. Any new LEDs purchased would most likely need to be diffused as well. There is a spare set of circuit boards that have been completely filled. These can be used for testing or replacement if something fails in the current set being used. There are also two unfilled boards that can be used for the same purpose only more parts would need to be purchased for them. There are two resistors used in parallel for each branch of the circuit. This was done so no resistors would need to be purchased. Instead two resistors that could be obtained at RIT were used to create the proper resistance.
Camera InfoThe camera used is the Logitech C510. It is a 720p camera. It was chosen because it was found that the infrared filter could be easily removed when we were looking for which camera to use in our project. It has wide-angle lens which helped to ensure things stayed in the field of view. It is an "always focused" kind of camera. The focus can only be changed by opening the camera up and manually moving the lens. 35 mm processed film was used as a replacement for the infrared filter. The film was cut to the same dimensions as the infrared filter and was placed over the image sensor. Two layers of film were used to better match the thickness of the infrared filter which aided when refocusing the camera. Pictures of the camera modification process can be seen here: /public/CameraModification.pdf
Camera FocusingThe camera was focused and tested mainly using the ISO 12233 chart. This chart is a good visual indicator of sharpness or deformity of an image. Test shots were done before and after focusing the camera. If the test shots taken as ideally as possible, then the results are worthless or deceptive. You are able to see how sharp an image is by looking at how many of the line pairs per millimeter are clear in the image. The camera should be lines up with the vertical black outline to test the horizontal resolution. Likewise, line up with the horizontal black outline to test the vertical resolution. This camera seemed to have a weird aspect ratio that did not exactly match 4:3. The lines that form a cross on the top right and bottom right of the target were very useful in focusing the camera because of how straightforward the method is. I kept the camera at the proposed test distance of 17 inches (this was chosen as the average distance between someone's eye and the tablet screen). The lens can be turned clockwise or counterclockwise to adjust the focus. Adjust the lens so that you are able to see the distinction in the converging lines. The closer to the center of the cross that can be discerned the better. A problem encountered when taking test images is that the image files were overly compressed during capturing. The image is more sharp when looking at image before capturing than after capturing. Because of this the software that is normally used to analyze the information from these resolution targets was not able to correctly analyze our images. These happened with the images taken with the lossless setting in the Logitech software. In the future if you are able to get higher resolution lossless shots saved from the camera then the resolution targets can probably be used again to get useful some information. Despite not being able to analyze the resolution target images the camera still functioned as needed. Some of the testing images used can be seen at: /public/CameraTestImages.pdf
Camera PlacementCurrently the camera is mounted on top of the tablet. This causes an issue when the user looks down towards the bottom of the tablet. The cause of the problem is that the eyelids partially cover the eyes when this happens. This problem may be fixed by placing the camera at the bottom of the tablet. This is one idea that should be experimented on in future iterations of this project.
General Handoff Information
Results and Advice
Many customer needs were met, but gaze tracking should be more robust
Gaze tracking is a very tough problem, especially on a tablet
Communication is key when working on a team project
Future teams should consist of more Computer Engineers. Also one Mechanical Engineer may be useful if adjustments are needed for the case and mount.
Things Learned• Scheduling Meetings around five people is very difficult. Setting up small meetings between the important people to each task is much easier. The results from the smaller meeting can be reported to the rest of the team via email or at a whole group meeting.
• For a project in Senior Design, Google Docs is a very useful tool when scheduling conflicts arrive. This method allows group members to work on important documents on their own time and other group members can see what each other has done. One good example is the power point presentations. Each team member can add their own slides to the presentation. Another very useful aspect of this is that multiple people can be editing at the same time.
• Division of labor is very important in Senior Design due to the time constraints. It is very hard to get everything done on time if everything is being tackled by the group.
• School provided soldering irons are very bad. Get one of your own if a lot of soldering is required.
• Safety of the user is always a concern and can be a factor in unexpected places. For example, at the beginning of the project illumination using infrared didn't seem a likely place for safety issues. But there came the issue of damaging the users eye with prolonged exposure to high intensity light and safety calculations needed to be done.
• Test early and often. The sooner things are tested the better. There are many individual pieces of the overall product that can be tested before integration. Any failure needs to be found as early as possible to speed up the process.
• Resourcefulness is a good skill to have. Camera film was used as a replacement filter instead of buying an infrared filter. A pillbox was used as the initial casing of the system mount.