Brain-Computer Interaction using functional Near-Infrared Spectroscopy (fNIRS)

Past Project
/

# Introduction

This project brings together three research areas in a combination that is now poised to advance the field of human-computer interaction. We combine new technology for brain activity measurement using functional near infrared spectroscopy (fNIRs), the use of machine learning for analyzing user data in human-computer interaction (HCI), and experience in designing, implementing, and evaluating non-command, adaptive user interfaces from our work on eye movement-based interaction.

# Abstract

In particular, fNIRs is still a research modality. It has rarely been used in combination with either machine learning or HCI, and, to our knowledge, the use of fNIRs as a real-time input to an adaptable interface breaks new ground. We believe this combination will also lead to new, more objective methods for evaluating next generation interaction styles. We will bring our concept of reality-based interaction, described below, to this study. It focuses on the ways that new interaction styles exploit the user's pre-existing skills and expectations from the real world more than trained computer skills. It also helps us differentiate mental effort devoted to interface-related or syntactic aspects from that devoted to the underlying task or semantic aspects. Bringing these three fields together will open up a new area of HCI research. We hope to advance the theory and evaluation of interaction styles as well as the development of new types of interactive interfaces by using human brain activity as 1) a more objective measure for evaluating emerging interaction styles and 2) an input to adaptable user interfaces.

The expertise and collaboration needed to connect fNIRs, machine learning, and HCI is difficult to obtain in a single research team. We are well positioned at Tufts, with strong researchers in all three fields in the same or neighboring departments within the Engineering School. This proposal begins with background information and a review of literature relevant to our proposed research. Next, we present results from our feasibility study, demonstrating that our technical approach is working and our goals appear viable. Finally, we describe four phases of our proposed research and summarize the intellectual merit and broader impacts of the project.

The feasibility study results presented below show that, using our fNIRs measurements and machine learning algorithms, we were able to classify five different types of user workload in a computer task with average accuracy exceeding 95%.

In Phase 1 of our proposed work, we will refine our process and algorithms to give us a reliable and repeatable procedure for real time user workload measurement. We will also extend our work to continuously varying workload levels and address the removal of motion artifacts from the data.

In Phase 2, we will combine EEG measurements with fNIRs, to use their complementary properties to obtain a fuller picture of the user's brain activity.

Phase 3 applies this technology as a tool to evaluate user interfaces. In particular, we will focus on next generation, non-WIMP, reality-based user interfaces, which are less amenable to traditional evaluation techniques. The need for new techniques to evaluate these interfaces is a growing problem within the field of HCI. The goal of new interfaces might differ from traditional ones. For example, a next generation interface might have an enhanced entertainment value, yet most evaluation tools will measure performance as error rate or time spent on a task. Metrics such as user frustration, workload, and enjoyment have a limited use in evaluation studies because of their subjective nature. We will explore using fNIRs data as an objective measurement of these properties and apply them to the evaluation of non-WIMP interfaces.

Phase 4 focuses on creating new interactive, real-time user interfaces, which can adapt their behavior based on the brain measurement information we obtain. The design challenge will be to use this information in a subtle and judicious way, as an additional, lightweight input that could make a mouse or keyboard-driven interface more intuitive or efficient. We will draw on our prior work in designing eye movement-based interaction techniques, because of the strong parallels between the two HCI design problems.

If successful, our work will pave the way for a new and useful combination of fNIRs, HCI, and machine learning. We will use it to produce better ways to study, characterize, and measure user interfaces, and we will classify them within the theoretical framework of reality-based interaction. We will then produce new types of interfaces that can adapt to the user's workload profile or other brain activity in real time, and we will also develop technical improvements in applying fNIRs to realistic HCI settings and combining it with other sensors.

# News Articles

# Papers

# Journal Articles

  • Discrimination of Mental Workload Levels in Human Subjects with Functional Near-Infrared Spectroscopy, A. Sassaroli, F. Zheng, L.M. Hirshfield, A. Girouard, E.T. Solovey, R.J.K. Jacob, and S. Fantini, Journal of Innovative Optical Health Sciences, vol. 1, no. 2, pp. 227-237 (2008).

# Refereed Conference Proceedings

  • Distinguishing Difficulty Levels with Non-invasive Brain Activity Measurements, Girouard, A., Solovey, E.T., Hirshfield, L.M., Chauncey, K., Sassaroli, A., Fantini, S. and Jacob, R.J.K., in INTERACT 2009, (Uppsala, Sweden, 2009), Springer.

  • Adaptive Brain Computer Interfaces, Girouard, A., in CHI '09 extended abstracts on Human factors in computing systems, (2009). [link](file:///Users/ziyuzhou/Documents/study/codebase/github_page/hci_lab_website/materials/public_html/papers/dc.girouard.chi09.pdf)

  • Brain Measurement for Usability Testing and Adaptive Interfaces: An Example of Uncovering Syntactic Workload with Functional Near Infrared Spectroscopy,, L.M. Hirshfield, E.T. Solovey, A. Girouard, J. Kebinger, R.J.K. Jacob, A. Sassaroli, and S. Fantini, Proc. ACM CHI 2009 Human Factors in Computing Systems Conference, ACM Press (2009). link (opens new window)

  • Combining Electroencephalograph and Near Infrared Spectroscopy to Explore Users' Instantaneous and Continuous Mental Workload States, L. Hirshfield, K. Chauncey, E. Solovey, A. Girouard, R. Jacob, A. Sassaroli, and S. Fantini, HCI International 2009 13th International Conference on Human-Computer Interaction, Springer (2009).

# Other Published Proceedings

  • Human-Computer Interaction and Brain Measurement Using Functional Near-Infrared Spectroscopy, L.M. Hirshfield, A. Girouard, E.T. Solovey, R.J.K. Jacob, A. Sassaroli, Y. Tong, and S. Fantini, ACM UIST 2007 Symposium on User Interface Software and Technology, ACM Press, Poster paper (2007). link (opens new window)

# Other Publications

  • Programming Reality within the Reality-Based Interaction Framework, E.T. Solovey, O. Shaer, A. Girouard, L.M. Hirshfield, M.S. Horn, J. Zigelbaum, and R.J.K. Jacob, Proc. ACM CHI 2009 Workshop on Programming Reality: From Transitive Materials to Organic User Interfaces (2009). link (opens new window)

  • Using Brain Measurement to Evaluate Reality Based Interactions, L.M. Hirshfield, E.T. Solovey, A. Girouard, R.J.K. Jacob, J. Kebinger, M.S. Horn, O. Shaer, J. Zigelbaum, and R.J.K. Jacob, Proc. ACM CHI 2009 Workshop on Challenges in Evaluating Usability and User Experience of Reality-Based Interaction (2009). link (opens new window)

  • Real-time assessment of mental workload with near infrared spectroscopy: potential for human-computer interaction， A. Sasaroli, Y. Tong, L.M. Hirshfield, A. Girouard, E.T. Solovey, R.J.K. Jacob, and S. Fantini, BIOMED OSA Topical Meeting, Poster presentation (2008). link (opens new window)

  • Using functional Near-Infrared Spectroscopy in HCI: Toward evaluation methods and adaptive interfaces， A. Girouard, L.M. Hirshfield, E. Solovey, and R.J.K. Jacob, Proc. ACM CHI 2008 Workshop on Brain-Computer Interfaces for HCI and Games (2008). link (opens new window)

# Technical Reports

  • Distinguishing Difficulty Levels with Non-invasive Brain Activity Measurements, A. Girouard, E.T. Solovey, L.M. Hirshfield, K. Chauncey, A. Sassaroli, S. Fantini, and R.J.K. Jacob, Technical Report 2008-3, Department of Computer Science, Tufts University, Medford, Mass. (2008). link (opens new window)

# Acknowledgments

We thank Claire Lee, Matthew Knowles, James Kebinger, Kristof Redei, Kelly Moran, Hadar Rosenhand, and the rest of our colleagues in the HCI research group at Tufts; Dr. Roni Khardon, Dr. Carla Brodley, Dr. Eric Miller, Rachel Lomasky, Umaa Rebbapragada, Amelio Vazquez-Reina, and D. Sculley from the computer science department at Tufts; Krysta Chauncy and Martin Paczynski from the Psychology Department at Tufts; and Desney Tan at Microsoft Research for his helpful inputs and encouragement.

We thank the National Science Foundation for support of this research (NSF Grant Nos. IIS-0713506 and IIS-0414389). Any opinions, findings, and conclusions or recommendations expressed in this article are those of the authors and do not necessarily reflect the views of the National Science Foundation. We also thank the Natural Sciences and Engineering Research Council of Canada for financial support.

# Courses

COMP250 Brain-Computer Interaction (opens new window)

Taming fNIRS-based BCI Input for Better Calibration and Broader Use

Taming fNIRS-based BCI Input for Better Calibration and Broader Use
Tangible Programming Languages A practical approach to computer programming in educational settings.

Tangible Programming Languages A practical approach to computer programming in educational settings.