The Tufts fNIRS to Mental Workload Dataset

If you have any questions, please reach out to Leon (leonwang_at_cs.tufts.edu).


# Paper Information

Submitted to NeurIPS 2021 Datasets and Benchmarks Track

Zhe Huang*, Liang Wang*, Giles Blaney, Christopher Slaughter, Devon McKeon, Ziyu Zhou, Alex Olwal, Robert Jacob*, Michael C Hughes* “The Tufts fNIRS to Mental Workload Dataset: Toward Brain-Computer Interfaces that Generalize”

*Lead authors ZH & LW contributed equally, as did supervisory authors RJ & MCH

Paper link (opens new window)


# Project Description

Functional near-infrared spectroscopy (fNIRS) promises a non-intrusive way to measure real-time brain activity and build responsive brain-computer interfaces (BCIs). However, in its first decade of research this technology has not yet realized its potential.

  • One common barrier to effective fNIRS-based BCIs is the lack of available data. Previous work typically collects proprietary datasets from only 10-20 subjects.

  • Another barrier to progress is the lack of a standardized evaluation protocol. Without standardized protocols, different papers may not follow the very same experimental design, making results incomparable and preventing scientific progress.

  • The toughest barrier of all to developing an accurate mental workload classifier is the high variation in fNIRS data, which makes generalizing to a new subject or session challenging.


Our contributions are:

  • We release a large open-access dataset of 68 participants. This dataset is the largest known to us by a factor of 2.5. Details are in Section Dataset below.

  • We suggest a standardized evaluation practice for assessing method performance on our dataset under three paradigms of training (clear instructions and code are provided in Section Code below):

    • subject-specific,
    • generic,
    • generic + fine-tuning.

# Dataset

Link to fNIRS2MW dataset (opens new window)


# License

CC-BY-4.0 (opens new window)


# IRB Approval

Procedures to collect data were approved by Tufts institution's IRB (opens new window), and our deidentified dataset was approved for public release (STUDY00000959). Each participant gave informed written consent, and was compensated $20 US.


# COVID-19 Safety Approval

Because collection occurred during the COVID-19 pandemic in early 2021, we also got approval from the Tufts Integrative Safety Committee (ISC). We followed required sanitation and social distancing practices: experimenters used personal protective equipment and disinfected the fNIRS probe for each subject.


# General Description

Totally, our large open-access dataset includes 68 participants. Each subject contributes 21.33 minutes of fNIRS recordings from a controlled experimental setting with corresponding labels of workload intensity.


# Task Description

  • Each participant completed 16 blocks of n-back trails (0→1→2→3→1→2→3→0→2→3→0→1→3→0→1→2).
  • Each block is composed of:
    • 40 trails ( display for 0.5 seconds, hidden for 1.5 seconds).

Details of task sessions is as below: task flowchart


# Task introduction

N-back video introduction:

Before experiment started, the system played a short introductory video, showing an example of a user completing n-back tasks, with voice-over and caption explanations.

Sample scripts for N-back task video are as below:

Today you will be completing a series of n-back tasks. You will be presented with many numbers, one after another. Some of these numbers will be targets. A number is a target if it is identical to the one shown n steps previously.

Here is an example of a 2-back task. In this case, n = 2. The highlighted 8 is a target number, since the number 2 steps back is also 8.

Your job will be to press the left arrow key each time you see one of these target numbers. If the number shown is not a target number, you should press the right arrow key.


Let’s view an example of a 0-back task.

At the beginning of each task, you will hear a few beeps which indicate the start of the task.

The current type of n-back task can be found at the top of the task window. In this example, n = 0. For the purposes of this demo only, the stack of numbers already shown is included beneath the task window.

When n = 0, every number is a target, since every number is identical to the number 0 steps back. That is, every number is the same as itself. For 0-back, the participant presses the left arrow key after each number appears, and never presses the right arrow key.


Now let’s view an example of a 1-back task.

The participant should press the right arrow key for all non-target numbers. When n = 1, a number is only a target if it is identical to the number 1 step back. Once the participant identifies a target number, they should press the left arrow key instead.

For example, this 4 is a target number because the previous number was also 4. The participant should press the left arrow key, and then continue pressing the right arrow key until they see the next target number.

Here is another target number. The number 1-step back was also 2, so the participant should press the left arrow key.

During today’s experiment, the stack of already shown numbers will not be visible to you. It is your job to remember the numbers that were n-steps back so that you are able to identify the correct target numbers.


There will be 16 rounds in total. During each round, you will complete an n-bask task, where n is either 0, 1, 2, or 3.

Before each task, there will be instructions reminding you how to find the target numbers for the current n. During the task, you will be shown 40 numbers. Each time you see a target number, you should press the left arrow key. For all non-target numbers, you should press the right arrow key.

After each task, you will complete a survey. Once you have completed the survey, you will be able to rest for 20 seconds.

The experiment will take approximately 40 minutes.

After the entire experiment is finished, you will complete a short interview about your experience.

Please remain seated and do not talk or adjust the headband while completing the experiment. However, if you do feel uncomfortable at any point, let the operator know and they will stop the experiment.


Generel introduction: Before each task, the system displayed a graphic depicting how to identify targets for the current n, with voice-over.

The graphic and sample scripts for 0-back are as below:

This is a 0 back task. Every number is a target number.


The graphic and sample scripts for 1-back are as below:

This is a 1 back task. A number is a target if it is identical to the previous number.


The graphic and sample scripts for 2-back are as below:

This is a 2 back task. A number is a target if it is identical to the number 2 steps back.


The graphic and sample scripts for 3-back are as below:

This is a 3 back task. A number is a target if it is identical to the number 3 steps back.


# Data Structure

Our released dataset includes (Link to fNIRS2MW dataset (opens new window)):

***************************************
** fNIRS2MW dataset folder structure **
***************************************
|- qualified_subjects_list.pdf            
|- pre-experiment                              //
|- experiment                                  //
| |- log                                       //
| |- task_accuracy                             //
| |- fNIRS_data                                //
| | |- raw_data                                //
| | |- band_pass_filtered                      //
| | | |- whole_data                            //
| | | |- slide_window_data                     //
| | | | |- size_02sec_10ts_stride_03ts
| | | | |- size_05sec_25ts_stride_03ts
| | | | |- size_10sec_50ts_stride_03ts
| | | | |- size_20sec_100ts_stride_03ts
| | | | |- size_30sec_150ts_stride_03ts
| | | | |- size_40sec_200ts_stride_03ts
|- post-experiment                             //
| |- nasa-tlx                                  //
| |- interview                                 //
| |- * (all other folders)  

# Data Format

We introduce and describe the data format of fNIRS data (raw and pre-processed) and supplementary data as below:


# fNIRS Data (opens new window)

68 participants were recruited, aged 18 to 44 years. None of the participants reported neurological, psychiatric, or other brain-related diseases that might affect the result.


# raw data (opens new window)

TODO


# band pass filtered data (opens new window)

After pre-processing (Dual-slope and band pass filter), we have features/columns as below:

  • label: The label for each row.

    • The values shoule be in the set {0, 1, 2, 3}, representing 0-back/1-back/2-back/3-back tasks respectively.
    • It should be the same for all rows in the same chunk.
  • chunk (Only in slide window data): The chunk number for each chunk.

    • It starts at 0 and increases by ​1 for each chunk.
    • It should be the same for all rows in the same chunk.
  • AB_I_O, AB_I_DO: Intensity of oxy & deoxy from detector AB.

  • CD_I_O, CD_I_DO: Intensity of oxy & deoxy from detector CD.

  • AB_PHI_O, AB_PHI_DO: Phase of oxy & deoxy from detector AB.

  • CD_PHI_O, CD_PHI_DO: Phase of oxy & deoxy from detector CD.

pre process

# whole data (opens new window)

Each subject's .csv file includes continuous data of 16 tasks (exclude data during self-evaluation (nasa-tlx) and rest period).

Screenshot of deidentified data sample is as below: whole data


# slide window data (opens new window)

We offer pre processed data in :

  • Window size: 10/25/50/100/150/200 timestamps (2/5/10/20/30/40 seconds)
  • Window stride: 3 timestamps (rough 0.6 second)

Screenshot of deidentified data sample is as below: slide window data


# Supplementary Data

To ensure quality and consistency, we used several criteria to identify which subjects' data are suitable for classifier evaluation.


# demographic and contextual information (opens new window)

Demographic and contextual information is recorded before the experiment.

Please see details in the data directly


# task_accuracy (opens new window)

We measured the subject's performance at the n-back task based on the accuracy of the subject's response for each digit.

the accuracy of each n-back task was recorded.

Please see details in the screenshot of fake data sample as below:


# nasa-tlx (opens new window)

This is a good way to evaluate the perceived/subjective mental workload.

Total 16 "serial_feedback" .csv files for 16 n-back tasks:

  • 12 files start with "train_1_1_" (1-12) match the first 1-12 tasks,
  • 4 files start with "test_1_2_" (1-4) match the last 4 tasks.
  • Useful features include:
    • movement: subject report if the headband is moved or not,
    • uncomfortable: subject report if feeling unfortable during the experiment or not,
    • mental: mental workload after the task from low (0) to high (100)
    • performance: performance after the task from low (0) to high (100)
    • effort: effort needed for the task from low (0) to high (100)
    • frustration: frustration felt during the task from low (0) to high (100)

Please see details in the screenshot of deidentified data sample as below: nasa-tlx


# log (opens new window)

Hair blocking, light leaking, fNIRS instrument settings and other issues during the experiment were recorded.


# interview (opens new window)

Post-experiment interviews were converted from audio to text (pdf version) by the operator.

The original audios were destroyed immediately following the IRB protocol.


# Code (opens new window)

Please check README in the repo.

Taming fNIRS-based BCI Input for Better Calibration and Broader Use

Taming fNIRS-based BCI Input for Better Calibration and Broader Use