The largest EEG-based Benchmark for Open Science

We report the results of the benchmark study performed in: The largest EEG-based BCI reproducibility study for open science: the MOABB benchmark

This study conducts an extensive Brain-computer interfaces (BCI) reproducibility analysis on open electroencephalography datasets, aiming to assess existing solutions and establish open and reproducible benchmarks for effective comparison within the field. Please note that the results are obtained using Within-Session evaluation. The results are reported regarding mean accuracy and standard deviation across all folds for all sessions and subjects.

If you use the same evaluation procedure, you should expect similar results if you use the same pipelines and datasets, with some minor variations due to the randomness of the cross-validation procedure.

You can copy and use the table in your work, but please **cite the paper** if you do so.

Motor Imagery

Motor Imagery is a BCI paradigm where the subject imagines performing a movement. Each imagery task is associated with a different class, and each task has its difficulty level related to how the brain generates the signal.

Here, we present three different scenarios for Motor Imagery classification:

1. Left vs Right Hand: We use only the classes Left Hand and Right Hand.

2. Right Hand vs Feet: We use only Right Hand and Feet classes.

3. All classes: We use all the classes in the dataset, when there are more than classes that are not Left Hand and Right Hand.

All the results here are for within-session evaluation, a 5-fold cross-validation, over the subject’s session.

Motor Imagery - Left vs Right Hand

Left vs Right Hand: We use only the classes Left Hand and Right Hand.

Pipelines	BNCI2014_001	BNCI2014_004	Cho2017	GrosseWentrup2009	Lee2019_MI	PhysionetMI	Schirrmeister2017	Shin2017A	Weibo2014	Zhou2016

---

Motor Imagery - Right Hand vs Feet

Right Hand vs Feet: We use only Right Hand and Feet classes.

Pipeline	AlexMI	BNCI2014_001	BNCI2014_002	BNCI2015_001	BNCI2015_004	PhysionetMI	Schirrmeister2017	Weibo2014	Zhou2016

Motor Imagery - All classes

All classes: We use all the classes in the dataset, when there are more than classes that are not Left Hand and Right Hand.

Pipelines	AlexMI	BNCI2014_001	PhysionetMI	Schirrmeister2017	Weibo2014	Zhou2016

SSVEP (All classes)

Here, we have the results of the within-session evaluation, a 5-fold cross-validation, over the subject’s session. We use all the classes available in the dataset.

Pipeline	Kalunga2016	Lee2019_SSVEP	MAMEM1	MAMEM2	MAMEM3	Nakanishi2015	Wang2016

P300/ERP (All classes)

Here, we have the results of the within-session evaluation, a 5-fold cross-validation, over the subject’s session. We use all the classes available in the dataset.

Pipelines	BNCI2014_008	BNCI2014_009	BNCI2015_003	BI2012	BI2013a	BI2014a	BI2014b	BI2015a	BI2015b	Cattan2019_VR	EPFLP300	Huebner2017	Huebner2018	Lee2019_ERP	Sosulski2019