Note
Click here to download the full example code
Examples of how to use MOABB to benchmark pipelines.#
Benchmarking with MOABB#
This example shows how to use MOABB to benchmark a set of pipelines on all available datasets. For this example, we will use only one dataset to keep the computation time low, but this benchmark is designed to easily scale to many datasets.
# Authors: Sylvain Chevallier <sylvain.chevallier@universite-paris-saclay.fr>
#
# License: BSD (3-clause)
import matplotlib.pyplot as plt
from moabb import benchmark, set_log_level
from moabb.analysis.plotting import score_plot
from moabb.paradigms import LeftRightImagery
set_log_level("info")
Loading the pipelines#
The ML pipelines used in benchmark are defined in YAML files, following a simple format. It simplifies sharing and reusing pipelines across benchmarks, reproducing state-of-the-art results.
MOABB comes with complete list of pipelines that cover most of the successful approaches in the literature. You can find them in the pipelines folder. For this example, we will use a folder with only 2 pipelines, to keep the computation time low.
This is an example of a pipeline defined in YAML, defining on which paradigms it can be used, the original publication, and the steps to perform using a scikit-learn API. In this case, a CSP + SVM pipeline, the covariance are estimated to compute a CSP filter and then a linear SVM is trained on the CSP filtered signals.
name: CSP + SVM
paradigms:
- LeftRightImagery
citations:
- https://doi.org/10.1007/BF01129656
- https://doi.org/10.1109/MSP.2008.4408441
pipeline:
- name: Covariances
from: pyriemann.estimation
parameters:
estimator: oas
- name: CSP
from: pyriemann.spatialfilters
parameters:
nfilter: 6
- name: SVC
from: sklearn.svm
parameters:
kernel: "linear"
The sample_pipelines folder contains a second pipeline, a logistic regression
performed in the tangent space using Riemannian geometry.
Selecting the datasets (optional)#
If you want to limit your benchmark on a subset of datasets, you can use the
include_datasets and exclude_datasets arguments. You will need either
to provide the dataset’s object, or a the dataset’s code. To get the list of
available dataset’s code for a given paradigm, you can use the following command:
paradigm = LeftRightImagery()
for d in paradigm.datasets:
print(d.code)
BNCI2014-001
BNCI2014-004
Cho2017
GrosseWentrup2009
Lee2019-MI
Liu2024
PhysionetMotorImagery
Schirrmeister2017
Shin2017A
Stieger2021
Weibo2014
Zhou2016
In this example, we will use only the last dataset, ‘Zhou 2016’.
Running the benchmark#
The benchmark is run using the benchmark function. You need to specify the
folder containing the pipelines to use, the kind of evaluation and the paradigm
to use. By default, the benchmark will use all available datasets for all
paradigms listed in the pipelines. You could restrict to specific evaluation and
paradigm using the evaluations and paradigms arguments.
To save computation time, the results are cached. If you want to re-run the
benchmark, you can set the overwrite argument to True.
It is possible to indicate the folder to cache the results and the one to save
the analysis & figures. By default, the results are saved in the results
folder, and the analysis & figures are saved in the benchmark folder.
results = benchmark(
pipelines="./sample_pipelines/",
evaluations=["WithinSession"],
paradigms=["LeftRightImagery"],
include_datasets=["Zhou2016"],
results="./results/",
overwrite=False,
plot=False,
output="./benchmark/",
)
Zhou2016-WithinSession: 0%| | 0/4 [00:00<?, ?it/s]
0%| | 0.00/156M [00:00<?, ?B/s]
0%| | 19.5k/156M [00:00<14:30, 179kB/s]
0%| | 69.6k/156M [00:00<07:32, 345kB/s]
0%| | 122k/156M [00:00<06:24, 406kB/s]
0%| | 279k/156M [00:00<03:11, 812kB/s]
0%|▏ | 592k/156M [00:00<01:40, 1.55MB/s]
1%|▏ | 1.03M/156M [00:00<01:05, 2.38MB/s]
1%|▌ | 2.10M/156M [00:00<00:32, 4.80MB/s]
2%|▊ | 3.16M/156M [00:00<00:23, 6.38MB/s]
3%|█ | 4.26M/156M [00:00<00:20, 7.52MB/s]
4%|█▋ | 6.75M/156M [00:01<00:12, 12.2MB/s]
5%|██ | 8.47M/156M [00:01<00:11, 13.3MB/s]
7%|██▌ | 10.7M/156M [00:01<00:09, 15.5MB/s]
8%|███ | 12.6M/156M [00:01<00:08, 16.1MB/s]
9%|███▌ | 14.8M/156M [00:01<00:08, 17.1MB/s]
11%|████ | 16.8M/156M [00:01<00:07, 17.5MB/s]
12%|████▌ | 18.9M/156M [00:01<00:07, 18.1MB/s]
13%|█████ | 21.0M/156M [00:01<00:07, 18.3MB/s]
15%|█████▌ | 23.0M/156M [00:01<00:07, 18.5MB/s]
16%|██████ | 25.1M/156M [00:02<00:06, 18.8MB/s]
17%|██████▋ | 27.2M/156M [00:02<00:06, 18.9MB/s]
19%|███████▏ | 29.3M/156M [00:02<00:06, 19.0MB/s]
20%|███████▋ | 31.4M/156M [00:02<00:06, 19.1MB/s]
21%|████████▏ | 33.6M/156M [00:02<00:06, 19.2MB/s]
23%|████████▋ | 35.7M/156M [00:02<00:06, 19.3MB/s]
24%|█████████▏ | 37.8M/156M [00:02<00:06, 19.3MB/s]
26%|█████████▋ | 40.0M/156M [00:02<00:05, 19.4MB/s]
27%|██████████▏ | 42.1M/156M [00:02<00:05, 19.4MB/s]
28%|██████████▊ | 44.2M/156M [00:03<00:05, 19.5MB/s]
30%|███████████▎ | 46.4M/156M [00:03<00:05, 19.5MB/s]
31%|███████████▊ | 48.5M/156M [00:03<00:05, 19.6MB/s]
32%|████████████▎ | 50.7M/156M [00:03<00:05, 19.6MB/s]
34%|████████████▊ | 52.9M/156M [00:03<00:05, 19.7MB/s]
35%|█████████████▍ | 55.0M/156M [00:03<00:05, 19.6MB/s]
37%|█████████████▉ | 57.2M/156M [00:03<00:04, 19.8MB/s]
38%|██████████████▍ | 59.4M/156M [00:03<00:04, 19.8MB/s]
39%|██████████████▉ | 61.6M/156M [00:03<00:04, 20.0MB/s]
41%|███████████████▌ | 63.8M/156M [00:04<00:04, 20.0MB/s]
42%|████████████████ | 66.1M/156M [00:04<00:04, 20.3MB/s]
44%|████████████████▌ | 68.3M/156M [00:04<00:04, 20.2MB/s]
45%|█████████████████▏ | 70.6M/156M [00:04<00:04, 20.5MB/s]
47%|█████████████████▋ | 72.8M/156M [00:04<00:04, 20.4MB/s]
48%|██████████████████▎ | 75.1M/156M [00:04<00:03, 20.6MB/s]
49%|██████████████████▊ | 77.3M/156M [00:04<00:03, 20.5MB/s]
51%|███████████████████▍ | 79.6M/156M [00:04<00:03, 20.8MB/s]
52%|███████████████████▉ | 81.9M/156M [00:04<00:03, 20.8MB/s]
54%|████████████████████▌ | 84.3M/156M [00:05<00:03, 21.0MB/s]
55%|█████████████████████ | 86.6M/156M [00:05<00:03, 21.0MB/s]
57%|█████████████████████▋ | 88.9M/156M [00:05<00:03, 21.1MB/s]
58%|██████████████████████▏ | 91.3M/156M [00:05<00:03, 21.2MB/s]
60%|██████████████████████▊ | 93.6M/156M [00:05<00:02, 21.2MB/s]
61%|███████████████████████▎ | 95.9M/156M [00:05<00:02, 21.2MB/s]
63%|███████████████████████▉ | 98.3M/156M [00:05<00:02, 21.4MB/s]
64%|█████████████████████████▏ | 101M/156M [00:05<00:02, 21.4MB/s]
66%|█████████████████████████▋ | 103M/156M [00:05<00:02, 21.4MB/s]
67%|██████████████████████████▎ | 105M/156M [00:06<00:02, 21.5MB/s]
69%|██████████████████████████▉ | 108M/156M [00:06<00:02, 21.6MB/s]
70%|███████████████████████████▍ | 110M/156M [00:06<00:02, 21.6MB/s]
72%|████████████████████████████ | 113M/156M [00:06<00:02, 21.7MB/s]
74%|████████████████████████████▋ | 115M/156M [00:06<00:01, 21.8MB/s]
75%|█████████████████████████████▏ | 117M/156M [00:06<00:01, 21.3MB/s]
77%|█████████████████████████████▊ | 120M/156M [00:06<00:01, 21.9MB/s]
78%|██████████████████████████████▍ | 122M/156M [00:06<00:01, 21.6MB/s]
80%|███████████████████████████████ | 124M/156M [00:06<00:01, 22.0MB/s]
81%|███████████████████████████████▋ | 127M/156M [00:06<00:01, 21.8MB/s]
83%|████████████████████████████████▎ | 129M/156M [00:07<00:01, 22.2MB/s]
84%|████████████████████████████████▉ | 132M/156M [00:07<00:01, 21.9MB/s]
86%|█████████████████████████████████▌ | 134M/156M [00:07<00:00, 22.2MB/s]
87%|██████████████████████████████████ | 137M/156M [00:07<00:00, 22.0MB/s]
89%|██████████████████████████████████▋ | 139M/156M [00:07<00:00, 22.2MB/s]
91%|███████████████████████████████████▎ | 141M/156M [00:07<00:00, 22.0MB/s]
92%|███████████████████████████████████▉ | 144M/156M [00:07<00:00, 22.1MB/s]
94%|████████████████████████████████████▌ | 146M/156M [00:07<00:00, 22.1MB/s]
95%|█████████████████████████████████████ | 149M/156M [00:07<00:00, 22.1MB/s]
97%|█████████████████████████████████████▋ | 151M/156M [00:08<00:00, 22.2MB/s]
98%|██████████████████████████████████████▎| 154M/156M [00:08<00:00, 22.1MB/s]
100%|██████████████████████████████████████▉| 156M/156M [00:08<00:00, 22.2MB/s]
0%| | 0.00/156M [00:00<?, ?B/s]
100%|████████████████████████████████████████| 156M/156M [00:00<00:00, 626GB/s]
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 60 events (all good), 0 – 5 s (baseline off), ~8.0 MB, data loaded,
'left_hand': 30
'right_hand': 30>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 59 events (all good), 0 – 5 s (baseline off), ~7.9 MB, data loaded,
'left_hand': 30
'right_hand': 29>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
Zhou2016-WithinSession: 25%|##5 | 1/4 [00:17<00:52, 17.37s/it]/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 40 events (all good), 0 – 5 s (baseline off), ~5.4 MB, data loaded,
'left_hand': 20
'right_hand': 20>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
Zhou2016-WithinSession: 50%|##### | 2/4 [00:21<00:19, 9.81s/it]/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
Zhou2016-WithinSession: 75%|#######5 | 3/4 [00:26<00:07, 7.48s/it]/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 40 events (all good), 0 – 5 s (baseline off), ~5.4 MB, data loaded,
'left_hand': 20
'right_hand': 20>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
'left_hand': 25
'right_hand': 25>
warn(f"warnEpochs {epochs}")
Zhou2016-WithinSession: 100%|##########| 4/4 [00:31<00:00, 6.32s/it]
Zhou2016-WithinSession: 100%|##########| 4/4 [00:31<00:00, 7.78s/it]
dataset evaluation pipeline avg score
0 Zhou2016 WithinSession CSP + SVM 0.932315
1 Zhou2016 WithinSession Tangent Space LR 0.941601
Benchmark prints a summary of the results. Detailed results are saved in a
pandas dataframe, and can be used to generate figures. The analysis & figures
are saved in the benchmark folder.
score_plot(results)
plt.show()
/home/runner/work/moabb/moabb/moabb/analysis/plotting.py:70: UserWarning: The palette list has more values (6) than needed (2), which may not be intended.
sea.stripplot(
Total running time of the script: ( 0 minutes 32.384 seconds)
Estimated memory usage: 325 MB