Skip to main content
Ctrl+K
Logo image

Site Navigation

  • The largest EEG benchmark
  • Datasets
  • Install
  • Overview
  • Getting Started
  • Gallery
  • How does Moabb work?
  • API reference
  • Release notes

Site Navigation

  • The largest EEG benchmark
  • Datasets
  • Install
  • Overview
  • Getting Started
  • Gallery
  • How does Moabb work?
  • API reference
  • Release notes

Section Navigation

  • Getting Started
    • Tutorial 4: Creating a dataset class
    • Tutorial 5: Creating a dataset class
    • Tutorial 0: Getting Started
    • Tutorial 1: Simple Motor Imagery
    • Tutorial 2: Using multiple datasets
    • Tutorial 3: Benchmarking multiple pipelines
  • Examples
    • Change Download Directory
    • Benchmarking with MOABB showing the CO2 footprint
    • Load Model (Scikit, Keras) with MOABB
    • Convert a MOABB dataset to BIDS
    • Spectral analysis of the trials
    • Changing epoch size in P300 VR dataset
    • Hinss2021 classification example
    • Examples of how to use MOABB to benchmark pipelines.
    • Benchmarking on MOABB with Tensorflow deep net architectures
    • Benchmarking with MOABB with Grid Search
    • Cross-Session Motor Imagery
    • Cross-Session on Multiple Datasets
    • Cross-Subject SSVEP
    • Cache on disk intermediate data processing states
    • Explore Paradigm Object
    • Fixed interval windows processing
    • Within Session P300
    • Within Session SSVEP
  • API
    • moabb.datasets.AlexMI
    • moabb.datasets.BNCI2014_001
    • moabb.datasets.BNCI2014_002
    • moabb.datasets.BNCI2014_004
    • moabb.datasets.BNCI2015_001
    • moabb.datasets.BNCI2015_004
    • moabb.datasets.Cho2017
    • moabb.datasets.Lee2019_MI
    • moabb.datasets.GrosseWentrup2009
    • moabb.datasets.Ofner2017
    • moabb.datasets.PhysionetMI
    • moabb.datasets.Schirrmeister2017
    • moabb.datasets.Shin2017A
    • moabb.datasets.Shin2017B
    • moabb.datasets.Weibo2014
    • moabb.datasets.Zhou2016
    • moabb.datasets.Stieger2021
    • moabb.datasets.Liu2024
    • moabb.datasets.BI2012
    • moabb.datasets.BI2013a
    • moabb.datasets.BI2014a
    • moabb.datasets.BI2014b
    • moabb.datasets.BI2015a
    • moabb.datasets.BI2015b
    • moabb.datasets.Cattan2019_VR
    • moabb.datasets.BNCI2014_008
    • moabb.datasets.BNCI2014_009
    • moabb.datasets.BNCI2015_003
    • moabb.datasets.DemonsP300
    • moabb.datasets.EPFLP300
    • moabb.datasets.Huebner2017
    • moabb.datasets.Huebner2018
    • moabb.datasets.Lee2019_ERP
    • moabb.datasets.Sosulski2019
    • moabb.datasets.Kalunga2016
    • moabb.datasets.Nakanishi2015
    • moabb.datasets.Wang2016
    • moabb.datasets.MAMEM1
    • moabb.datasets.MAMEM2
    • moabb.datasets.MAMEM3
    • moabb.datasets.Lee2019_SSVEP
    • moabb.datasets.Thielen2015
    • moabb.datasets.Thielen2021
    • moabb.datasets.CastillosBurstVEP40
    • moabb.datasets.CastillosBurstVEP100
    • moabb.datasets.CastillosCVEP40
    • moabb.datasets.CastillosCVEP100
    • moabb.datasets.Cattan2019_PHMD
    • moabb.datasets.Hinss2021
    • moabb.datasets.Rodrigues2017
    • moabb.datasets.base.BaseDataset
    • moabb.datasets.base.CacheConfig
    • moabb.datasets.fake.FakeDataset
    • moabb.datasets.fake.FakeVirtualRealityDataset
    • moabb.datasets.download.data_path
    • moabb.datasets.download.data_dl
    • moabb.datasets.download.fs_issue_request
    • moabb.datasets.download.fs_get_file_list
    • moabb.datasets.download.fs_get_file_hash
    • moabb.datasets.download.fs_get_file_id
    • moabb.datasets.download.fs_get_file_name
    • moabb.datasets.utils.dataset_search
    • moabb.datasets.utils.find_intersecting_channels
    • moabb.datasets.compound_dataset.BI2014a_Il
    • moabb.datasets.compound_dataset.BI2014b_Il
    • moabb.datasets.compound_dataset.BI2015a_Il
    • moabb.datasets.compound_dataset.BI2015b_Il
    • moabb.datasets.compound_dataset.Cattan2019_VR_Il
    • moabb.datasets.compound_dataset.BI_Il
    • moabb.evaluations.WithinSessionEvaluation
    • moabb.evaluations.CrossSessionEvaluation
    • moabb.evaluations.CrossSubjectEvaluation
    • moabb.evaluations.base.BaseEvaluation
    • moabb.paradigms.MotorImagery
    • moabb.paradigms.LeftRightImagery
    • moabb.paradigms.FilterBankLeftRightImagery
    • moabb.paradigms.FilterBankMotorImagery
    • moabb.paradigms.SinglePass
    • moabb.paradigms.P300
    • moabb.paradigms.SSVEP
    • moabb.paradigms.FilterBankSSVEP
    • moabb.paradigms.CVEP
    • moabb.paradigms.FilterBankCVEP
    • moabb.paradigms.RestingStateToP300Adapter
    • moabb.paradigms.FixedIntervalWindowsProcessing
    • moabb.paradigms.FilterBankFixedIntervalWindowsProcessing
    • moabb.paradigms.motor_imagery.BaseMotorImagery
    • moabb.paradigms.motor_imagery.SinglePass
    • moabb.paradigms.motor_imagery.FilterBank
    • moabb.paradigms.p300.BaseP300
    • moabb.paradigms.ssvep.BaseSSVEP
    • moabb.paradigms.BaseFixedIntervalWindowsProcessing
    • moabb.paradigms.base.BaseParadigm
    • moabb.paradigms.base.BaseProcessing
    • moabb.pipelines.features.LogVariance
    • moabb.pipelines.features.FM
    • moabb.pipelines.features.ExtendedSSVEPSignal
    • moabb.pipelines.features.AugmentedDataset
    • moabb.pipelines.features.StandardScaler_Epoch
    • moabb.pipelines.csp.TRCSP
    • moabb.pipelines.classification.SSVEP_CCA
    • moabb.pipelines.classification.SSVEP_TRCA
    • moabb.pipelines.classification.SSVEP_MsetCCA
    • moabb.pipelines.deep_learning.KerasDeepConvNet
    • moabb.pipelines.deep_learning.KerasEEGITNet
    • moabb.pipelines.deep_learning.KerasEEGNet_8_2
    • moabb.pipelines.deep_learning.KerasEEGNeX
    • moabb.pipelines.deep_learning.KerasEEGTCNet
    • moabb.pipelines.deep_learning.KerasShallowConvNet
    • moabb.pipelines.utils.create_pipeline_from_config
    • moabb.pipelines.utils.FilterBank
    • moabb.pipelines.utils_deep_model.EEGNet
    • moabb.pipelines.utils_deep_model.EEGNet_TC
    • moabb.pipelines.utils_deep_model.TCN_block
    • moabb.analysis.plotting.score_plot
    • moabb.analysis.plotting.paired_plot
    • moabb.analysis.plotting.summary_plot
    • moabb.analysis.plotting.meta_analysis_plot
    • moabb.analysis.meta_analysis.find_significant_differences
    • moabb.analysis.meta_analysis.compute_dataset_statistics
    • moabb.analysis.meta_analysis.combine_effects
    • moabb.analysis.meta_analysis.combine_pvalues
    • moabb.analysis.meta_analysis.collapse_session_scores
    • moabb.benchmark
    • moabb.set_log_level
    • moabb.setup_seed
    • moabb.set_download_dir
    • moabb.make_process_pipelines
  • Documentation overview
  • Examples
  • Benchmarking with MOABB with Grid Search

Note

Click here to download the full example code

Benchmarking with MOABB with Grid Search#

This example shows how to use MOABB to benchmark a set of pipelines on all available datasets. In particular we run the Gridsearch to select the best hyperparameter of some pipelines and save the gridsearch. For this example, we will use only one dataset to keep the computation time low, but this benchmark is designed to easily scale to many datasets.

# Authors: Igor Carrara <igor.carrara@inria.fr>
#
# License: BSD (3-clause)

import matplotlib.pyplot as plt

from moabb import benchmark, set_log_level
from moabb.analysis.plotting import score_plot


set_log_level("info")

In this example, we will use only the dataset ‘Zhou 2016’.

Running the benchmark#

The benchmark is run using the benchmark function. You need to specify the folder containing the pipelines to use, the kind of evaluation and the paradigm to use. By default, the benchmark will use all available datasets for all paradigms listed in the pipelines. You could restrict to specific evaluation and paradigm using the evaluations and paradigms arguments.

To save computation time, the results are cached. If you want to re-run the benchmark, you can set the overwrite argument to True.

It is possible to indicate the folder to cache the results and the one to save the analysis & figures. By default, the results are saved in the results folder, and the analysis & figures are saved in the benchmark folder.

# In the results folder we will save the gridsearch evaluation
# When write the pipeline in ylm file we need to specify the parameter that we want to test, in format
# pipeline-name__estimator-name_parameter. Note that pipeline and estimator names MUST
# be in lower case (no capital letters allowed).
# If the grid search is already implemented it will load the previous results

results = benchmark(
    pipelines="./pipelines_grid/",
    evaluations=["WithinSession"],
    paradigms=["LeftRightImagery"],
    include_datasets=["Zhou2016"],
    results="./results/",
    overwrite=False,
    plot=False,
    output="./benchmark/",
)
Zhou2016-WithinSession:   0%|          | 0/4 [00:00<?, ?it/s]/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 60 events (all good), 0 – 5 s (baseline off), ~8.0 MB, data loaded,
 'left_hand': 30
 'right_hand': 30>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 59 events (all good), 0 – 5 s (baseline off), ~7.9 MB, data loaded,
 'left_hand': 30
 'right_hand': 29>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")

Zhou2016-WithinSession:  25%|##5       | 1/4 [00:25<01:15, 25.02s/it]/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 40 events (all good), 0 – 5 s (baseline off), ~5.4 MB, data loaded,
 'left_hand': 20
 'right_hand': 20>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")

Zhou2016-WithinSession:  50%|#####     | 2/4 [00:47<00:46, 23.32s/it]/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")

Zhou2016-WithinSession:  75%|#######5  | 3/4 [01:09<00:22, 22.75s/it]/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 40 events (all good), 0 – 5 s (baseline off), ~5.4 MB, data loaded,
 'left_hand': 20
 'right_hand': 20>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:278: UserWarning: warnEpochs <Epochs | 50 events (all good), 0 – 5 s (baseline off), ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")

Zhou2016-WithinSession: 100%|##########| 4/4 [01:30<00:00, 22.37s/it]
Zhou2016-WithinSession: 100%|##########| 4/4 [01:31<00:00, 22.75s/it]
    dataset     evaluation   pipeline  avg score
0  Zhou2016  WithinSession         EN   0.943619
1  Zhou2016  WithinSession  CSP + LDA   0.937954
2  Zhou2016  WithinSession    EN Grid   0.943531

Benchmark prints a summary of the results. Detailed results are saved in a pandas dataframe, and can be used to generate figures. The analysis & figures are saved in the benchmark folder.

score_plot(results)
plt.show()
Scores per dataset and algorithm
/home/runner/work/moabb/moabb/moabb/analysis/plotting.py:70: UserWarning: The palette list has more values (6) than needed (3), which may not be intended.
  sea.stripplot(

Total running time of the script: ( 1 minutes 34.081 seconds)

Estimated memory usage: 522 MB

Download Python source code: plot_benchmark_grid_search.py

Download Jupyter notebook: plot_benchmark_grid_search.ipynb

Gallery generated by Sphinx-Gallery

previous

Benchmarking on MOABB with Tensorflow deep net architectures

next

Cross-Session Motor Imagery

On this page
  • Running the benchmark

© Copyright 2018-2025 MOABB contributors.

Built with the PyData Sphinx Theme 0.13.3.