Note
Click here to download the full example code
Statistical Analysis#
The MOABB codebase comes with convenience plotting utilities and some statistical testing. This tutorial focuses on what those exactly are and how they can be used.
# Authors: Vinay Jayaram <vinayjayaram13@gmail.com>
#
# License: BSD (3-clause)
# sphinx_gallery_thumbnail_number = -2
import matplotlib.pyplot as plt
from mne.decoding import CSP
from pyriemann.estimation import Covariances
from pyriemann.tangentspace import TangentSpace
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
import moabb
import moabb.analysis.plotting as moabb_plt
from moabb.analysis.meta_analysis import ( # noqa: E501
compute_dataset_statistics,
find_significant_differences,
)
from moabb.datasets import BNCI2014_001
from moabb.evaluations import CrossSessionEvaluation
from moabb.paradigms import LeftRightImagery
moabb.set_log_level("info")
print(__doc__)
Results Generation#
First we need to set up a paradigm, dataset list, and some pipelines to test. This is explored more in the examples – we choose left vs right imagery paradigm with a single bandpass. There is only one dataset here but any number can be added without changing this workflow.
Create Pipelines#
Pipelines must be a dict of sklearn pipeline transformer.
The CSP implementation from MNE is used. We selected 8 CSP components, as usually done in the literature.
The Riemannian geometry pipeline consists in covariance estimation, tangent space mapping and finally a logistic regression for the classification.
pipelines = {}
pipelines["CSP+LDA"] = make_pipeline(CSP(n_components=8), LDA())
pipelines["RG+LR"] = make_pipeline(Covariances(), TangentSpace(), LogisticRegression())
pipelines["CSP+LR"] = make_pipeline(CSP(n_components=8), LogisticRegression())
pipelines["RG+LDA"] = make_pipeline(Covariances(), TangentSpace(), LDA())
Evaluation#
We define the paradigm (LeftRightImagery) and the dataset (BNCI2014_001). The evaluation will return a DataFrame containing a single AUC score for each subject / session of the dataset, and for each pipeline.
Results are saved into the database, so that if you add a new pipeline, it will not run again the evaluation unless a parameter has changed. Results can be overwritten if necessary.
paradigm = LeftRightImagery()
dataset = BNCI2014_001()
dataset.subject_list = dataset.subject_list[:4]
datasets = [dataset]
overwrite = True # set to False if we want to use cached results
evaluation = CrossSessionEvaluation(
paradigm=paradigm, datasets=datasets, suffix="stats", overwrite=overwrite
)
results = evaluation.process(pipelines)
BNCI2014-001-CrossSession: 0%| | 0/4 [00:00<?, ?it/s]/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
BNCI2014-001-CrossSession: 25%|##5 | 1/4 [00:05<00:17, 5.89s/it]/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
BNCI2014-001-CrossSession: 50%|##### | 2/4 [00:11<00:11, 5.77s/it]/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'lampx.tugraz.at'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
0%| | 0.00/44.1M [00:00<?, ?B/s]
0%| | 8.19k/44.1M [00:00<10:52, 67.5kB/s]
0%| | 32.8k/44.1M [00:00<05:01, 146kB/s]
0%| | 96.3k/44.1M [00:00<02:19, 315kB/s]
0%|▏ | 209k/44.1M [00:00<01:19, 553kB/s]
1%|▎ | 432k/44.1M [00:00<00:43, 1.01MB/s]
2%|▊ | 889k/44.1M [00:00<00:22, 1.93MB/s]
4%|█▌ | 1.80M/44.1M [00:00<00:11, 3.73MB/s]
8%|███ | 3.62M/44.1M [00:00<00:05, 7.26MB/s]
15%|█████▌ | 6.68M/44.1M [00:01<00:02, 12.8MB/s]
22%|████████ | 9.54M/44.1M [00:01<00:02, 16.0MB/s]
27%|██████████▏ | 12.1M/44.1M [00:01<00:01, 17.5MB/s]
34%|████████████▍ | 14.8M/44.1M [00:01<00:01, 18.9MB/s]
40%|██████████████▋ | 17.5M/44.1M [00:01<00:01, 19.9MB/s]
46%|████████████████▉ | 20.2M/44.1M [00:01<00:01, 20.4MB/s]
52%|███████████████████▏ | 22.9M/44.1M [00:01<00:01, 20.9MB/s]
59%|█████████████████████▋ | 25.9M/44.1M [00:01<00:00, 22.0MB/s]
65%|████████████████████████▏ | 28.8M/44.1M [00:02<00:00, 22.6MB/s]
72%|██████████████████████████▋ | 31.8M/44.1M [00:02<00:00, 22.9MB/s]
79%|█████████████████████████████▎ | 34.9M/44.1M [00:02<00:00, 23.6MB/s]
86%|███████████████████████████████▉ | 38.0M/44.1M [00:02<00:00, 24.1MB/s]
93%|██████████████████████████████████▍ | 41.0M/44.1M [00:02<00:00, 24.2MB/s]
100%|████████████████████████████████████▉| 44.0M/44.1M [00:02<00:00, 24.3MB/s]
0%| | 0.00/44.1M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 44.1M/44.1M [00:00<00:00, 188GB/s]
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'lampx.tugraz.at'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
0%| | 0.00/42.3M [00:00<?, ?B/s]
0%| | 8.19k/42.3M [00:00<10:27, 67.4kB/s]
0%| | 32.8k/42.3M [00:00<04:50, 146kB/s]
0%| | 88.1k/42.3M [00:00<02:28, 285kB/s]
0%|▏ | 193k/42.3M [00:00<01:22, 508kB/s]
1%|▍ | 409k/42.3M [00:00<00:43, 959kB/s]
2%|▊ | 849k/42.3M [00:00<00:22, 1.85MB/s]
4%|█▍ | 1.71M/42.3M [00:00<00:11, 3.54MB/s]
8%|███ | 3.44M/42.3M [00:00<00:05, 6.89MB/s]
15%|█████▋ | 6.54M/42.3M [00:01<00:02, 12.6MB/s]
22%|████████▎ | 9.46M/42.3M [00:01<00:02, 16.1MB/s]
29%|██████████▌ | 12.1M/42.3M [00:01<00:01, 17.8MB/s]
35%|████████████▉ | 14.8M/42.3M [00:01<00:01, 18.9MB/s]
40%|██████████████▋ | 16.7M/42.3M [00:01<00:01, 17.0MB/s]
46%|█████████████████▏ | 19.7M/42.3M [00:01<00:01, 18.9MB/s]
52%|███████████████████▎ | 22.0M/42.3M [00:01<00:01, 19.0MB/s]
58%|█████████████████████▌ | 24.7M/42.3M [00:01<00:00, 19.9MB/s]
65%|████████████████████████ | 27.5M/42.3M [00:02<00:00, 20.6MB/s]
71%|██████████████████████████▎ | 30.1M/42.3M [00:02<00:00, 21.0MB/s]
78%|████████████████████████████▋ | 32.8M/42.3M [00:02<00:00, 21.2MB/s]
84%|███████████████████████████████ | 35.5M/42.3M [00:02<00:00, 21.4MB/s]
91%|█████████████████████████████████▍ | 38.3M/42.3M [00:02<00:00, 21.7MB/s]
97%|███████████████████████████████████▊ | 41.0M/42.3M [00:02<00:00, 21.9MB/s]
0%| | 0.00/42.3M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 42.3M/42.3M [00:00<00:00, 150GB/s]
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
BNCI2014-001-CrossSession: 75%|#######5 | 3/4 [00:24<00:08, 8.92s/it]/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'lampx.tugraz.at'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
0%| | 0.00/37.2M [00:00<?, ?B/s]
0%| | 8.19k/37.2M [00:00<09:10, 67.4kB/s]
0%| | 32.8k/37.2M [00:00<04:14, 146kB/s]
0%| | 88.1k/37.2M [00:00<02:09, 285kB/s]
1%|▏ | 193k/37.2M [00:00<01:12, 509kB/s]
1%|▍ | 409k/37.2M [00:00<00:38, 960kB/s]
2%|▊ | 841k/37.2M [00:00<00:19, 1.83MB/s]
5%|█▋ | 1.71M/37.2M [00:00<00:09, 3.55MB/s]
9%|███▍ | 3.44M/37.2M [00:00<00:04, 6.90MB/s]
17%|██████▍ | 6.50M/37.2M [00:01<00:02, 12.4MB/s]
25%|█████████▍ | 9.46M/37.2M [00:01<00:01, 16.0MB/s]
33%|████████████▎ | 12.3M/37.2M [00:01<00:01, 18.2MB/s]
41%|███████████████ | 15.1M/37.2M [00:01<00:01, 19.6MB/s]
48%|█████████████████▉ | 18.0M/37.2M [00:01<00:00, 20.6MB/s]
56%|████████████████████▊ | 20.9M/37.2M [00:01<00:00, 21.6MB/s]
64%|███████████████████████▋ | 23.8M/37.2M [00:01<00:00, 22.1MB/s]
72%|██████████████████████████▌ | 26.6M/37.2M [00:01<00:00, 22.5MB/s]
79%|█████████████████████████████▏ | 29.4M/37.2M [00:02<00:00, 22.4MB/s]
87%|████████████████████████████████▏ | 32.3M/37.2M [00:02<00:00, 22.9MB/s]
94%|██████████████████████████████████▉ | 35.0M/37.2M [00:02<00:00, 22.7MB/s]
0%| | 0.00/37.2M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 37.2M/37.2M [00:00<00:00, 105GB/s]
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'lampx.tugraz.at'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
0%| | 0.00/41.7M [00:00<?, ?B/s]
0%| | 8.19k/41.7M [00:00<10:20, 67.3kB/s]
0%| | 32.8k/41.7M [00:00<04:46, 146kB/s]
0%| | 96.3k/41.7M [00:00<02:12, 315kB/s]
1%|▏ | 209k/41.7M [00:00<01:15, 553kB/s]
1%|▍ | 432k/41.7M [00:00<00:41, 1.01MB/s]
2%|▊ | 889k/41.7M [00:00<00:21, 1.92MB/s]
4%|█▌ | 1.80M/41.7M [00:00<00:10, 3.71MB/s]
9%|███▏ | 3.62M/41.7M [00:00<00:05, 7.24MB/s]
16%|█████▊ | 6.53M/41.7M [00:01<00:02, 12.3MB/s]
23%|████████▍ | 9.57M/41.7M [00:01<00:01, 16.1MB/s]
30%|███████████ | 12.4M/41.7M [00:01<00:01, 18.3MB/s]
36%|█████████████▍ | 15.2M/41.7M [00:01<00:01, 19.6MB/s]
44%|████████████████▏ | 18.2M/41.7M [00:01<00:01, 21.0MB/s]
50%|██████████████████▋ | 21.1M/41.7M [00:01<00:00, 21.7MB/s]
58%|█████████████████████▎ | 24.0M/41.7M [00:01<00:00, 22.4MB/s]
65%|████████████████████████ | 27.1M/41.7M [00:01<00:00, 23.1MB/s]
72%|██████████████████████████▌ | 30.0M/41.7M [00:02<00:00, 23.4MB/s]
79%|█████████████████████████████▏ | 32.9M/41.7M [00:02<00:00, 23.3MB/s]
86%|███████████████████████████████▊ | 35.9M/41.7M [00:02<00:00, 23.5MB/s]
93%|██████████████████████████████████▍ | 38.8M/41.7M [00:02<00:00, 23.7MB/s]
0%| | 0.00/41.7M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 41.7M/41.7M [00:00<00:00, 115GB/s]
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s (baseline off), ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
BNCI2014-001-CrossSession: 100%|##########| 4/4 [00:35<00:00, 10.04s/it]
BNCI2014-001-CrossSession: 100%|##########| 4/4 [00:35<00:00, 9.00s/it]
MOABB Plotting#
Here we plot the results using some of the convenience methods within the toolkit. The score_plot visualizes all the data with one score per subject for every dataset and pipeline.
fig = moabb_plt.score_plot(results)
plt.show()
/home/runner/work/moabb/moabb/moabb/analysis/plotting.py:70: UserWarning: The palette list has more values (6) than needed (4), which may not be intended.
sea.stripplot(
For a comparison of two algorithms, there is the paired_plot, which plots performance in one versus the performance in the other over all chosen datasets. Note that there is only one score per subject, regardless of the number of sessions.
fig = moabb_plt.paired_plot(results, "CSP+LDA", "RG+LDA")
plt.show()
Statistical Testing and Further Plots#
If the statistical significance of results is of interest, the method compute_dataset_statistics allows one to show a meta-analysis style plot as well. For an overview of how all algorithms perform in comparison with each other, the method find_significant_differences and the summary_plot are possible.
stats = compute_dataset_statistics(results)
P, T = find_significant_differences(stats)
The meta-analysis style plot shows the standardized mean difference within each tested dataset for the two algorithms in question, in addition to a meta-effect and significance both per-dataset and overall.
fig = moabb_plt.meta_analysis_plot(stats, "CSP+LDA", "RG+LDA")
plt.show()
The summary plot shows the effect and significance related to the hypothesis that the algorithm on the y-axis significantly outperformed the algorithm on the x-axis over all datasets
moabb_plt.summary_plot(P, T)
plt.show()
Total running time of the script: ( 0 minutes 46.616 seconds)
Estimated memory usage: 239 MB