Note
Click here to download the full example code
Within Session P300#
This example shows how to perform a within session analysis on three different P300 datasets.
We will compare two pipelines :
Riemannian geometry
XDAWN with Linear Discriminant Analysis
We will use the P300 paradigm, which uses the AUC as metric.
# Authors: Pedro Rodrigues <pedro.rodrigues01@gmail.com>
#
# License: BSD (3-clause)
import warnings
import matplotlib.pyplot as plt
import seaborn as sns
from mne.decoding import Vectorizer
from pyriemann.estimation import Xdawn, XdawnCovariances
from pyriemann.tangentspace import TangentSpace
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.pipeline import make_pipeline
import moabb
from moabb.datasets import BNCI2014_009
from moabb.evaluations import WithinSessionEvaluation
from moabb.paradigms import P300
getting rid of the warnings about the future
warnings.simplefilter(action="ignore", category=FutureWarning)
warnings.simplefilter(action="ignore", category=RuntimeWarning)
moabb.set_log_level("info")
Create Pipelines#
Pipelines must be a dict of sklearn pipeline transformer.
pipelines = {}
We have to do this because the classes are called ‘Target’ and ‘NonTarget’ but the evaluation function uses a LabelEncoder, transforming them to 0 and 1
labels_dict = {"Target": 1, "NonTarget": 0}
pipelines["RG+LDA"] = make_pipeline(
XdawnCovariances(
nfilter=2, classes=[labels_dict["Target"]], estimator="lwf", xdawn_estimator="scm"
),
TangentSpace(),
LDA(solver="lsqr", shrinkage="auto"),
)
pipelines["Xdw+LDA"] = make_pipeline(
Xdawn(nfilter=2, estimator="scm"), Vectorizer(), LDA(solver="lsqr", shrinkage="auto")
)
Evaluation#
We define the paradigm (P300) and use all three datasets available for it. The evaluation will return a DataFrame containing a single AUC score for each subject / session of the dataset, and for each pipeline.
Results are saved into the database, so that if you add a new pipeline, it will not run again the evaluation unless a parameter has changed. Results can be overwritten if necessary.
paradigm = P300(resample=128)
dataset = BNCI2014_009()
dataset.subject_list = dataset.subject_list[:2]
datasets = [dataset]
overwrite = True # set to True if we want to overwrite cached results
evaluation = WithinSessionEvaluation(
paradigm=paradigm, datasets=datasets, suffix="examples", overwrite=overwrite
)
results = evaluation.process(pipelines)
BNCI2014-009-WithinSession: 0%| | 0/2 [00:00<?, ?it/s]/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'lampx.tugraz.at'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
0%| | 0.00/18.5M [00:00<?, ?B/s]
0%| | 8.19k/18.5M [00:00<04:36, 67.0kB/s]
0%| | 32.8k/18.5M [00:00<02:07, 145kB/s]
0%|▏ | 88.1k/18.5M [00:00<01:04, 285kB/s]
1%|▎ | 153k/18.5M [00:00<00:48, 379kB/s]
2%|▋ | 329k/18.5M [00:00<00:23, 759kB/s]
4%|█▍ | 688k/18.5M [00:00<00:11, 1.50MB/s]
8%|██▊ | 1.40M/18.5M [00:00<00:05, 2.90MB/s]
15%|█████▋ | 2.82M/18.5M [00:00<00:02, 5.66MB/s]
30%|██████████▉ | 5.47M/18.5M [00:01<00:01, 10.6MB/s]
42%|███████████████▍ | 7.72M/18.5M [00:01<00:00, 13.0MB/s]
55%|████████████████████▍ | 10.3M/18.5M [00:01<00:00, 15.3MB/s]
68%|█████████████████████████▏ | 12.6M/18.5M [00:01<00:00, 16.5MB/s]
82%|██████████████████████████████▏ | 15.1M/18.5M [00:01<00:00, 17.7MB/s]
95%|███████████████████████████████████ | 17.6M/18.5M [00:01<00:00, 18.4MB/s]
0%| | 0.00/18.5M [00:00<?, ?B/s]
100%|█████████████████████████████████████| 18.5M/18.5M [00:00<00:00, 75.0GB/s]
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 576 events (all good), 0 – 0.801 s (baseline off), ~14.5 MB, data loaded,
'Target': 96
'NonTarget': 480>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 576 events (all good), 0 – 0.801 s (baseline off), ~14.5 MB, data loaded,
'Target': 96
'NonTarget': 480>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 576 events (all good), 0 – 0.801 s (baseline off), ~14.5 MB, data loaded,
'Target': 96
'NonTarget': 480>
warn(f"warnEpochs {epochs}")
No hdf5_path provided, models will not be saved.
No hdf5_path provided, models will not be saved.
No hdf5_path provided, models will not be saved.
No hdf5_path provided, models will not be saved.
No hdf5_path provided, models will not be saved.
No hdf5_path provided, models will not be saved.
BNCI2014-009-WithinSession: 50%|##### | 1/2 [00:12<00:12, 13.00s/it]/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'lampx.tugraz.at'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
0%| | 0.00/18.5M [00:00<?, ?B/s]
0%| | 8.19k/18.5M [00:00<04:34, 67.5kB/s]
0%| | 32.8k/18.5M [00:00<02:06, 146kB/s]
1%|▏ | 96.3k/18.5M [00:00<00:58, 315kB/s]
1%|▍ | 209k/18.5M [00:00<00:33, 552kB/s]
2%|▉ | 432k/18.5M [00:00<00:17, 1.01MB/s]
5%|█▊ | 889k/18.5M [00:00<00:09, 1.93MB/s]
10%|███▌ | 1.80M/18.5M [00:00<00:04, 3.72MB/s]
20%|███████▏ | 3.62M/18.5M [00:00<00:02, 7.26MB/s]
36%|█████████████▎ | 6.67M/18.5M [00:01<00:00, 12.7MB/s]
53%|███████████████████▌ | 9.78M/18.5M [00:01<00:00, 16.6MB/s]
67%|████████████████████████▉ | 12.5M/18.5M [00:01<00:00, 18.3MB/s]
82%|██████████████████████████████▍ | 15.3M/18.5M [00:01<00:00, 19.6MB/s]
98%|████████████████████████████████████▍| 18.2M/18.5M [00:01<00:00, 21.0MB/s]
0%| | 0.00/18.5M [00:00<?, ?B/s]
100%|█████████████████████████████████████| 18.5M/18.5M [00:00<00:00, 79.5GB/s]
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 576 events (all good), 0 – 0.801 s (baseline off), ~14.5 MB, data loaded,
'Target': 96
'NonTarget': 480>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 576 events (all good), 0 – 0.801 s (baseline off), ~14.5 MB, data loaded,
'Target': 96
'NonTarget': 480>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:279: UserWarning: warnEpochs <Epochs | 576 events (all good), 0 – 0.801 s (baseline off), ~14.5 MB, data loaded,
'Target': 96
'NonTarget': 480>
warn(f"warnEpochs {epochs}")
No hdf5_path provided, models will not be saved.
No hdf5_path provided, models will not be saved.
No hdf5_path provided, models will not be saved.
No hdf5_path provided, models will not be saved.
No hdf5_path provided, models will not be saved.
No hdf5_path provided, models will not be saved.
BNCI2014-009-WithinSession: 100%|##########| 2/2 [00:25<00:00, 12.85s/it]
BNCI2014-009-WithinSession: 100%|##########| 2/2 [00:25<00:00, 12.88s/it]
Plot Results#
Here we plot the results to compare the two pipelines
fig, ax = plt.subplots(facecolor="white", figsize=[8, 4])
sns.stripplot(
data=results,
y="score",
x="pipeline",
ax=ax,
jitter=True,
alpha=0.5,
zorder=1,
palette="Set1",
)
sns.pointplot(data=results, y="score", x="pipeline", ax=ax, palette="Set1")
ax.set_ylabel("ROC AUC")
ax.set_ylim(0.5, 1)
plt.show()
Total running time of the script: ( 0 minutes 31.378 seconds)
Estimated memory usage: 9 MB