基于混合自编码器的脑力负荷识别

周颖; 陈兰岚

doi:10.14135/j.cnki.1006-3080.20210922003

基于混合自编码器的脑力负荷识别

doi: 10.14135/j.cnki.1006-3080.20210922003

周颖,
陈兰岚^,

华东理工大学能源化工过程智能制造教育部重点实验室, 上海200237

基金项目: 国家自然科学基金(61976091) ；中央高校基本科研业务费专项资金

详细信息

作者简介:
周颖：周　颖（1996—），女，江苏无锡人，硕士生，主要研究方向为机器学习及其应用。E-mail：zhou1124zy@163.com

通讯作者:
陈兰岚，E-mail：llchen@ecust.edu.cn

中图分类号: TP391
计量
- 文章访问数: 182
- HTML全文浏览量: 140
- PDF下载量: 21
- 被引次数: 0
出版历程
- 收稿日期: 2021-09-22
- 录用日期: 2021-12-07
- 网络出版日期: 2022-04-16
- 刊出日期: 2023-02-28

Recognition of Mental Workload Based on Hybrid Autoencoders

ZHOU Ying,
CHEN Lanlan^,

Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai 200237, China

摘要

摘要: 为了消除多通道近红外光谱信号中存在的冗余信息并提取抽象特征，构建了基于混合自编码器的脑力负荷识别模型。首先，将原始信号送入栈式自编码器中进行通道降维；然后使用卷积自编码器对降维后的信号进行无监督学习从而提取抽象特征，并将特征依次送入支持向量机、K 最近邻、随机森林这3种基分类器中进行建模；最后，用软、硬投票的集成策略来提高模型对脑力负荷识别的准确性。实验结果表明，混合自编码器具有良好的通道降维和提取抽象特征的能力，该模型在脑力负荷三分类任务中的准确率可以达到95.12%，相对于同类研究准确率有明显提升。
- 脑力负荷识别 /
- 近红外光谱 /
- 自编码器 /
- 抽象特征 /
- 集成学习
Abstract: Mental workload reflects people’s capability of processing information when performing a task, and it can be employed as an indicator of the brain effort. Recently, the assessment on mental workload has been widely studied in various tasks such as simulated flight, cognitive, and so on. The traditional methods of evaluating mental workload include subjective scale method, task performance method and physiological signal parameters method. Near-infrared spectroscopy (NIRS) have many advantages over other physiological techniques as it has better spatial resolution than EEG and better temporal resolution than fMRI. Besides, it is portable and lightweight with simple data acquisition and less exposed to electrical artifacts. In this paper, NIRS are selected to build an assessment model on mental workload. As well known, deep learning with convolutional neural networks has revolutionized signal processing through end-to-end learning due to its efficiency and convenience. In order to eliminate redundant information and extract features from multi-channel NIRS, a novel recognition model on mental workload is created based on the hybrid autoencoders. Firstly, the original signals are sent to the stack autoencoder for channel dimensionality reduction, and these processed signals are fed to the convolutional autoencoder to extract the abstract features. Then, we employ three base classifiers, i.e., the support vector machine (SVM), K nearest neighbors (KNN), random forest (RF), for building models. Finally, the integration strategy of soft voting and hard voting is applied to improve the assessment accuracy for mental workload. It is shown from the results that proper compressing signal channels can help improve the recognition accuracy of the recognition model. The best accuracy of this proposed model for classifying three levels of mental workload can reach 95.12%, which has significant improvement compared to similar studies.
- mental workload recognition /
- near-infrared spectroscopy /
- autoencoder /
- abstract feature /
- ensemble learning

HTML全文

图 1 脑力负荷识别模型框架

Figure 1. Framework of mental workload estimation model

下载: 全尺寸图片幻灯片

图 2 栈式自编码器结构

Figure 2. Structure of stack autoencoder

下载: 全尺寸图片幻灯片

图 3 卷积自编码器结构

Figure 3. Structure of convolutional autoencoder

下载: 全尺寸图片幻灯片

图 4 实验流程

Figure 4. Experiment flow

下载: 全尺寸图片幻灯片

图 5 NIRS通道图

Figure 5. Channels of NIRS

下载: 全尺寸图片幻灯片

图 6 栈式自编码器的训练过程

Figure 6. Training process of stack autoencoder

下载: 全尺寸图片幻灯片

图 7 降维至不同通道数后的分类精度对比

Figure 7. Comparison of classification accuracy under different channel numbers after dimensionality reduction

下载: 全尺寸图片幻灯片

图 8 不同通道降维方法的对比

Figure 8. Comparison of different channel dimensionality reduction methods

下载: 全尺寸图片幻灯片

图 9 卷积自编码器训练过程

Figure 9. Training process of convolutional autoencoder

下载: 全尺寸图片幻灯片

图 10 部分受试者的特征可视化

Figure 10. Visualization of several subjects

下载: 全尺寸图片幻灯片

表 1 卷积自编码器的超参数

Table 1. Hyperparameters of the convolutional autoencoder

Layer	Type	Hyperparameters
1	Input	—
2	Conv 1D	16×5
3	MaxPool 1D	2
4	Conv 1D	64×5
5	BatchNormalization	—
6	MaxPool 1D	2
7	Conv 1D	32×3
8	Conv 1D	1×3
9	Conv 1D	1×3
10	Conv 1D	32×3
11	Up-Sampling	2
12	Conv 1D	64×5
13	Up-Sampling	2
14	Conv 1D	16×5
15	Flatten	—
16	Dense	—
17	Output	—

下载: 导出CSV

表 2 不同模型的复杂度和精度对比

Table 2. Comparison of complexity and accuracy for different models

Feature	Channel reduction	Accuracy/%	Training time/s	Testing time/s
Traditional features	—	82.59	37.98	1.20
	PCA	84.85	35.39	0.80
	SAE	91.17	124.39	1.12
Deep features	—	84.36	9.18	1.08
	PCA	86.75	7.54	0.98
	SAE	93.26	118.53	0.95

下载: 导出CSV

表 3 基分类器的参数设置

Table 3. Parameter settings for base classifiers

Classifer	Parameter
KNN	k = 4
SVM	C = 8, gamma = 0.5, kernel = rbf
RF	n_estimators = 190, max_feature =12

下载: 导出CSV

表 4 基分类器的投票集成结果

Table 4. Voting integration results of base classifiers

Method	Accuracy/%	Recall/%	Precision/%	F1/%
Mean(base classifiers)	93.26	93.47	92.78	92.45
Voting(soft)	95.12	94.87	95.34	94.99
Voting(hard)	94.87	93.96	93.95	93.87

下载: 导出CSV

表 5 同类研究的对比

Table 5. Comparison of similar studies

Method	Physiological signal	Categories	Accuracy/%
Reference[21]	NIRS	2	76.9/76.0
Reference[1]	EEG	2	87.61
Reference[26]	NIRS	3	83.42
Reference[27]	EEG	3	86.12
Reference[28]	NIRS	4	85.90
Reference[29]	EEG、NIRS	3	85.30
Reference[30]	EEG、NIRS	3	96.20
This paper	NIRS	3	95.10

下载: 导出CSV

参考文献(30)

[1]	LIU P K, BEH W K, SHIH C Y, et al. Entropy and complexity assisted EEG-based mental workload assessment system[C]// 2019 IEEE Biomedical Circuits and Systems Conference (BioCAS). Japan : IEEE, 2019: 1-4.
[2]	REINERMAN-JONES L, LIN J, BARBER D, et al. Multi-dimensional workload in two types of nuclear power plant simulators[C]//Proceedings of the Human Factors and Ergonomics Society Annual Meeting. USA: SAGE Publications, 2019: 1777-1781.
[3]	BOMMER S C, FENDLEY M. A theoretical framework for evaluating mental workload resources in human systems design for manufacturing operations[J]. International Journal of Industrial Ergonomics, 2018, 63: 7-17. doi: 10.1016/j.ergon.2016.10.007
[4]	江润强, 陈兰岚, 谌鈫. 基于单模态生理信号无监督特征学习的驾驶压力识别[J]. 华东理工大学学报(自然科学版), 2021, 47(4): 475-482.
[5]	SO W, WONG S, MAK J N, et al. An evaluation of mental workload with frontal EEG[J]. PLoS ONE, 2017, 12(4): e0174949. doi: 10.1371/journal.pone.0174949
[6]	明东, 柯余峰, 何峰, 等. 基于生理信号的脑力负荷检测及自适应自动化系统研究: 40年回顾与最新进展[J]. 电子测量与仪器学报, 2015, 29(1): 1-13.
[7]	SHIVABALAN K R, BROTOTO D, SHIVAM G, et al. Discriminating cognitive performance using biomarkers extracted from linear and nonlinear analysis of EEG signals by machine learning[EB/OL]. (2020-06-02) [2021-09-20], https://www.medrxiv.org/content/10.1101/2020.06.30.20143610v1.
[8]	QU H, SHAN Y, LIU Y, PANG L, et al. Mental workload classification method based on EEG independent component features[J]. Applied Sciences, 2020, 10(9): 3036. doi: 10.3390/app10093036
[9]	VON LÜHMANN A, BOUKOUVALAS Z, MÜLLER K R, et al. A new blind source separation framework for signal analysis and artifact rejection in functional near-infrared spectroscopy[J]. NeuroImage, 2019, 200: 72-88. doi: 10.1016/j.neuroimage.2019.06.021
[10]	MIDHA S, MAIOR H A, WILSON M L, et al. Measuring mental workload variations in office work tasks using fNIRS[J]. International Journal of Human-Computer Studies, 2021, 147: 102580. doi: 10.1016/j.ijhcs.2020.102580
[11]	RAHMAN M, UDDIN M S, AHMAD M. Modeling and classification of voluntary and imagery movements for brain–computer interface from fNIR and EEG signals through convolutional neural network[J]. Health Information Science and Systems, 2019, 7(1): 1-22. doi: 10.1007/s13755-018-0062-0
[12]	SAADATI M, NELSON J, AYAZ H. Mental workload classification from spatial representation of FNIRS recordings using convolutional neural networks[C]// 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP). USA: IEEE, 2019: 1-6.
[13]	KHANMOHAMMADI S, CHOU C A. Adaptive seizure onset detection framework using a hybrid PCA-CSP approach[J]. IEEE Journal of Biomedical and Health Informatics, 2017, 22(1): 154-160.
[14]	ARICAN M, POLAT K. Binary particle swarm optimization (BPSO) based channel selection in the EEG signals and its application to speller systems[J]. Journal of Artificial Intelligence and Systems, 2020, 2(1): 27-37. doi: 10.33969/AIS.2020.21003
[15]	XING X, LI Z, XU T, et al. SAE+LSTM: A new framework for emotion recognition from multi-channel EEG[J]. Frontiers in Neurorobotics, 2019, 13: 37. doi: 10.3389/fnbot.2019.00037
[16]	WEN T, ZHANG Z. Deep convolution neural network and autoencoders-based unsupervised feature learning of EEG signals[J]. IEEE Access, 2018, 6: 25399-25410. doi: 10.1109/ACCESS.2018.2833746
[17]	袁非牛, 章琳, 史劲亭, 等. 自编码神经网络理论及应用综述[J]. 计算机学报, 2019, 42(1): 203-230.
[18]	HINTON G E, ZEMEL R S. Autoencoders, minimum description length and helmholtz free energy[C]//6th International Conference on Neural Information Processing Systems. USA: ACM, 1993: 3-10.
[19]	DU B, XIONG W, WU J, et al. Stacked convolutional denoising auto-encoders for feature representation[J]. IEEE Transactions on Cybernetics, 2017, 47(4): 1017-1027. doi: 10.1109/TCYB.2016.2536638
[20]	魏琛, 陈兰岚, 张傲. 基于集成卷积神经网络的脑电情感识别[J]. 华东理工大学学报(自然科学版), 2019, 45(4): 614-622.
[21]	SHIN J, VON LÜHMANN A, KIM D W, et al. Simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset[J]. Scientific Data, 2018, 5(1): 1-16. doi: 10.1038/s41597-018-0002-5
[22]	KUNCHEVA L I, FAITHFULL W J. PCA feature extraction for change detection in multidimensional unlabeled data[J]. IEEE Transactions on Neural Networks and Learning Systems, 2014, 25(1): 69-80. doi: 10.1109/TNNLS.2013.2248094
[23]	CHEN S, ZHANG J, JIN M. A simplified ICA-based local similarity stereo matching[J]. The Visual Computer, 2020(12): 1-9.
[24]	MCINNES L, HEALY J. UMAP: Uniform manifold approximation and projection for dimension reduction[J]. The Journal of Open Source Software, 2018, 3(29): 861. doi: 10.21105/joss.00861
[25]	焦学军, 张朕, 姜劲, 等. 基于功能性近红外光谱技术的脑机接口[J]. 上海交通大学学报, 2017, 51(12): 1456-1463.
[26]	ASGHER U, KHALIL K, AYAZ Y, et al. Classification of mental workload (MWL) using support vector machines (SVM) and convolutional neural networks (CNN)[C]//2020 3rd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET). USA: IEEE, 2020: 1-6.
[27]	GUPTA S S, MANTHALKAR R R. Classification of visual cognitive workload using analytic wavelet transform[J]. Biomedical Signal Processing and Control, 2020, 61: 101961. doi: 10.1016/j.bspc.2020.101961
[28]	MUGHAL N E, KHALIL K, KHAN M J. fNIRS based multi-class mental workload classification using recurrence plots and CNN-LSTM[C]//2021 International Conference on Artificial Intelligence and Mechatronics Systems (AIMS). USA: IEEE, 2021: 1-6.
[29]	AL-HURAIBI H, RAO K P. Brain-computer interfacing and classification of cognitive activities[J]. Journal of Technological Science & Engineering (JTSE), 2020, 1(2): 26-29.
[30]	KHAN M U, HASAN M A H. Hybrid EEG-fNIRS BCI fusion using multi-resolution singular value decomposition (MSVD)[J]. Frontiers in Human Neuroscience, 2020, 2: 27-37.