基于恒虚警的深度神经网络Dropout正则化方法

肖家麟; 李钰; 袁晴龙; 唐志祺

doi:10.14135/j.cnki.1006-3080.20201127005

基于恒虚警的深度神经网络Dropout正则化方法

doi: 10.14135/j.cnki.1006-3080.20201127005

华东理工大学信息科学与工程学院，上海 200237

详细信息

作者简介:
肖家麟（1995-），男，湖南长沙人，硕士生，主要研究方向为智能传感与信息处理、物体识别。E-mail：y30180629@mail.ecust.edu.cn

通讯作者:
李钰，E-mail：liyu@ecust.edu.cn

中图分类号: TP391
计量
- 文章访问数: 173
- HTML全文浏览量: 156
- PDF下载量: 1
- 被引次数: 0
出版历程
- 收稿日期: 2020-11-27
- 网络出版日期: 2021-03-24

Dropout Regularization Method of Convolutional Neural Network Based on Constant False Alarm Rate

School of Information Science and Engineering, East China University of Science and Technology, Shanghai 200237, China

摘要

摘要: 为进一步提高深度神经网络算法在嵌入式机器人系统中的物体识别性能，提出了一种基于恒虚警检测的深度神经网络Dropout正则化方法（CFAR-Dropout）。首先，通过对权重进行量化，将权重和激活从浮点数减少到二进制值；然后，设计了一个恒虚警检测器（CFAR），保持一定的虚警率，自适应地删减一些神经元节点，优化参与计算的神经元节点；最后，在嵌入式平台PYNQ-Z2上，使用基于VGG16的优化模型对算法的物体识别性能进行实验验证。实验结果表明，与使用经典的Dropout正则化方法相比，CFAR-Dropout正则化方法的错误率降低了2%，有效防止了过拟合；与原本的网络结构相比，参数量所占内存减少到8%左右，有效防止了过参数化。
- 物体识别 /
- 嵌入式 /
- 深度神经网络 /
- 恒虚警 /
- 正则化
Abstract: Compared with traditional machine learning algorithms, deploying deep neural networks in embedded systems can significantly improve the performance of robot system object recognition. However, the computing resources and memory capacity of the embedded platform are limited. It is necessary to simplify the network structure and improve the system efficiency through methods such as model pruning and parameter quantification. It is also necessary to prevent overfitting through Dropout regularization and improve the accuracy of system recognition. In order to further improve the object recognition performance of the deep neural network algorithm in the embedded robot system, this paper proposes a deep neural network dropout regularization method based on constant false alarm detection (CFAR-Dropout). First, by quantizing the weights, the weights and activations are reduced from floating point numbers to binary values; then, a constant false alarm detector (CFAR) is designed to maintain a certain false alarm rate and adaptively delete some neuron nodes. Optimize the neuron nodes involved in the calculation; finally, on the embedded platform PYNQ-Z2, an optimization model based on VGG16 is used to experimentally verify the object recognition performance of the algorithm. Experimental results show that, compared with the classic Dropout regularization method, the error rate of the CFAR-Dropout regularization method is reduced by 2%, effectively preventing overfitting; compared with the original network structure, the amount of memory occupied by the parameters is reduced to Around 8%, effectively preventing over-parameterization.
- object recognition /
- embedded /
- deep neural network /
- constant false alarm /
- regularization

HTML全文

图 1 Dropout正则化前后对比

Figure 1. Comparison before and after Dropout

下载: 全尺寸图片幻灯片

图 2 权重初始值是标准差为1和0.01的高斯分布时激活值的分布

Figure 2. Initial weight value is the distribution of the activation value for a Gaussian distribution with standard deviations of 1 and 0.01

下载: 全尺寸图片幻灯片

图 3 权重初始值是标准差为$ 1/\sqrt{n} $的高斯分布时激活值的分布

Figure 3. Initial value of the weight is the distribution of the activation value with a Gaussian distribution of standard deviation $ 1/\sqrt{n} $

下载: 全尺寸图片幻灯片

图 4 CFAR-Dropout原理简图

Figure 4. CFAR-Dropout schematic diagram

下载: 全尺寸图片幻灯片

图 5 卷积和池化部分IP核

Figure 5. Binarization error rate convolution and pooling of IP cores

下载: 全尺寸图片幻灯片

图 6 物体识别系统检测步骤

Figure 6. Object recognition system detection steps

下载: 全尺寸图片幻灯片

图 7 恒虚警率不同时的测试错误率

Figure 7. Test error rates with different constant false alarm rates

下载: 全尺寸图片幻灯片

图 8 实验设备

Figure 8. Experimental equipment

下载: 全尺寸图片幻灯片

图 9 一般物体边缘提取

Figure 9. General object edge extraction

下载: 全尺寸图片幻灯片

图 10 细胞培养板边缘提取

Figure 10. Cell cultrue plate edge extraction

下载: 全尺寸图片幻灯片

图 11 细胞培养板的识别结果

Figure 11. Recognition results of cell culture plate

下载: 全尺寸图片幻灯片

表 1 网络结构参数表

Table 1. Network structure

Type	Channel	Kernel
CONV1	64	3×3
CONV2	64	3×3
MAXPOOL1	64	2×2
CONV3	128	3×3
CONV4	128	3×3
MAXPOOL2	128	2×2
CONV5	256	3×3
CONV6	256	3×3
CONV7	256	3×3
MAXPOOL3	256	2×2
CONV8	512	3×3
CONV9	512	3×3
CONV10	512	3×3
MAXPOOL4	512	2×2
CONV11	512	3×3
CONV12	512	3×3
CONV13	512	3×3
MAXPOOL5	512	2×2
FC1（Dropout）	/	/
FC2（Dropout）	/	/
FC3	/	/

下载: 导出CSV

表 2 不同正则化方法的错误率

Table 2. Error rates of different regularization methods

Model	Method	Training error rate/%	Testing error rate /%
A- MNIST	/	1.56	16.32
A- MNIST	Dropout	2.95	3.12
A- MNIST	CFAR-Dropout	1.89	2.01
A-CIFAR10	/	12.16	22.04
A-CIFAR10	Dropout	14.23	15.26
A-CIFAR10	CFAR-Dropout	12.21	13.51
A-SVHN	/	4.46	18.53
A-SVHN	Dropout	7.38	7.44
A- SVHN	CFAR-Dropout	5.37	5.58

下载: 导出CSV

表 3 不同的正则化方法在PYNQ上的错误率

Table 3. Different regularization methods have different training error rates on PYNQ

Model	Method	Error rate (LAN)/%	Error rate (Cam)/%
A- MNIST	Dropout	2.98	3.32
A- MNIST	CFAR-Dropout	1.93	2.78
A-CIFAR10	Dropout	14.56	15.04
A-CIFAR10	CFAR-Dropout	12.42	13.05
A-SVHN	Dropout	7.62	9.68
A- SVHN	CFAR-Dropout	5.59	6.05

下载: 导出CSV

表 4 网络层数对正确率的影响

Table 4. Influence of network layer numbers on accuracy

Network layer number	Binarization error rate/%	Floating point error rate/%	Energy efficiency ratio
64	6.43	3.78	2.67
128	6.11	2.73	3.35
512	4.11	1.72	3.54
1 024	2.23	1.21	4.23
2 048	1.37	0.98	6.34

下载: 导出CSV

表 5 不同网络结构下参数与运算量变化

Table 5. Variation of parameters and computation under different network structures

Type	Parameters/MB	Convolution layer	Connection layer	GFLOPS
VGG16	138.36	13	3	15.5
MobileNet	4.2	5	3	0.54
ResNet50	25.6	53	1	3.9
A	10.5	13	3	241.3

下载: 导出CSV

参考文献(23)

[1]	ALEX K, ILYA S, GEOFFREY E H. Imagenet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90. doi: 10.1145/3065386
[2]	ANDRI R, CAVIGELLI L, ROSSI D, et al. YodaNN: An architecture for ultralow power binary-weight CNN acceleration[J]. IEEE Transactions on CAD of Integrated Circuits and Systems, 2018, 37(1): 48-60. doi: 10.1109/TCAD.2017.2682138
[3]	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[C]//International Conference on Learning Representations. [s. l.]: [s. n.], 2015: 1-14.
[4]	SALEHINEJAD H, VALAEE S. Ising-dropout: A regularization method for training and compression of deep neural networks[C]//2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). USA: IEEE, 2019: 3602-3606.
[5]	CHEN L, GAUTIER P, AYDORE S. DropCluster: A structured dropout for convolutional networks[EB/OL]. arxiv. org, (2020-2-7) [2020-10-10]. https:arxiv.org/abs/2002.02997. arXiv preprint arXiv: 2002.02997, 2020.
[6]	QI J, LIU X, TEJEDOR J. Variational inference-based Dropout in recurrent neural networks for slot filling in spoken language understanding[EB/OL]. arxiv. org, (2020-8-23)[2020-10-10]. https:arxiv.org/abs/2009.01003.
[7]	LUO R, ZHONG X, CHEN E. Image classification with a MSF dropout[J]. Multimedia Tools and Applications, 2020, 79(7): 4365-4375.
[8]	TAM E, DUNSON D. Fiedler regularization: Learning neural networks with graph sparsity[EB/OL]. arxiv. org, (2020-3-2)[2020-10-10]. https:arxiv.org/abs/2003.00992?context=cs.LG.
[9]	TSENG H Y, CHEN Y W, TSAI Y H, et al. Regularizing meta-learning via gradient Dropout[EB/OL]. arxiv. org, (2020-4-13)[2020-10-10]. https:arxiv.org/abs/2004.05859.
[10]	STEVERSON K, MULLIN J, AHISKALI M. Adversarial robustness for machine learning cyber defenses using log data[EB/OL]. arxiv. org, (2020-6-9)[2020-10-10]. https:arxiv.org/abs/2007.14983v1.
[11]	CAI S, SHU Y, WANG W, et al. Effective and efficient dropout for deep convolutional neural networks[EB/OL]. arxiv. org, (2019-4-6)[2020-10-10]. https:arxiv.org/abs/1904.03392.
[12]	胡辉, 司凤洋, 曾琛, 等. 一种结合Dropblock和Dropout的正则化策略[J]. 河南师范大学学报(自然科学版), 2019, 47(6): 51-56.
[13]	钟忺, 陈恩晓, 罗瑞奇, 等. 多尺度融合dropout优化算法[J]. 华中科技大学学报(自然科学版), 2018, 46(9): 35-39.
[14]	刘磊. 基于深度神经网络的视网膜病变检测方法研究[D]. 合肥: 中国科学技术大学, 2019.
[15]	WAN L, ZEILER M, ZHANG S, et al. Regularization of neural networks using dropconnect[J]. Journal of Machine Learning Research, 2013, 28(1): 1058-1066.
[16]	MOLCHANOV D, ASHUKHA A, VETROV D. Variational dropout sparsifies deep neural networks[C]// Proceedings of the 34th International Conference on Machine Learning. USA: ACM, 2017: 2498-2507.
[17]	SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al. Dropout: A simple way to prevent neural networks from overfitting[J]. Journal of Machine Learning Research, 2014, 15(1): 1929-1958.
[18]	PROVILKOV I, EMELIANENKO D, VOITA E. Bpe-dropout: Simple and effective subword regularization[EB/OL]. arxiv. org, (2019-10-29)[2020-10-10]. https:arxiv.org/abs/1910.13267.
[19]	LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324. doi: 10.1109/5.726791
[20]	COURBARIAUX M, HUBARA I, SOUDRY D, et al. Binarized neural networks: Training deep neural networks with weights and activations constrained to+1 or-1[EB/OL]. arxiv. org, (2016-2-9)[2020-10-10]. https:arxiv.org/abs/1602.02830v3.
[21]	HEGDE G, RAMASAMY N, KAPRE N. CaffePresso: An optimized library for deep learning on embedded accelerator-based platforms[C]//2016 International Conference on Compliers, Architectures, and Sythesis of Embedded Systems (CASES). USA: IEEE, 2016: 1-10.
[22]	何友, 关键, 孟祥伟. 雷达目标检测与恒虚警处理[M]. 北京: 清华大学出版社, 2011: 15-32.
[23]	SUN Y, WANG X, TANG X. Sparsifying neural network connections for face recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. USA: IEEE, 2016: 4856-4864.