高级检索

  • ISSN 1006-3080
  • CN 31-1691/TQ

基于恒虚警的深度神经网络Dropout正则化方法

肖家麟 李钰 袁晴龙 唐志祺

肖家麟, 李钰, 袁晴龙, 唐志祺. 基于恒虚警的深度神经网络Dropout正则化方法[J]. 华东理工大学学报(自然科学版). doi: 10.14135/j.cnki.1006-3080.20201127005
引用本文: 肖家麟, 李钰, 袁晴龙, 唐志祺. 基于恒虚警的深度神经网络Dropout正则化方法[J]. 华东理工大学学报(自然科学版). doi: 10.14135/j.cnki.1006-3080.20201127005
XIAO Jialin, LI Yu, YUAN Qinglong, TANG Zhiqi. Dropout Regularization Method of Convolutional Neural Network Based on Constant False Alarm Rate[J]. Journal of East China University of Science and Technology. doi: 10.14135/j.cnki.1006-3080.20201127005
Citation: XIAO Jialin, LI Yu, YUAN Qinglong, TANG Zhiqi. Dropout Regularization Method of Convolutional Neural Network Based on Constant False Alarm Rate[J]. Journal of East China University of Science and Technology. doi: 10.14135/j.cnki.1006-3080.20201127005

基于恒虚警的深度神经网络Dropout正则化方法

doi: 10.14135/j.cnki.1006-3080.20201127005
详细信息
    作者简介:

    肖家麟(1995-),男,湖南长沙人,硕士生,主要研究方向为智能传感与信息处理、物体识别。E-mail:y30180629@mail.ecust.edu.cn

    通讯作者:

    李 钰,E-mail:liyu@ecust.edu.cn

  • 中图分类号: TP391

Dropout Regularization Method of Convolutional Neural Network Based on Constant False Alarm Rate

  • 摘要: 为进一步提高深度神经网络算法在嵌入式机器人系统中的物体识别性能,提出了一种基于恒虚警检测的深度神经网络Dropout正则化方法(CFAR-Dropout)。首先,通过对权重进行量化,将权重和激活从浮点数减少到二进制值;然后,设计了一个恒虚警检测器(CFAR),保持一定的虚警率,自适应地删减一些神经元节点,优化参与计算的神经元节点;最后,在嵌入式平台PYNQ-Z2上,使用基于VGG16的优化模型对算法的物体识别性能进行实验验证。实验结果表明,与使用经典的Dropout正则化方法相比,CFAR-Dropout正则化方法的错误率降低了2%,有效防止了过拟合;与原本的网络结构相比,参数量所占内存减少到8%左右,有效防止了过参数化。

     

  • 图  1  Dropout正则化前后对比

    Figure  1.  Comparison before and after Dropout

    图  2  权重初始值是标准差为1和0.01的高斯分布时激活值的分布

    Figure  2.  Initial weight value is the distribution of the activation value for a Gaussian distribution with standard deviations of 1 and 0.01

    图  3  权重初始值是标准差为$ 1/\sqrt{n} $的高斯分布时激活值的分布

    Figure  3.  Initial value of the weight is the distribution of the activation value with a Gaussian distribution of standard deviation $ 1/\sqrt{n} $

    图  4  CFAR-Dropout原理简图

    Figure  4.  CFAR-Dropout schematic diagram

    图  5  卷积和池化部分IP核

    Figure  5.  Binarization error rate convolution and pooling of IP cores

    图  6  物体识别系统检测步骤

    Figure  6.  Object recognition system detection steps

    图  7  恒虚警率不同时的测试错误率

    Figure  7.  Test error rates with different constant false alarm rates

    图  8  实验设备

    Figure  8.  Experimental equipment

    图  9  一般物体边缘提取

    Figure  9.  General object edge extraction

    图  10  细胞培养板边缘提取

    Figure  10.  Cell cultrue plate edge extraction

    图  11  细胞培养板的识别结果

    Figure  11.  Recognition results of cell culture plate

    表  1  网络结构参数表

    Table  1.   Network structure

    TypeChannelKernel
    CONV1643×3
    CONV2643×3
    MAXPOOL1642×2
    CONV31283×3
    CONV41283×3
    MAXPOOL21282×2
    CONV52563×3
    CONV62563×3
    CONV72563×3
    MAXPOOL32562×2
    CONV85123×3
    CONV95123×3
    CONV105123×3
    MAXPOOL45122×2
    CONV115123×3
    CONV125123×3
    CONV135123×3
    MAXPOOL55122×2
    FC1(Dropout)//
    FC2(Dropout)//
    FC3//
    下载: 导出CSV

    表  2  不同正则化方法的错误率

    Table  2.   Error rates of different regularization methods

    ModelMethodTraining error rate/%Testing error rate /%
    A- MNIST/1.5616.32
    A- MNISTDropout2.953.12
    A- MNISTCFAR-Dropout1.892.01
    A-CIFAR10/12.1622.04
    A-CIFAR10Dropout14.2315.26
    A-CIFAR10CFAR-Dropout12.2113.51
    A-SVHN/4.4618.53
    A-SVHNDropout7.387.44
    A- SVHNCFAR-Dropout5.375.58
    下载: 导出CSV

    表  3  不同的正则化方法在PYNQ上的错误率

    Table  3.   Different regularization methods have different training error rates on PYNQ

    ModelMethodError rate (LAN)/%Error rate (Cam)/%
    A- MNISTDropout2.983.32
    A- MNISTCFAR-Dropout1.932.78
    A-CIFAR10Dropout14.5615.04
    A-CIFAR10CFAR-Dropout12.4213.05
    A-SVHNDropout7.629.68
    A- SVHNCFAR-Dropout5.596.05
    下载: 导出CSV

    表  4  网络层数对正确率的影响

    Table  4.   Influence of network layer numbers on accuracy

    Network layer numberBinarization error rate/%Floating point error rate/%Energy efficiency ratio
    646.433.782.67
    1286.112.733.35
    5124.111.723.54
    1 0242.231.214.23
    2 0481.370.986.34
    下载: 导出CSV

    表  5  不同网络结构下参数与运算量变化

    Table  5.   Variation of parameters and computation under different network structures

    TypeParameters/MBConvolution layerConnection layerGFLOPS
    VGG16138.3613315.5
    MobileNet4.2530.54
    ResNet5025.65313.9
    A10.5133241.3
    下载: 导出CSV
  • [1] ALEX K, ILYA S, GEOFFREY E H. Imagenet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90. doi: 10.1145/3065386
    [2] ANDRI R, CAVIGELLI L, ROSSI D, et al. YodaNN: An architecture for ultralow power binary-weight CNN acceleration[J]. IEEE Transactions on CAD of Integrated Circuits and Systems, 2018, 37(1): 48-60. doi: 10.1109/TCAD.2017.2682138
    [3] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[C]//International Conference on Learning Representations. [s. l.]: [s. n.], 2015: 1-14.
    [4] SALEHINEJAD H, VALAEE S. Ising-dropout: A regularization method for training and compression of deep neural networks[C]//2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). USA: IEEE, 2019: 3602-3606.
    [5] CHEN L, GAUTIER P, AYDORE S. DropCluster: A structured dropout for convolutional networks[EB/OL]. arxiv. org, (2020-2-7) [2020-10-10]. https:arxiv.org/abs/2002.02997. arXiv preprint arXiv: 2002.02997, 2020.
    [6] QI J, LIU X, TEJEDOR J. Variational inference-based Dropout in recurrent neural networks for slot filling in spoken language understanding[EB/OL]. arxiv. org, (2020-8-23)[2020-10-10]. https:arxiv.org/abs/2009.01003.
    [7] LUO R, ZHONG X, CHEN E. Image classification with a MSF dropout[J]. Multimedia Tools and Applications, 2020, 79(7): 4365-4375.
    [8] TAM E, DUNSON D. Fiedler regularization: Learning neural networks with graph sparsity[EB/OL]. arxiv. org, (2020-3-2)[2020-10-10]. https:arxiv.org/abs/2003.00992?context=cs.LG.
    [9] TSENG H Y, CHEN Y W, TSAI Y H, et al. Regularizing meta-learning via gradient Dropout[EB/OL]. arxiv. org, (2020-4-13)[2020-10-10]. https:arxiv.org/abs/2004.05859.
    [10] STEVERSON K, MULLIN J, AHISKALI M. Adversarial robustness for machine learning cyber defenses using log data[EB/OL]. arxiv. org, (2020-6-9)[2020-10-10]. https:arxiv.org/abs/2007.14983v1.
    [11] CAI S, SHU Y, WANG W, et al. Effective and efficient dropout for deep convolutional neural networks[EB/OL]. arxiv. org, (2019-4-6)[2020-10-10]. https:arxiv.org/abs/1904.03392.
    [12] 胡辉, 司凤洋, 曾琛, 等. 一种结合Dropblock和Dropout的正则化策略[J]. 河南师范大学学报(自然科学版), 2019, 47(6): 51-56.
    [13] 钟忺, 陈恩晓, 罗瑞奇, 等. 多尺度融合dropout优化算法[J]. 华中科技大学学报(自然科学版), 2018, 46(9): 35-39.
    [14] 刘磊. 基于深度神经网络的视网膜病变检测方法研究[D]. 合肥: 中国科学技术大学, 2019.
    [15] WAN L, ZEILER M, ZHANG S, et al. Regularization of neural networks using dropconnect[J]. Journal of Machine Learning Research, 2013, 28(1): 1058-1066.
    [16] MOLCHANOV D, ASHUKHA A, VETROV D. Variational dropout sparsifies deep neural networks[C]// Proceedings of the 34th International Conference on Machine Learning. USA: ACM, 2017: 2498-2507.
    [17] SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al. Dropout: A simple way to prevent neural networks from overfitting[J]. Journal of Machine Learning Research, 2014, 15(1): 1929-1958.
    [18] PROVILKOV I, EMELIANENKO D, VOITA E. Bpe-dropout: Simple and effective subword regularization[EB/OL]. arxiv. org, (2019-10-29)[2020-10-10]. https:arxiv.org/abs/1910.13267.
    [19] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324. doi: 10.1109/5.726791
    [20] COURBARIAUX M, HUBARA I, SOUDRY D, et al. Binarized neural networks: Training deep neural networks with weights and activations constrained to+1 or-1[EB/OL]. arxiv. org, (2016-2-9)[2020-10-10]. https:arxiv.org/abs/1602.02830v3.
    [21] HEGDE G, RAMASAMY N, KAPRE N. CaffePresso: An optimized library for deep learning on embedded accelerator-based platforms[C]//2016 International Conference on Compliers, Architectures, and Sythesis of Embedded Systems (CASES). USA: IEEE, 2016: 1-10.
    [22] 何友, 关键, 孟祥伟. 雷达目标检测与恒虚警处理[M]. 北京: 清华大学出版社, 2011: 15-32.
    [23] SUN Y, WANG X, TANG X. Sparsifying neural network connections for face recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. USA: IEEE, 2016: 4856-4864.
  • 加载中
图(11) / 表(5)
计量
  • 文章访问数:  261
  • HTML全文浏览量:  198
  • PDF下载量:  6
  • 被引次数: 0
出版历程
  • 收稿日期:  2020-11-27
  • 网络出版日期:  2021-03-24

目录

    /

    返回文章
    返回