Abstract: Obesity is associated with the increased risk of diseases. For obesity treatment, it is necessary to record all food intakes per day. However, in most cases, patients do have troubles in estimating the amount of food intake because they are unwillingness to record or lack of related nutritional information. The calorie in food can be estimated via computer vision methods, whose estimated accuracy is determined by two main factors:object detection algorithm and volume estimation method. In order to increase the accuracy of detection and reduce the error of volume estimation in food calorie estimation, this paper proposes a calorie estimation method based on deep learning. This proposed method takes two food images as its inputs:a top view and a side view. Each image includes a calibration object that is used to estimate image's scale factor. Food(s) and calibration object are detected by object detection method called faster region-based convolutional neural networks (Faster R-CNN) and each food's contour is obtained by applying GrabCut algorithm. The calibration object judged by Faster R-CNN is used to calculate the scale factor of each view. Each food's volume can be estimated according to its contour in top view, contour in side view, and scale factors. For improving the volume estimation accuracy, this paper divides different types of food shape into four types, for which the corresponding volume estimation formula is adopted. And then, each food's mass and calorie are estimated by means of density table and nutrition table. In the proposed volume estimation experiments, the error between a estimation result and its corresponding true value does not exceed ±20% for most food. The experimental results show that the estimation results are accurate. Hence, the proposed method in this paper is helpful for those patients who want to control calorie intake. In future research, we will keep on improving our method and develop mobile application.