Food ingestion rate estimation method via portable device camera

FOOD INGESTION RATE ESTIMATION METHOD USING A PORTABLE DEVICE CAMERA

The invention relates to a method of estimating the individual rate of food ingestion by means of a portable device, a suitable PC program (interface) and a deep learning algorithm. In particular, this invention allows the recording of a person by a camera of a portable device during a meal, the processing of the video in near real time and the estimation of the individual rate of food ingestion, as calculated in the number of food ingestion events per minute. It has been shown in the relevant literature that the rate of food ingestion is directly linked to the development or existence of diet-related medical problems, such as obesity, type 2 diabetes, gastroesophageal reflux disease, and metabolic syndrome. Therefore, the purpose of the present invention is to apply to people with such medical problems or in general to people who are careful about their diet and metabolic rate, so that they do not develop medical problems in the future. Through visual and acoustic signals from the mobile device, the present invention can intervene during a meal and inform the user that he is exceeding the normal and permissible limits of the rate of food ingestion, thus managing to slow down or even prevent the growth medical problems related to nutrition.

The need to accurately estimate the rate of food ingestion as part of the overall problem of recording eating behavior has been recognized in the literature, and there have been several attempts in this direction, each with its own advantages and disadvantages. Some inventions, such as US2013267794 and US2015379238, use the camera as the dominant element, but it is placed on the person, as a result of which it creates disturbances during their movements and thus limits its usefulness. In fact, the first of the aforementioned inventions uses two cameras, simultaneously increasing the cost and complexity of such a system. Also, both inventions require powerful computing systems to estimate the rate of food ingestion, further limiting their outdoor utility and therefore their portability and scope. Other inventions, such as KR20180116779, trying to avoid the aforementioned problems resort to portable devices, equipped with a camera and a processor, take photos of the food at regular intervals or at the beginning and end of the meal, calculate the differences from the photos and estimate the rate and volume of food ingestion. However, such inventions either estimate only an overall rate of food ingestion once the meal is finished, or exhibit serious accuracy problems due to the great difficulty of estimating food volume from photographs.

Many attempts to estimate the changing rate of food ingestion have also been made using smart watches or wristbands. Such applications take advantage of the time signals generated by the accelerometer and gyroscope of these devices to recognize hand movements and correlate them with the corresponding movements made during the process of swallowing food. A major disadvantage of these applications lies in the need to collect the entire signals before processing them to estimate the rate of ingestion, so that it is not possible to intervene in the person during the meal. Also, such apps face accuracy issues because movements of the person to wipe or touch their face can be mistakenly identified as ingesting movements.

Finally, some recent studies have relied on portable scales that allow accurate estimation of the changing rate of ingestion and also the volume of food per bite. A serious disadvantage of such an application lies in the inherent limitation of the scale to provide weight information from only one plate of food, with the result that an accurate estimation of the rate of food ingestion is not possible when the subject consumes food from two or more plates. Also, another disadvantage of such an application is the need to acquire and transport the portable scale where necessary, burdening the budget of each interested party and limiting its usefulness.

The invention addresses the problem of estimating a person's food ingestion rate in near real time using a handheld device camera and therefore offers specific advantages as listed below:

A) Estimation of food ingestion rate with high accuracy thanks to the processing of video sequences, which provide information about both the body (including the hands) and the face of the user.

B) Estimation of food ingestion rate without the use of specialized or additional sensors (more than one camera or scales).

C) Estimation of food ingestion rate in near real time during a meal.

A detailed explanation and description of each of the advantages of the invention will be given separately.

Advantage A: Accuracy in food ingestion rate estimation

The invention estimates food ingestion rate through high-speed video processing and estimation of the number of food ingestion events per minute (bites/min). The invention is based on the use of a deep learning algorithm that has been trained on a database of labeled videos of people recorded during various meals and has achieved high accuracy in estimating the rate of food ingestion. The fact that the algorithm has been trained with different people and meals allows the invention to be used with great precision on new videos. In addition, the fact that it is based on the combined processing of visual data of the user's movement (body, hands, face) gives the invention the distinctive ability to avoid false detections that may be due to movements of the person to wipe, touch his face or simply raise his hand, thus bypassing the disadvantages of applications based on non-visual data. Finally, the invention is not limited by the number of dishes, from which a person can consume, thus avoiding the serious drawback of applications based on portable scales.

Advantage B: Portability and cost

The invention does not require additional or specialized sensors and therefore the cost of its use is zero, since it works as a PC program that can be executed on a mobile device and using only its camera, without forcing the user to acquire additional sensors or processing units . Since the majority of the general population has a portable device, which is available wherever they go, it allows the invention to be used to estimate the rate of food ingestion without limitation under any conditions either indoors or outdoors. This fact increases the portability, utility and scope of the invention as opposed to existing applications or inventions.

Advantage C: Valuation in near real time

The invention is based on a deep learning algorithm of low technical specifications that allows its very fast execution even on devices with low processing power, such as a mobile device. In this way the estimation of the rate of food ingestion during a meal is achieved, allowing the invention to intervene through visual and auditory information and enabling the person to modify the rate of food ingestion on the spot when a deviation from normal limits is observed. This fact has the effect of increasing the utility of the invention over other applications that estimate the rate of food ingestion after a meal.

The invention relates to an innovative method of estimating the rate of food ingestion during a meal using video data from a portable device. The main parts of which it consists are the following:

• System for recording and estimating individual food ingestion rate (Figure 1)

The system for recording and estimating individual food ingestion rate includes a user (1), who sits on a chair and consumes food from one or more plates (2) placed on a table. The user's handheld device (3) must be placed on the table at a distance of at least 60 cm from the user so that the user's upper body, including hands and face throughout, can be seen by the handheld device's camera during the meal. Also, the distance of the mobile device from the user must not be more than 100 cm as the user's movements must be clearly visible from the camera of the mobile device. Objects that interfere with the camera's field of view and obscure user movements must be removed. To achieve the above, a support base (4) of the portable device can be used for better recording of the user. It should be noted at this point that the conditions and distances mentioned above are ideal and ensure the proper functioning of the method of estimating the individual food ingestion rate. The system has also been tested at shorter or longer distances and/or with interfering objects, but in such cases a loss of accuracy may be observed. When ready the user can press the start recording button from the PC program and start eating. The food ingestion rate estimation algorithm runs automatically and calculates the food ingestion rate, which is received by the PC program and presented to the user via visual and audio signals. When the user finishes their meal, they can press the button again to stop recording and estimating their food intake rate.

• Food ingestion rate estimation algorithm (Figure 2)

The food ingestion rate estimation algorithm starts running when the user presses the corresponding button to start recording and stops running when the user presses the corresponding button to stop recording. The algorithm is based on deep learning techniques to process video data and estimate food ingestion rate. First, the video during recording is divided into separate image frames (Step 1). Next, a highly discriminative neural network (Neural Network 1) is applied, which outputs spatial feature vectors that describe the content depicted in each frame (Step 2). The extracted feature vectors from consecutive image frames are collected using a rolling window (two seconds in size) to form an overall temporal information for each 2-second video sequence (Step 3). Then, this overall information is fed to a neural network (Neural network 2), which analyzes the time sequence of the information in order to recognize the presence or absence of a food ingestion event (Step 4). Finally, an accumulation variable is used to accumulate the total number of food ingestion events per minute and thereby estimate the food ingestion rate (Step 5). Due to the rolling window, the algorithm updates the food ingestion rate every 10 seconds, providing the user with valuable information about the current food ingestion rate and allowing them to alter it appropriately during the meal.

• PC program (Figure 3)

Part of the invention is the computer program, which has a simple and user-friendly interface that enables the user to easily operate the invention for estimating the rate of food ingestion. The user interface consists of the video display frame (1), the food ingestion rate display bar (2) and the record button (3). More specifically, the user can observe in the display frame in real time what the camera of the mobile phone is recording, so that it can be placed in an appropriate place to image him from the waist up during the meal without interfering with other objects or obstacles. The record button allows the user to start and stop recording at will. On the other hand, the food ingestion rate display bar informs the user of the estimated number of food ingestion events per minute by changing the size and color of the bar accordingly. The color of the display bar is also used to indicate whether the rate of food ingestion is within or outside normal limits, which are defined in the literature and can vary depending on the individual's condition. In the event that the user exceeds normal limits of the rate of food ingestion, the color of the bar changes accordingly and the PC program plays through the speaker (4) of the mobile device an acoustic alert (audio tone) as an additional notification that will guide the user towards reducing the rate of food ingestion and returning it within normal limits.