Activity classification in a multi-axis activity monitor device

CROSS-REFERENCE TO RELATED APPLICATION

0001This application claims the benefit under 35 USC 119(e) of U.S. Provisional Patent Application No. 61/584,710, filed on Jan. 9, 2012, entitled “ACTIVITY CLASSIFICATION IN A MULTI-AXIS ACTIVITY MONITOR DEVICE,” which is incorporated herein by reference in its entirety.

FIELD OF THE INVENTION

0002The present invention relates generally to monitoring devices using sensors and more specifically to classifying activities utilizing such devices.

BACKGROUND OF THE INVENTION

0003Devices for providing inertial activity classification are utilized in a variety of environments. These devices can comprise of accelerometers. Typically the devices record or transmit data that are used for classifying activity based on features derived from accelerometer to a host device for processing.

0004However, it is important for power consumption to be limited in portable device applications such as laptops, tablets, smartphones and the like. Therefore what is desired is a system and method to provide an activity classification in a monitoring device that is accurate, low power, and cost effective. The present invention addresses such a need.

SUMMARY OF THE INVENTION

0005The present invention discloses a device. The device comprises one or more motion sensors and a memory configured to receive signals from the one or more motion sensors. The device further includes a processor in communication with the memory. Finally, the device includes a classification algorithm executed by the processor, the classification algorithm for identifying activities. The device also may include a memory unit to record data such as activity log and calorie count among other data. In an embodiment, the sensor may include (Radio Frequency) RF module to wirelessly transmit the recorded data to the host either upon request or continuously.

BRIEF DESCRIPTION OF THE DRAWINGS

0006<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram of a system that utilizes an activity classification algorithm in accordance with an embodiment.

0007<figref idref="DRAWINGS">FIG. 2A</figref> is a diagram of an activity classification system wherein more than one processor is operating in accordance with an embodiment.

0008<figref idref="DRAWINGS">FIG. 2B</figref> is a diagram of an activity classification system wherein more than one sensor is utilized in a wireless environment in accordance with an embodiment.

0009<figref idref="DRAWINGS">FIG. 3</figref> illustrates an embodiment of an activity classification device to be worn as if it were a wrist watch.

0010<figref idref="DRAWINGS">FIG. 4</figref> illustrates an example of an algorithm for a decision tree generation that is utilized to identify a plurality of activities.

0011<figref idref="DRAWINGS">FIG. 5</figref> illustrates the power management process for accelerometer variance tracking in accordance with an embodiment.

DETAILED DESCRIPTION

0012The present invention relates generally to monitoring devices and more specifically to classifying activities utilizing such devices. The following description is presented to enable one of ordinary skill in the art to make and use the invention and is provided in the context of a patent application and its requirements. Various modifications to the preferred embodiments and the generic principles and features described herein will be readily apparent to those skilled in the art. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features described herein.

0013A system and method in accordance with the present invention describes an inertial activity classification device, comprising of any combination of sensors, such as an accelerometer, gyroscope, temperature sensor, pressure sensor, magnetometer, or microphone, and a method for classifying an activity based on features derived from inertial data, and the ability to continually report an activity derived from physical activity.

0014In an embodiment the system and method improves the long-term viability of a multi-sensor inertial activity classification device by offering a decision scheme for power management, allowing the usage of relatively higher power devices, such as a gyroscope, for more accurate sensing and activity classification without the associated reduction in battery life.

0015A system and method in accordance with an embodiment also provides an improvement in the accuracy of a device by being able to identify walking, jogging, and running phases of motion, for the purpose of increasing the relevance of the reported data for calorie counting purposes, and also to reject false positive “steps” in the counted number of steps. The identification of activities allows the device to more accurately track steps when walking, running, etc., while rejecting “steps” recorded during shaking, driving, or other inactive movements. Accordingly motion tracking information, such as number of steps to be reported at an appropriate time is allowed based on an activity.

0016To describe the features of the present invention in more detail refer now to the following description in conjunction with the accompanying Figures.

0017<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram of a system <b>10</b> that utilizes an activity classification algorithm <b>100</b> in accordance with an embodiment. The system <b>10</b> includes a device <b>11</b>. The device <b>11</b> may include any type of mobile device including but not limited to, a cell phone, a tablet PC, or other type of portable electronic device. The device <b>11</b> receives input data from a plurality of sensors such as one or more magnetometers <b>12</b>, one or more accelerometers <b>14</b> and one or more gyroscopes <b>16</b>. Although the sensors <b>12</b>, <b>14</b> and <b>16</b> are shown external to the device <b>11</b>, one of ordinary skill in the art readily recognizes that those sensors could be internal to the device <b>11</b> and that would be within the spirit and scope of the present invention.

0018The device <b>11</b> includes memory <b>18</b> for storing data. In the described embodiment, sensors refer to magnetometer <b>12</b>, accelerometer <b>14</b> or gyroscope <b>16</b>. Data <b>20</b> stores data from the one or more sensors. Memory <b>18</b> also includes algorithm <b>100</b> in accordance with the present invention. Algorithm <b>100</b> may include software code for, but not limited to activity classification and sensor fusion. In the described embodiments, sensor fusion may include consolidating data from magnetometer <b>12</b>, accelerometer <b>14</b> and gyroscope <b>16</b> and generating sensor fusion data. A processor <b>24</b> executes the algorithm <b>100</b> which operates on the data received from the sensors . The processor <b>24</b> provides the sensor fusion data <b>19</b>. The sensor fusion data <b>19</b> provides orientation information of the device.

0019A system that utilizes an activity classification algorithm <b>100</b> in accordance with the present invention can take the form of an entirely hardware implementation, an entirely software implementation, or an implementation containing both hardware and software elements. In one implementation, this activity classification algorithm is implemented in software, which includes, but is not limited to, application software, firmware, resident software, microcode, etc.

0020Furthermore, the activity classification algorithm <b>100</b> can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

0021The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. Examples of a computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk, and an optical disk. Current examples of optical disks include DVD, compact disk-read-only memory (CD-ROM), and compact disk-read/write (CD-R/W). To describe the features of the present invention in more detail, refer now to the following description in conjunction with the accompanying Figures.

0022In an embodiment an inertial activity classification device includes any combination of sensors, such as an accelerometer, gyroscope, temperature sensor, pressure sensor, magnetometer, or microphone, and an algorithm for classifying an activity based on features derived from inertial data, and the ability to continually report an activity derived from physical activity. A system in accordance with an embodiment relies on multiple sensors and an activity classification algorithm in order to improve accuracy of the activity recognition results. The inclusion of a sensor has the effect of improving accuracy, stability, or algorithm performance.

0023<figref idref="DRAWINGS">FIG. 2A</figref> is a diagram of an activity classification system <b>100</b> wherein more than one processor is operating in accordance with an embodiment. In an embodiment, components include one or more sensors <b>102</b>, raw data <b>104</b>, the activity classification algorithm <b>100</b>, and one or more processors <b>108</b><i>a</i>-<b>108</b><i>n. </i>The set of sensors could be any combination of a 1, 2, or 3-axis accelerometer, a 1, 2, or 3-axis gyroscope, a 1, 2, or 3-axis magnetometer, a pressure sensor, a temperature sensor, and a microphone.

0024Such an activity could also be utilized in an embodiment where there are one or more sensors worn by a user on various parts of the body. <figref idref="DRAWINGS">FIG. 2B</figref> is a diagram of an activity system <b>150</b> wherein one or more sensors are utilized in a wireless environment in accordance with an embodiment.

0025As is seen, the activity stem includes two sensor systems <b>160</b><i>a </i>and <b>160</b><i>b </i>that communicate wirelessly to a host system <b>165</b>. Host system <b>165</b> can be connected wirelessly to any number of sensor systems. In an example, sensor system <b>160</b><i>a </i>could be attached to an arm of a user and sensor system <b>160</b><i>b </i>could be could be coupled to a leg of the user. Each of the sensor systems <b>160</b><i>a </i>and <b>160</b><i>b </i>include a sensor <b>170</b><i>a </i>and <b>170</b><i>b </i>respectively for receiving activity data. Each of the sensors <b>170</b><i>a </i>and <b>170</b><i>b </i>is coupled to a processor <b>172</b><i>a </i>and <b>172</b><i>b </i>respectively which receives the activity data from the sensor. Each of the processors <b>172</b><i>a </i>and <b>172</b><i>b </i>then provides the data to its respective radio <b>174</b><i>a </i>and <b>174</b><i>b. </i>The radios <b>174</b><i>a </i>and <b>174</b><i>b </i>transmit the data wirelessly to the host system <b>165</b>. The host system <b>165</b> comprises a wireless receiver <b>176</b> which receives the data from radios <b>174</b><i>a </i>and <b>174</b><i>b </i>and a processor <b>178</b> which executes the activity classification algorithm <b>100</b>′.

0026In another embodiment, sensor systems and the host system can be connected by wires. In another embodiment, the host system can reside on a sensor system and connect to other sensor systems.

0027<figref idref="DRAWINGS">FIG. 3</figref> illustrates an activity classification device <b>200</b> to be worn as if it were a wrist watch. Activity classification device <b>200</b> is strapped to a user and is designed for portable media devices. In operation, a wearable module's microcontroller within the device <b>200</b> collects data from the inertial sensors, such as a gyroscope and or accelerometer within the device <b>200</b>, calculating a set of features from each data. In an implementation, included are a signal mean, standard deviation, energy, and entropy, as calculated from signal sources; raw accelerometer magnitude, raw gyroscope magnitude, and calculated signals resulting from a set of libraries that provide data for motion processing, such as: linear world-frame acceleration (X, Y, Z), linear body-frame acceleration (X, Y, Z), angular world frame rate (X, Y, Z), angular body frame rate (X, Y, Z), utilizing the following calculators.

0000<br />Accelerometer/Gyro Magnitude=√{square root over (<i>X</i>²<i>+Y</i>²<i>+Z</i>²)}  Equation 1

0028Mean and standard deviation are simple measures of the average value, and typical variance of the source signal. Energy and Entropy are the result of measures of a frequency space transformation, using an FFT. Energy is a measure of the “energy” of a signal which results from frequency information (i.e. steady state or DC signals have energy of 0), while Entropy (also called “spectral entropy”) is a measure of how spread apart the signal frequencies are (lower value indicate fewer frequencies, while higher values indicate multiple frequencies present in the source signal. For example, these features can be calculated over a 10 second data collection period (approximately 1024 data points @ 100 Hz data collection rate), using the following equations:

0000<maths id="MATH-US-00001" num="00001"><math overflow="scroll"><mtable><mtr><mtd><mrow><mi>Mean</mi><mo>=</mo><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></munderover><mo></mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo></mo><mfrac><mrow><mi>X</mi><mo></mo><mrow><mo>(</mo><mi>i</mi><mo>)</mo></mrow></mrow><mi>n</mi></mfrac></mrow></mrow></mtd><mtd><mrow><mi>Equation</mi><mo></mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo></mo><mn>2</mn></mrow></mtd></mtr></mtable></math></maths>

0000<br />Standard Deviation=√{square root over (Σ_i=1ⁿ(<i>X</i>(<i>i</i>)−<i>X</i>_mean)²/(<i>n−</i>1))}{square root over (Σ_i=1ⁿ(<i>X</i>(<i>i</i>)−<i>X</i>_mean)²/(<i>n−</i>1))}  Equation 3

0000<br />Energy=Σ_i=1ⁿ<i>FFT</i>_x(<i>i</i>)²<i>/n </i>  Equation 4

0000<br />Entropy=Σ_i=1ⁿ(<i>FFT</i>_x(<i>i</i>))(log(<i>FFT</i>_x(<i>i</i>))/<i>n </i>  Equation 5

0029Where n=number of samples within the collection period, and FFTx contains the elements of a Fast Fourier Transform of the original signal.

0030Referring back to <figref idref="DRAWINGS">FIG. 2</figref>, in an embodiment, signal features calculated within the device, or raw data <b>104</b> can be transmitted to a host device <b>108</b><i>a</i>-<b>108</b><i>n </i>or other microcontroller based system, where the features can be calculated in real time. The processing power required is largely dominated by the ability to take an FFT of the signal. While not required for the present invention, the inclusion of the FFT based features improve the accuracy of the system and improve the performance of the pedometer feature.

0031Additional features may be added, based on the FFT results, or other transformations, i.e. Wavelet Transforms (used commonly in image compression technology), run length-encoding, differentiation, integration, or a combination of those techniques.

0032Using the features, activities can be differentiated from one another for analyzing whether the user is wearing the device (or not), and detecting what activity the user is engaged in. For example, when the activity transitions between standing, to walking, to running, the values of features increase, and a simple rule can be created for a feature. If an accelerometer standard deviation is above some threshold A, then the user is assumed to be running; otherwise, if the accelerometer standard deviation is above threshold B, the user is assumed to be walking, else the user is assumed to be standing, and therefore no “steps” should be recorded, as they likely occur from another type of activity. With a collection of independent features for every 10 seconds of data collected, numerous rules can be created which identify an activity into one of those categories.

0033<figref idref="DRAWINGS">FIG. 4</figref> illustrates an example of an algorithm for a decision tree <b>400</b> generation that is utilized to identify a plurality of activities. To improve the performance and accuracy of the approach, modern machine learning and statistical processing techniques can be applied. For example, the algorithm can be applied to previously classified sets of data collected during “walking ,” “running,” “stair climbing,” or a number of other activities that would not be associated with a step, such as “waving ,” “shaking ,” “sitting ,” “driving a car,” etc. There are published algorithms such as C4.5 which uses a technique of choosing features to classify the data based on maximizing the information entropy gained by using a specific feature.

0034Using the C4.5, a decision tree <b>400</b> can be generated to classify the current sensed features, calculated from the inertial sensor values, with a high degree of accuracy (typically above 95%). The decision tree <b>400</b> may be calculated on the device itself, or as a post or pre-process in which the decision tree <b>400</b> is classified based on previously collected data, then programmed or “hard coded” into the device itself. Ideally a user will have been “trained” into the classifier set, to maximize the performance and accuracy of user activity classification, although with a sufficient sample of similar users, “training” is not required. Including a multitude of users within a training set creates a generalized classifier, which improves performance of un-trained users.

0035Alternate algorithms exist (ID3, See5, K-means clustering, HMM's, etc.) which could be applied instead of the C4.5 technique to generate the decision tree. Clustering techniques involve computing the likelihood a current set of features matches a specific activity by comparing the relative “distance” between the feature groups. Such techniques have been successfully applied and tested in accordance with an embodiment.

0036In an embodiment the system and method improves the long-term viability of a multi-sensor inertial activity classification device by offering a decision scheme for power management, allowing the usage of relatively higher power devices, such as a gyroscope, for more accurate sensing and activity classification without the associated reduction in battery life.

0037<figref idref="DRAWINGS">FIG. 5</figref> illustrates the power management process <b>500</b> for low power mode sensor variance tracking in accordance with an embodiment. In an embodiment, the power management process <b>500</b> resides within the activity classification algorithm. The power management process <b>500</b> utilizes the set of sensors and their derived features that can be utilized to correctly classify activities with a high degree of accuracy. The power management process <b>500</b> can then be put in a low power mode until the low power mode sensors measure an activity transition. In the described embodiments, a low power mode refers to reducing the sampling data rate of sensors or placing the device in standby or shutting off one or more sensors that are being fed into the algorithm. Typically, low power is accomplished by placing the gyroscope in a standby mode.

0038Sensor output magnitude, linear world frame acceleration, linear body acceleration, angular world frame rate, and angular body frame rate, are used for the activity classification step.

0039Referring back to <figref idref="DRAWINGS">FIG. 5</figref>, first the activity classification device operates in “Full Power Mode” where a gyroscope, an accelerometer, and a magnetometer (if present) collect data, and a microcontroller processes all relevant features. The device then performs activity classification based on clustering, decision tree, or other some other model with confidence, via step <b>502</b> where the activity is recognized.

0040Second, when the confidence level of the reported activity is sufficiently high, the device can store the current accelerometer values in <b>506</b> and then shut off the gyroscope, v, to enter a “Low Power Mode.” Confidence may be computed, via step <b>502</b> as the percentage of feature sets/samples identified as the current activity during a predetermined time period such as the last 10 seconds, or by distance from the current sample to the appropriate generated cluster, or other method.

0041Third, when in “Low Power Mode” the accelerometer is still active and collecting data. The values and associated features stored in step <b>506</b> are compared as to whether transitioning to a new activity, by comparing the variance between the current data and stored values when sensor fusion is turned off, via step <b>508</b>.

0042Fourth, the value of the variance of accelerometer derived features is evaluated. If the variance is sufficiently large, the device will transition back to “full power mode”, to classify the current (assumed to be different) activity, via step <b>510</b>. It is possible, and perhaps desirable, to also trigger the reclassification if the activity transitions to sub-activities (slower running/jogging as compared to sprint running), so levels can be chosen which allow for the activity transitions while maintaining a reasonable range of values where the sensor will remain off, thus saving power.

0043The power management process is the equivalent of storing the low power mode features of an activity e.g. “running,” detecting a change in the low power mode features which would indicate a new activity. By comparing each feature independently, the likelihood of detecting a new trained activity with the power management process is high.

0044By utilizing the power management process, the device will save power according to the amount of time a specific activity is maintained. “Sitting,” “Sleeping,” “Standing,” “Driving,” or “Walking” can be assumed to last for several hours throughout the day, while short periods of heavy activity, such as: “Making dinner” or “Getting into the car” would be expected to require the gyroscope to be “on” for correct identification, but would last for much shorter amounts of time.

0045When the activity classification process is applied to a device, it allows a device carried by the user (attached to wrist, foot, hip, chest, etc., through various attachment techniques) to track different phases of activity, from stopped/standing to a slow walk, fast walk, jog, run, sprint, etc. and report them via an on-screen display or other indicator, as well as report analytically the calories burned for each of those activities.

0046Once the activity has successfully been classified, an application may run on the data, analyzing the step frequency and event. In an embodiment, the application may choose to increment a step counter when a user is classified as “walking” or “running,” but may reject a user standing and shaking the pedometer up and down, or a device jostling loosely in the back of a vehicle, for example.

0047The application of machine learning and activity classification allows an activity classification device to utilize a set of sensors, including accelerometers, gyroscopes, magnetometers, pressure sensors, temperature sensors, microphones, etc. by implementing a power management methodology based on the measured feature-space variance of any sensor, or combination of sensors. With the activity classification device, a lower power sensor (or combination of sensors) can be used to detected transitions between activities, while a larger set of sensors may be used for a short period of time to classify the activity.

0048The application of machine learning and activity classification allow a typical digital pedometer to increase its level of accuracy and improve functionality, particularly for long-term data collection. An implementation in accordance with an embodiment requires certain training steps to be performed with the device, in recognizing the user (or a representative set of users) to analyze walking, running, standing, etc. For best results, a training step should be included to allow the end user to “classify” their own activities, so that their specific motion features might be correctly identified.

0049Although the present invention has been described in accordance with the embodiments shown, one of ordinary skill in the art will readily recognize that there could be variations to the embodiments and those variations would be within the spirit and scope of the present invention. Accordingly, many modifications may be made by one of ordinary skill in the art without departing from the spirit and scope of the present invention.