Analysis of the behaviour of a subject

The present invention relates to a method and an apparatus for analyzing a subject's behavior using artificial intelligence, and in particular, referring to an analysis of the subject's non-verbal visual behavior using an artificial neural network (to this). It relates to methods and devices for analyzing subject behavior (but not limited to).

There is great interest in the technique of performing psychological profiling of a subject in order to obtain information about the subject who is reluctant to vomit a fact or who cannot really do so. A typical example is the field of detecting falsehood.

Various false detectors (so-called "lie detectors") have been used for this purpose. While these devices detect some clues that suggest falsehood, they are generally not suitable for detecting other mental, behavioral and / or physical conditions. U.S. Pat. No. 5,507,291 discloses a method for remotely measuring information about a subject's emotional state and wirelessly transmitting waveform energy to a distant subject. In this method, the waveform energy emitted from the subject is detected, and the measured value is automatically analyzed by comparing it with the reference value to acquire information on the emotional state of the subject.

One of the well-known devices is the polygraph, which was first developed in the 1920s. In the United States, the device may be used for job interviews and insurance claims, and is also permitted in court in New Mexico. A polygraph usually records heart rate, respiratory rate and sweating rate; a tube is fastened around the chest to monitor respiratory rate; blood pressure is wrapped around the crotch to monitor heart rate. To monitor the degree of sweating; measure the electrical skin response of the fingertips. The operator then checks the chart paper during the examination to correlate the physical response with a particular phrase or question. It will be apparent to those skilled in the art that such tests are invasive to humans.

Although attempts have been made to automate the analysis of test records, test reliability depends on the skill level of the operator performing the test and the expertise in analyzing the test records. Furthermore, the test results depend on the subject, and some people can control their own response to distort the test results.

The polygraph test relies on only three measurements (channels) that indicate stress. Even an innocent person can be stressed or confused when, for example, being told to be guilty. Emotions such as irritation, embarrassment, fear, anger, or innocent sin can also cause a stress response. Therefore, it is possible that the response from an innocent subject may be mistakenly determined to be false. Another problem is that a dishonest person (or even an honest person) who knows some simple tricks can control the stress response and deceive the polygraph operator. As a result, this instrument will damage the innocent.

Another false detector is a voice analyzer, which attempts to detect falsehoods in real time by recording and measuring changes in the fundamental frequency of the voice. When a person is stressed, the amount of blood in the vocal cords decreases. When nerves are involuntarily interfered, the vocal cords emit distorted sound waves of slightly different frequencies. Again, because it is a stress-related measurement, the subject may be able to control his or her stress response, especially if he or she knows that he or she is being monitored. The stress measurements obtained by the above device are applicable only when assessing the subject's honesty.

It is known that nonverbal visual behavior can suggest a subject's psychological profile. Here, the nonverbal "visual" behavior means the external qualities of the subject observed by the bystander, for example, the movement of the subject. It targets non-visual behaviors such as heartbeats and non-visual behaviors such as voice.

Charles Darwin could be said to have been the first scientist to systematically study patterns of nonverbal, non-visual behavior in humans and animals. In classical studies, it was proposed by Efron to analyze nonverbal behavior by the "frame by frame" method (Efron, D., Gesture). and Environment, 1941, King's Crown, New York). In the "frame-by-frame" method, a film or video of a subject is recorded, and "channel" data from each frame of the film is observed and manually coded by a human judge. A channel is a well-known term in the field of psychoanalysis and means one aspect of the overall behavior exhibited by a subject. A channel is, for example, eye contact, contemplation, or body movement. The coding consists, for example, simply recording whether or not a particular action was taken, measuring the channel (over several frames), or the subjective opinion of the judge.

In general, many channels are selected for one test, so one multi-frame-by-frame method must be performed for each channel. This is done by one judge taking the video multiple times, or by having one judge take charge of one or two channels and many judges watching the video.

Channel data is then categorized by a fixed amount of time, or by an hour according to an event (eg, an answer to a question). The classified data is then analyzed by a person who has experience with nonverbal behavior and analysis experience with respect to a particular channel set. It is desirable to detect patterns in the data that indicate some mental, behavioral or physical condition.

As described below, there are many challenges and drawbacks to using the "frame-by-frame" method manually. First, in the "frame-by-frame" method, the judge must manually code each frame for each channel. The time required increases, at least depending on the number of channels, and more time is required for analysis. At best, the final result will be hours after the event. It will often take days or even weeks. Second, hiring many highly proficient judges and experienced analysts is expensive. Third, in certain psychological studies, the relative importance of different channels is unknown. Manual research will limit the number of channels that are actually collected and analyzed in terms of cost. Channel selection will be based, for example, on the investigator's previous knowledge, rumors, or literature. Several channels may be equated in importance, or one channel may be overemphasized by the investigator.

In addition to being subjective, human judges can make mistakes by encoding variegated and inconsistent channel information, for example due to fatigue. Different investigators will choose different channels, encodings or parsing methods and will therefore produce different results for the same set of frames. That is, the analysis is essentially arbitrary.

Fourth, when analyzing a large number of channels for a large number of frames, the analysis becomes particularly complicated because the data becomes high-dimensional. As the number of channels increases, it is more likely that you will miss important patterns in your data. Humans can only focus on a limited number, so important channels can be inadvertently ignored.

As is clear from the above, the conventional frame-by-frame method of channel coding and subsequent analysis is time-consuming, costly, complex, and error-prone.

<p> Thus, improved methods and devices have long been sought to analyze behaviors such as falsehood. The present invention is intended to solve the problems and drawbacks so far in response to such a request. To eliminate doubt, "nonverbal" The word "behaviour" refers to all actions exhibited by a subject, except for meaningful language spoken by the subject. "Nonverbal visual behavior" means the subject's external qualities that can be seen by bystanders, such as the subject's movements. This contrasts with "nonverbal, nonvisual behavior" such as heartbeat and tone of voice.</p>

<p> According to the present invention, first, it is an automated method for analyzing the behavior of a subject: The step of making one or more measurements or observations on a subject; The process of encoding the measurement or observation into multiple channels; and The process of analyzing the channel using automatic classification technology and outputting information related to the psychological state of the subject; Methods are provided that include.</p><p> The method of the present invention has many advantages. The method of the present invention is automated, extremely rapid, and can output subject analysis and information in real time or near real time. "Automation" means that the method is performed by a machine and without human intervention. The methods of the present invention are extremely economical, objective, and consistently reliable. Importantly, the methods of the invention do not overestimate and overlook important behavioral information contained (even temporarily) within a given channel. It is possible to accept and analyze the channel of. This ability especially improves accuracy and reliability. For example, a subject with false behavior may be able to control channels related to some behaviors, but it is quite possible that they can control all channels that are consistent throughout a period of time. Because you can't get it.</p><p> In a preferred embodiment, the automatic classification technique comprises artificial intelligence. However, other methods using the device, such as statistical classification methods, can also be used, for example. Such statistical methods are purely mathematical and do not include artificial intelligence. The classification method can be performed using a computer or other microprocessor. By using the classification method by the automatic device, the defects derived from the human judge are analyzed. Artificial neural networks, genetic algorithms, decision trees, fuzzy theories, symbolic rules, machine learning, and other methodologies may be used as artificial intelligence to analyze channels. These can be used in a complicated manner or in combination of several.</p><p> According to the present invention, the non-verbal behavior of a subject can be measured, which is to measure the non-verbal non-visual behavior of the subject. That is, the movement of the subject can be monitored. Such observations have the advantage of being visual in nature and therefore non-invasive to the subject. Another advantage is that a lot of important information is presented by the subject's movements. Another advantage is the ability to encode multiple channels from the subject's movements. Yet another advantage is the ability to encode multiple channels from a single observation set (eg, a frame sequence obtained using a single video camera).</p><p> Observation of a subject's movement consists of multiple subject images, and coding this observation (result) into multiple channels involves extracting features from those images (images). Channels include eye movements (eg, blinking, staring), facial movements (eg, up / down, left / right movements, or head tilting or facial movements), as well as the subject's hands, legs, or The movement of the torso is included. All of these exemplify the movement of the subject.</p><p> In the present invention, in addition to or instead of observing the movement of the subject, other nonverbal behavior may be measured or observed. Examples of such other measurements or observations include heart rate, sweating rate, respiratory rate, brain waveforms, thermal imaging, detecting tremors in the chair in which the subject is sitting, and paralinguistic measurements (Zuckerman). , M., Driver, RE, Telling Lies: verbal and nonverbal correlates of deception, in Mathematical integrations of nonverbal behavior, Eds: AW Siegmans S. Feldstein, 1985). In selecting measurement and observation items, as well as channels, subjects must control their response (as opposed to prior art such as the polygraphs described above) to prevent the technique from being "deceived". It doesn't become.</p><p> In addition to, or instead of, measuring or observing non-verbal behavior, linguistic behavior (ie, utterance of meaningful language) can also be measured or observed. Channels are analyzed to determine if a subject's behavior is false. This is probably the most difficult psychological behavior to detect. However, the present invention can detect false behavior with excellent accuracy.</p><p> In another area of application, channel analysis can also be performed to measure a subject's emotional, behavioral and / or physical condition, such as stress, guilt, well-being, fear, etc. Emotional traits such as self-confidence; introversion, extroversion, cooperation, integrity, personality to experience anything, emotionally motivated personality, arrogant personality, compliant personality traits, and even really pain Medical characteristics such as having or having a mental illness can be mentioned.</p><p> According to the second aspect of the present invention, it is a device for automatically analyzing the behavior of a subject: Detection means for making one or more measurements or observations on a subject; Encoding means for encoding the results of the above measurements or observations into multiple channels; and An automatic classification means that analyzes the channel and can output information related to the psychological state of the subject; Equipment including is provided.</p><p> The detection means measures or observes the nonverbal behavior of the subject, including measuring or observing the nonverbal visual behavior of the subject. The movement of the subject is observed by this detection means. Detection means also provide multiple images of the subject; and coding means include feature extraction means for extracting features extracted from those images. The coding means encodes the observations into one or more channels involved in eye movement. The coding means encodes the observations into one or more channels involved in facial movement. The coding means encodes the observations into one or more channels associated with the subject's hand, leg or torso movements. Taken together, in the present invention, the detection means encodes measurements or observations into one or more channels involved in nonverbal visual behavior, nonverbal nonvisual behavior and / or verbal behavior. To do.</p><p> The detection means should have one or more cameras or other image capture means. Thus, for example, coding means can encode the measurement or observation of a subject's nonverbal non-visual behavior into one or more channels. Further, the detecting means may have a microphone.</p><p> As a preferred embodiment, the automatic machine classification means has an artificial intelligence means, and the artificial intelligence means includes an artificial neural network (artificial neural network), a gene algorithm, a decision tree, fuzzy theory, symbol rules, machine learning, and the like. And other intellectual base systems. A plurality of these may be used and / or used in combination. Other forms of automatic classification means, such as statistical classification means, can also be used. Statistical classification means include a microprocessor and are configured to make use of statistical methods, for example by running software. The artificial intelligence means is configured to be able to determine whether or not the subject's behavior is false.</p><p> In addition, artificial intelligence means can be configured to determine a subject's emotional, behavioral and / or physical condition, such as stress, guilt, happiness, etc. Emotional traits such as fear and self-confidence; introverted, extroverted, emphatic, honesty, personality to experience anything, emotionally motivated, arrogant, compliant, and more. Has medical characteristics such as being really painful or having a mental illness. Artificial intelligence means are configured to be able to output information about a subject's personality traits.</p>

Hereinafter, examples of the method and apparatus of the present invention will be described with reference to the accompanying drawings. The present invention utilizes artificial intelligence to analyze channels related to subject behavior. In a preferred embodiment, an artificial neural network is used, but is not limited thereto.

A neural network is a large-scale parallel network consisting of multiple simple processors, or "neurons," and has the ability to store knowledge gained from experiments. Knowledge is acquired by training, not programming, and stored as connectivity between neurons in the network (eg, hassoun, MH, See Fundamentals of Artificial Networks, 1995, MIT Press).

For systems based on such machines, the relative importance of channels does not matter. The reason is that the "neural network" accepts all the channels provided and decides for itself which channels are important, insignificant, or extra. In the initial training, virtually all channels have the same importance, but as the training progresses, the machine decides its own level of importance for each channel. Generally speaking, the more humans and channels employed in training, the more accurate the results after using a neural network to evaluate humans for the first time. Even with higher dimensional inputs (ie, with multiple channels), the neural network can detect patterns in the data about the subject's behavior. Channels are automatically weighted appropriately during training. Once trained, newly provided data patterns on subject behavior are quickly categorized and important patterns are not overlooked.

FIG. 1 shows a general embodiment of the device of the present invention for analyzing the behavior of a subject. The device of the present invention Detection means for making one or more measurements or observations on a subject 12; Coding means 14, 16, 18, 20; and for encoding the measurement or observation result into multiple channels. Includes artificial intelligence means 22 configured to analyze the channels and provide information about the subject's psychological state.

FIG. 2 shows a specific embodiment of the device of the invention for analyzing nonverbal visual behavior. In this particular embodiment, the detection means 12 is a camera (eg, a video camera), i.e., a group of recorded frames that provide an image of the subject as multiple frames.

The "object locator" 14 finds an object on the image (eg, face, eyes, nose, etc.). "The pattern detector (pattern detector) 16 detects patterns within the area of interest (eg, closed eyes, narrowed eyes, or left-facing eyeballs). In this example, these objects. The detector and pattern detection use a trained neural network 14a / 16a. The channel coding device (channel coder) 18 uses the data from the pattern detector 16 and the target detector to generate channel data. Automatically coded. The classified (grouped channel coder) 20 collates the data over a period of time or variable time to obtain an evaluated (scaled) result. In the particular embodiment illustrated, the decision device (classifier) 22 (which has a neural network 22a) uses data from the post-classification channel coding device 20 to make a final decision regarding the subject's psychological state. I do.

FIG. 3 shows in more detail the hierarchy of coding devices 14, 16, 18, 20 and artificial intelligence means 22. The same reference numbers used in FIG. 2 are used to represent components such as the body detector 18, the post-classification channel coding device 20, and the determination device 22. In FIG. 3, these numbers indicate the overall position of the above elements. In FIG. 3, the elements common to the constituent elements are arranged vertically side by side above each number. That is, for example, the target detector 14 consists of a number of individual elements, such as the face detector 24, the eye detector 26, and the eyebrows detector 28, each of which is a face detector. It has a neural network 26a for eyebrows and a neural network 28a for eyebrows detectors. The elements shown in FIG. 3 are not all by this, and other qualities may be adopted or other channels may be encoded. The element 30 represented by the broken line in FIG. 3 intends to indicate such another processing element. The channel from the post-classification channel coding device 20 is sent to the guilty determination device 32 and the false determination device 34. Neural networks 32a and 34a are connected to each determination device. Behaviors related to guilt may appear similar to behaviors related to falsehood. Since guilty-related behavior can lead to the conclusion of "false positives" for false-related behavior, the above similarity has hampered the accuracy of traditional false systems. However, based on a number of channels (as described herein), the overall difference between guilty and false behavior is clear. By providing a judgment device specifically trained to recognize guilty behavior and a judgment device 34 specifically trained to recognize false behavior, guilty behavior is separated from false behavior. To. By the way, it is possible that a subject is guilty and false.

As will be apparent to those skilled in the art, neural networks are used in three areas in this embodiment. That is, it is used for target detection, target pattern recognition, and determination of channel data after classification. By using the neural network 14a for target detection and the neural network 16a for target pattern recognition, channel data is generated for each frame, and therefore channel data classified for a large number of frames is generated. The neural network 22a for the determination device makes a determination regarding the mental, behavioral and / or physical condition of the subject using the classified channel data.

By using neural networks, coding and analysis of coded data becomes extremely powerful, adaptable and fast. However, those skilled in the art will come up with other techniques to perform these functions, and the use of neural networks (or other artificial intelligence techniques) does not limit the invention. Further examples will be shown below, but the present invention is not limited thereto.

<u style="single">Detailed description of an embodiment of the apparatus</u> For nonverbal visual behavior, the most probable detection means 12 is a video camera. The frame sequence data enters the control detector.

<u style="single">Target detector</u> The target detector extracts a predetermined feature (characteristic) from the frame sequence. One of the important objects to detect is the face. Once the face is found, it is possible to assume facial features and the location of other parts of the body.

Detecting a face in an image is a complex task. The image contains a large amount of data to process. Faces can appear in different places in the image, in different scales and orders. Subjects may be wearing make-up, glasses, mustaches and mustaches, or hats and hairstyles that hide their faces. The intensity and direction of the light may be different, or the background may be complicated. In addition, the main target must be quickly found from the requirement that channels must be coded and analyzed in real time. There are various methods for detecting a face.

The current configuration uses two learning-capable neural networks to recognize the difference between facial and non-face images. One network functions to compress pixel data and the other network functions to determine the compressed data. Manually collected image sets (sets of vectors for the face and sets of vectors for the non-face) are used to train the network. With training, those networks can determine from a frame whether a particular area is "face" or "non-face".

Other objects, such as the nose, mouth, eyebrows, hands, etc., can be found in a similar manner. The greater the number of targets, the stronger the system. For example, the face detection operation can be fine-tuned from the knowledge about the eyes. Each object helps improve the accuracy of detecting other objects. For example, there is a relatively clear ratio between each facial feature, and some simple rules (eg, the eyes are above the nose) are available. If one object is detected correctly, the results from the other object will give a reasonable response. Neural networks are tolerant of small deviations in patterns. With many channels based on many targets, one or two channel errors have little effect.

<u style="single">Pattern detector</u> The pattern detector (pattern detector) recognizes the detected pattern of the target. For example, a network trained to specifically recognize "closed eyes" would be "1" for closed eyes and "-1" for open eyes. Thus, the eye object is available in compressed form and is used as an input to each eye pattern detector, so each network has the advantage of being relatively small and therefore quick to process. Has.

The pattern detector is trained with a dataset showing a large amount of pattern variation for a particular subject. Here, in the examples described, a multiple-choice coding method is used for each target pattern. This method has the advantage of giving uncomplicated code and not requiring the coding device to be trained and the subject to be face-based. But other more elaborate methods, such as Ekman's Facial Action Coding System (FACS) [Ekman, P., Friesen, WV (1978) Consulting Psychologists Press, Palo Alto, CA, US] also belongs to the scope of the present invention. FACS is a thorough coding system related to the movement of facial muscles and requires training of the coding device.

<u style="single">Extraction of channel data and channel data after classification</u> The next step is to extract the channel information for each frame. Such channels depend on one or more target position or pattern detectors. Further information can be obtained by statistically processing the grouped channel data for each channel. Channel data is collected by the dual digital or analog method. A channel indicates what the current frame is, or the relationship between the current frame and one or more previous frames. The signal on the channel is essentially two-dimensional, eg, "0" or "1", or "analog", with a range of numbers (eg, between "-1" and "1"). Has a real number). Also, a single channel can be represented by a dual or analog collection from data (eg, small pixel area). For video images, there are three types of channel extraction, referred to here as "basic", "simple" and "complex" channels.

<u style="single">Basic channel</u> This type of channel depends on the relative positional relationship of the object, the change in the position of the object, or the basic function that acts on the pixel of the object. For example, the channel of "head movement" depends on the position of the face in each frame. Positional measurements rely on common measurements, as the distance between the subject and the camera varies from video to video and can vary within the same video. Use the width and length of the face to determine relative distance or movement. Numerical values are standardized (normalized) to numbers between "-1" and "1".

For simple distances between objects (eg hands and faces), "-1" is for the selected minimum distance (eg, face width 0) and "1" is the selected maximum distance. (For example, the width of the face is 10). Covering the mouth or nose or near the face with hands is a signal of "concealment".

Alternatively, the distance between the objects in both the vertical and horizontal directions may be measured. A negative vertical number indicates that the object "a" is above the object "b", and a positive value indicates that the object "b" is above the object "a". Become. A negative number can also indicate that it has moved in one direction, and a positive number can indicate that it has moved in the opposite direction.

An example of a basic function is to determine the blush of a face. Take the red component of the target face and compare it with the previous frame to see if the face is red or pale. Fit the results between "-1" and "1" according to the maximum and minimum criteria.

By classifying (grouping) the data, you can get one or more statistics for each channel. For example, the mean, median, and mode values can be obtained from valid data. In other cases, valid data may yield a single statistic (eg, the percentage of "1" that is normalized to the range "-1" to "1"). The data may be encoded using other conversion methods.

<u style="single">Simple channel</u> This type of channel uses a pattern detector to make a simple determination for each frame. For example, the channel "closed eyes" has four consequences: "closed left eye", "closed right eye", "fully or half closed left eye", "completely closed right eye". Or it depends on "half closed". Each network gives one output from "-1" to "1". The channel is the average value of the network response, or makes some theoretical judgment. When the results are grouped, one or more statistics are obtained for each channel (for example, the percentage of "1" among those standardized in the range of "-1" to "1"). Can be done.

<u style="single">Complex channel</u> This type of channel uses a pattern detector in a more complex manner. For example, blinking is considered to have occurred when the eyes were opened immediately after the eyes were closed. Use insights to prevent confusion between blinking and network errors in subjects and frames looking at the floor.

Obtaining statistics is even more complicated. This is because the minimum and maximum values that are correlated with time must be considered. In the case of a simple channel, the minimum value is the reference 0 per second and the maximum value is the reference fps per frame, where fps is the frame per second. Complex channels generally have non-standard minimum and maximum values. For example, the minimum value for blinking may be set to 0 every second, and the maximum value may simply be set to 2 every second.

<u style="single">Channel statistics after classification</u> Collect channel signals (signals) for many frames. One or more statistics for a channel, such as the percentage of "1", the mean, and the average of the pixel array, of those standardized in the range "-1" to "1". , You can get the selected maximum value and the number of "1" related to the maximum value.

Statistics can be calculated for a fixed period or an invariant period (eg, the time required for a particular answer to a question). In either case, valid channel statistics are calculated only when the amount of valid data collected exceeds a certain percentage. If an object is not found in a frame, the position of the object is "invalid", and the pattern detector depending on the object has an "invalid" result, and its position. Target-dependent channel output is "invalid". However, the channel statistics after classification (after grouping) may be "valid" or "invalid" depending on the number of results of the preceding channel data. As an example, the amount of valid data collected should be greater than 95%.

<u style="single">Matching channel statistics</u> By concatenating the statistics from each channel (in the range "-1" to "1"), a vector representing all channel statistics over a given time period is obtained. Some of the vectors are stored for training and testing. In use, other vectors that were previously "unseen" are determined. Since some behaviors are quick if they have a slow pattern, it is convenient to match channel statistics from one or more time periods in order to obtain the vector. Each channel has its own optimal measurement period.

<u style="single">Analysis of matched channel statistics</u> If a previously unknown vector is included, the neural network is trained using the matched channel statistic vector to output information related to the subject's psychological state. With training and testing, the network gives similar results for similar situations with similar groups of subjects. Increasing the number of training tests to include as many people, situations and behaviors as possible will increase the universality of the network. A combination of "fine-tuned" training and testing (eg, by male Caucasian psychopaths / non-psychopathy) can be used to achieve excellent results in certain areas. ..

The output of one or more determination device networks may provide additional input to networks for different purposes. For example, if the determination device is for determining "false", "false" and "guilty", the output from the "false" network and the "guilty" network will be "false". The performance of the latter can be improved by providing two types of additional inputs to the decision-making device network.

It is understood that the above-mentioned examples are merely examples, and there are many possibilities for those skilled in the art. Other algorithms and / or networks than those described above can also be used to extract the traits. It is advantageous to take advantage of faster processing methods (perhaps using parallel processors). Auxiliary hardware such as digital signal processors, neural network accelerators and analog processors may be used. It is also possible to access the system in various forms (eg, remote access via the Internet) and various sensors such as voice stress or tone sensors, microphones for paralinguistic or linguistic behavior, heat. Observation by the method described above using an imaging camera, load cell, microwave perturbation detector, skin resistance monitor, heart rate measuring device, blood pressure monitor, respiratory rate measuring device and / or bio-measuring device such as EEG monitor. It can be replenished or replaced.

Linguistic behavior, the meaning of the spoken language, can be the subject of artificial intelligence analysis. For example, analyze the number of negative words such as "not" and "never", and the number of owned words such as "me", "mine" and "I". The literature shows that falsehoods increase the number of negative words and decrease the number of owned words. Channels can be supplied as dual or analog via an interface such as a data logging device. By combining with suitable techniques, the interaction with the subject can be altered, and examples of such suitable techniques include the Conversational Agent (eg, The Zen of Scripting Verbots (RTM)). See virtual personalities Inc, VPI, Los Angeles, USA). Another possibility is to analyze multiple subjects at once. Information can also be collected from methods in which subjects interact with each other, instead of or in combination with methods that give external stimuli.

Other population paradigms (eg, unsupervised learning paradigms) can be used instead of artificial neural networks. Moreover, it will be apparent to those skilled in the art that various neural networks can be used at various stages of the examples and other embodiments described herein. Try to analyze different channels and record many parts of the subject's body (especially the whole body) with one or more cameras to obtain such channels.

The present invention can be used in many fields and is not limited to the detection of falsehood and guilty. For example, the present invention can also be used to evaluate whether a subject is stressed at work or to evaluate the subject's suitability for a particular task or role.

As a secret test, we asked 41 volunteers from various racial backgrounds to do a simple job and interviewed them about the story. The method of collecting data was in line with the literature on nonverbal behavior. When we arrived, we gave each volunteer an information sheet. On the sheet, along with the purpose of the survey, information that would make volunteers want to lie was written. For example, lying can be seen as a special skill, and we have stated that we sometimes have to deceive people as an expedient. In addition, they told volunteers that their behavior was being scrutinized by interviewers, their colleagues and computers. Next, I randomly assigned a number from the box and asked them to enter the interview room and read the instruction sheet.

The instruction sheet instructed the subject to remove the box from under the chair, look inside it, and remove / examine the contents of the box. The box contained a pair of glasses and a £ 10 bill. When the subject had an odd number, he was required to put the money in his pocket (Job 1). When the subject had an even number, the subject was required to put the money in the box (Job 2). The box was supposed to be returned under the chair. Subjects were informed that they would be interviewed about the location of the money and that they were to be accused of stealing the money. Subjects who put money in their pockets were instructed not to admit it. In addition, each subject answered the question in writing and was instructed to return the instruction sheet to its original location.

A few minutes later, the interviewer arrived. After a brief greeting, the interviewer asked 10 questions in turn. During the interview, the interviewer did not know if a volunteer was lying or telling the truth. Shortly after, the volunteers performed a task they hadn't done before and a second interview began. 78 videos were used in which the same questions as in the first interview were asked again in the same order.

The interview was recorded in three videos with microphones. One camera recorded the interviewee's entire body, and a third camera recorded the interviewee's movements. We obtained non-verbal channels from both images and sounds. For each interview, the camera operator entered the room just before the interviewer started the camera. After the interviewer left the room, the camera was stopped and the camera operator left the room.

In this example, only the subject's head and shoulder cameras were used in the analysis and sound was excluded from the analysis. It is also within the scope of the invention to utilize other additional information and sound recordings contained in other videos. Each video was digitized into 15 frames per second of AVI film using a low-cost capture card. Due to the large amount of data generated, the capture parameters were adopted as a compromise between quality and storage size. The video frame was 380 pixels wide and 288 pixels high. We saved 10 grayscale frames from each of the head and shoulder video images to get 780 static frames. In selecting these frames, we made sure that there was width variation between the images.

The upper part (eyes and nose) of the area with the face was manually extracted from each frame so that the line connecting the eyes became horizontal and the distance between the eyes was scaled. The obtained image was collected. The literature indicates that the eyes and nose can be used to detect faces. The captured image was 12 pixels wide by 10 pixels high and was reduced to a distance of 8 pixels between the centers of the eyes. The smaller size speeds up the examination and gives reliability in determining whether it is a face or not, while being more tolerant of positional errors. The reduced image was standardized by the histogram method, smoothed (smoothed), copied, and mirrored to obtain a face data set 1560. Images 7800 "other than the face" were randomly extracted from all frames (especially the head area). These images were also reduced, histogram standardized, and processed smoothly.

120 with standardized elements in the range "-1" to "1" by concatenating each column of the pixel matrix and using the simple function P (x) = (x * 2/255) -1 -Converted an image to an element vector. I trained one neural network to provide a low-dimensional display (image) of the face. Another network trained low-dimensional images to determine whether they were faces or not. Each of these networks was a fully-connected three-layer network, each with a bias, and was trained using a straight-forward back-propagation algorithm with bipolar sigmoid squash capability. Back-propagation networks are one of the common networks known to those of skill in the art (see, eg, Hassoun, MH above).

One of the networks is a "compression" type network, which has the same number of outputs (120) as the inputs, but a small number of neurons in the concealment layer (14). Using. The weight of the network was initialized to a small random value, and the weight was adjusted while gradually lowering as the training progressed, thereby stimulating the compressed network and restoring the image of the output level. Back propagation is for extracting regularity (significant traits) from an input vector to enable image restoration in the output layer. From the concealment layer, a low-dimensional distribution image used as facial training data was obtained.

There are two compression functions. First, by using a compressed display (image), the number of inputs is reduced and therefore the number of connections for the face detection network is reduced, resulting in various "faces" / "non-faces". A subset of "things" can be used to expedite learning and relearning to detect faces. In addition, a simple gray level is important, using preliminary processing for multiple facial images, while properly displaying changes in one facial image at a constant brightness. The traits can be extracted.

After training the compressed network, the face and non-face images were passed through the network and a vector of 14-elements was recorded on the concealment layer. Next, these vectors (which also have elements in the range "-1" to "1") were used to train and test the face detection network. This face detection network has six neurons in the concealment layer, but only one output neuron. The network was stimulated to generate "1" for faces and "-1" for non-faces.

After training the network, I was able to detect faces during video frames. The initial inspection area was defined from the moved area, edge or high contrast area. In this initial inspection area, the position of the (camera) window was changed on various scales, starting from the center and swirling outward. A vector of 120-elements was obtained for each window position by reducing the window content, standardizing the histogram, smoothing it, and performing array processing. This vector was fed to the post-trained compressed network to obtain a 14-element vector. Next, this was supplied to the face detection network. When the output exceeded the limit value (0.997 in some examples), the face was assumed to be detected and the examination was terminated. Faces of slightly different scales were tested in the detected positions and the best face was selected.

For the next frame, a new inspection area was defined based on the size of the user-defined boundary in addition to the face area and scale detected in the previous frame. Here, as in the above case, the inspection was carried out by the spiral method starting from the center. Face tracing performance can be improved by increasing the frame speed and reasonably estimating the speed and direction of the face.

If no face was found, the examination area was left in place and the next frame was examined. The lack of face detection for a few frames did not affect the overall result . This is because the channel statistics depend on the results from multiple frames. This is especially true when nonverbal visual and linguistic channels are available. If no face is found for many frames, the percentage of valid data will be small and the verdict will show a result that is considered "unknown".

When the face was found, it was possible to assume the relative size and position of the facial features and the approximate position of the torso. For example, the position of the eyes was detected as well as the head. In this example (4680), 16 × 12 eye training images were manually extracted from a highly resolved image of a face consisting of 48 × 40.

Since the face had already been detected, the fairly accurate position of each eye was also known. The position of the face was determined using a window on a 12 × 10 scale. The eye detection window, which consists of a 16x12 scale, was able to accurately detect the position of the eyes. The initial position was estimated and the inspection was carried out by a spiral method. The test was stopped when it was concluded that an eye was found (for example, when the network output was 0.99). In addition to this, three methods were selected. The first choice was chosen as the "best" eye in the inspection area, for example if the network gave an output of 0.8. This is probably an eye, but unreliable. However, since it is derived from the area including the face and is in the vicinity where the eyes are expected to be present, it is considered that the eyes may be present. In the second choice, the eye position determined by the face detector was adopted. In the third choice, eye detection was determined to be invalid. In this case, all channels that depend on the eye network are invalid.

In this embodiment, an eyebrows-targeted detector and a nose-targeted detector were also used to help determine if the eyes were effectively found. Pixels were mirrored to detect the right eye and right eyebrows and passed through a detector targeting the left eye and left eyebrows. The pattern detector was initially based on eye patterns. Three types of multiple-choice questions were asked for each subject (eye). Q1: Are your eyes open? It is wide open. It is a little wide open. It is open normally. It is closed a little. It is completely closed. It is a little thin. It is completely thin. Q2: How do you see it in the horizontal direction? Looking completely to the right. Looking a little to the right. Looking at the center horizontally. I'm looking a little to the left. Looking completely to the left. N / A (completely closed / narrowed) Q3: How do you see it in the vertical direction? Looking up completely. I'm looking up a little. Looking at the center vertically. Looking down a little. Looking down completely. N / A (completely closed / narrowed) The following answer was given to the eyes looking at the upper left. For Q1: It's open normally. For Q2: Looking completely to the left. For Q3: Looking up completely.

In this way, the eyes were classified into 17 types for the above multiple-choice questions. 17 different basic networks were trained and tested using 4680 eyes classified manually. For example, in a network for "eyes looking completely to the right", the eye looking to the right is "1", and the other eyes in the data, that is, the eyes looking slightly to the right, horizontally. The center-looking eye, slightly left-looking eye, completely left-looking eye and N / A were trained to be "-1". In this case, the network trained to recognize the right-looking eye also included the lower-right-looking eye and the upper-right-looking eye. Other networks, such as those for the "fully or slightly right-looking eye," were similarly trained to increase the strength of the next stage. Similar training was given to networks of "eyes" or non-eyes. In this example, the following channels were selected.

<tables num="1"><img file="JP4401079B2_D0001.tif" /></tables>

For movement and distance, simply set it to "1" if it exceeds a certain value, and set it to "-1" if it is smaller than that value. In classifying (grouping) data, the percentage of "1" was standardized in the range of "-1" to "1". For simple channels (eg, with eyes closed), logical decisions were made based on a number of pattern detectors. Here, too, the percentage of "1" was standardized in the range of "-1" to "1" when classifying the data. The complex channel on "blinking" utilized outputs from pattern detectors from current and previous frames, as well as insights into the frequency of blinking in humans.

Statistics were collected for all responses given by one subject and for a fixed period of 3 seconds. Each response was considered to begin 1 second before the subject began to move his mouth and 2 seconds after he stopped moving his mouth. For each channel, a sufficient amount of storage area for multiple elements was formed during the response time. One element now holds one frame of channel data. Each location in that storage area (one for each frame) was initially set to be "invalid" by "88" stored in all locations. For each new frame after processing, if the target required for the channel is found, the channel data is added to the next storage location, otherwise the next storage location is "88". Was added.

Once the address was sent to each storage location, the element statistic was calculated. Channel statistics were calculated when the amount of valid data in the storage area was greater than 95% or new frames occurred. Since the frame speed of the video is 15 frames per second, the storage area held 45 frames of data for a period of 3 seconds. Statistics are only available if 95% or more of the data is valid (ie, there are 43-45 valid frames).

Using 40% of the matched channel statistics, we trained three fully coupled, three-layer backpropagation decision networks. In addition, 20% of the matched channel statistics were used for the purpose of examining the validity, and the remaining 40% was used for the test. Judgment networks were trained to detect periods of falsehood, guilt and whether the interview was false overall. Each network had a 14-element input vector as input, eight neurons in the concealment layer, and one neuron in the output layer.

By accumulating channel statistics according to the length of the answers, we were able to make a clear decision on most of the answers and know whether the answers were true or false. Three points were considered when accumulating channel statistics for a time of 3 seconds. First, this time may be related to the case where the subject was false, in which case the desired output was set to "1". In another case, the subject was telling the truth (-1). However, this time may be on the intersection. The desired output ranges from "-1" to "1", so if 1.5 seconds is telling the truth or telling the truth, the desired output is "0". And said.

The second consideration was the "gap" between the responses. I've ignored these vectors, but another option is to set the "desired output" to zero. Other methods are also possible.

The third consideration is that the two answers may overlap, as we assumed that the answers would start one second before the subject started speaking and two seconds after the subject finished speaking. If one answer is true and the other is false, then two vectors are generated for the same time, but one vector is for truth and the other vector is for falsehood. In general, such a vector is rarely generated, so this situation is left as it is. Addressing this situation in a more rigorous manner could further improve training and test results.

In general, the results obtained were well-applicable. In early experiments with a channel set as described above and a slightly expanded channel set, the false positive detection accuracy was 75-85%.

<figref num="1">An apparatus according to the present invention is schematically shown.</figref><figref num="2">A device for analyzing nonverbal visual behavior is schematically shown.</figref><figref num="3">The components of the target detection device, the pattern detection device, the channel coding device, the classified channel coding device, and the determination device shown in FIG. 2 are schematically shown.</figref>