System and methods for dynamically injecting expression information into an animated facial mesh

CROSS-REFERENCE TO RELATED APPLICATIONS

p-0002This application claims priority from U.S. Provisional Patent Application No. 61/119,975, filed on Dec. 4, 2008, the entire disclosure of which is incorporated herein by reference.

STATEMENT REGARDING FEDERALLY SPONSORED R&D

p-0003The invention was made with Government support under Subcontract 199573 of Government Prime Contract N00173-03-C-2037, awarded by the Office of Naval Research. The Government has certain rights in the invention.

BACKGROUND

p-00041. Field of the Invention

p-0005This application relates generally to computer animation and specifically to computer animated faces.

p-00062. Description of the Related Art

p-0007Computer animations typically comprise a succession of frames. Each frame comprises a collection of “bones” and “vertices.” “Bones” are data structures associated with groups of “vertices” such that as the bones move with each frame, so do the associated vertices. In this manner, the vertices of a mesh may be drawn and displaced on-screen based on bone movements, giving the impression of a dynamically moving mesh without having to individually dictate the motion of each vertex. The vertex mesh may be arranged in the shape of a face, as is the basis for computer character animation. For example, a “jaw bone” may be used to move vertices along the mouth to simulate speech.

p-0008Imbuing computer characters with the appearance of real human emotions has traditionally involved capturing the performance of an actor using a marker-based or similar system (also referred to as “motion-capture”). The 3D data derived from the marker capture is then mapped to a model mesh on a computer. Marker-based systems capture the actor's performance by placing small illuminated markers on the actor at points of interest. Precisely placed digital cameras are then used to record the markers' movements from different angles while the markers are illuminated during the performance. Motion capture systems thus capture the movement of a real object and translate that movement to a computer-generated representation. Such captures have been used in the motion-picture and the game industries to create source data used to lend authenticity to synthetic and otherwise noticeably artificial creations.

p-0009Live-actor motion-capture may be mapped to synthetic facial models, or to models derived from other actors through “retargeting” methods. Animation retargeting is a process by which animation data from one collection of bones (known as a skeleton) is applied to another collection of bones. That is, the captured points of interest of a real-world performance will be mapped to the equivalent bones and vertices on a separately created synthetic model. This effectively causes the synthetic model to execute the actor's performance, albeit with lower fidelity to the original. Because the synthetic model to which the live performance is to be retargeted often comprises relatively fewer vertices and bones, only a handful of markers are necessary.

p-0010While adequate for most motion-picture and entertainment needs, certain applications require high fidelity representations of an actor's performance. One such application is the depiction of microexpressions, minute involuntary physical changes on an individual's face, typically as a consequence of a suppressed emotion. Microexpressions typically last a few tenths of a second, such as less than half a second. For example, a microexpression can be less than 0.25 seconds, less than 0.2 second, or less than 0.1 seconds.

p-0011The psychologist Paul Ekman helped categorize these involuntary responses with the creation of a procedure to analyze human facial expressions, known as the facial action coding system (FACS). FACS linked most of the known emotions to specific muscle movements in the face, known as Action Units, which work together to achieve a particular expression. FACS identifies 72 such units and explains how they are triggered and what combinations result in specific emotions. FACS conveniently serves as an “alphabet” of emotional expression components and has been used both by psychologists and computer scientists to structure an otherwise amorphous set of emotions.

p-0012Because they indicate a suppressed emotion, microexpressions may be used to detect the statement of falsehoods, or ulterior motives. Some individuals are trained to recognize such microexpressions as part of interrogation or interview processes. Typically, these individuals must undergo extensive training with a live trainer who is already familiar with the techniques.

SUMMARY OF THE INVENTION

p-0013In one aspect of the present application, a system is provided for training a trainee to recognize facial expressions from animations of computer-simulated faces. The system comprises a trainee interface configured to display a facial animation, and a machine-readable repository of facial expression overlays each corresponding to an emotion. The system also includes an expression injection module configured to combine one of the expression overlays into a facial animation to form a combined animation, by mapping at least a portion of the expression overlay to at least a portion of the facial animation. The expression injection module may be configured to provide the combined animation for display in the trainee interface. A trainee evaluation module is also provided, which is configured to receive from the trainee a response to the combined animation displayed in the trainee interface. A computer system is operative to execute at least one of the trainee interface, the expression injection module, and the trainee evaluation module. The computer system is also operative to access the repository.

p-0014In another aspect of the present application, a method is provided for training a trainee to recognize facial expressions from animations of computer-simulated faces. The method comprises accessing at least a portion of a facial expression overlay from a machine-readable repository of facial expression overlays each corresponding to an emotion. The accessed expression overlay is combined into a facial animation to form a combined animation by mapping at least a portion of the accessed expression overlay to at least a portion of the facial animation. The combined animation is provided for display in a trainee interface. A trainee's response to the combined animation shown in the trainee interface is received. The received response is then used to determine a performance level indicative of an extent to which the trainee has correctly interpreted the combined animation. At least a portion of the method is performed by one or more computing devices.

p-0015Another aspect of the present application provides a computer-readable storage medium having computer-executable modules, for training a trainee to recognize facial expressions from computer-displayed facial animations. The storage medium comprises an expression injection module and a trainee evaluation module. The expression injection module is operative to access at least a portion of a facial expression overlay from a machine-readable repository of facial expression overlays each corresponding to an emotion, and to combine the accessed expression overlay into a facial animation to form a combined animation, by mapping at least a portion of the accessed expression overlay to a digital mesh of the facial animation. The expression injection module is also operative to provide the combined animation for display in a trainee interface configured to display facial animations of an avatar. The trainee evaluation module is operative to receive a trainee's response to the combined animation displayed in the trainee interface.

p-0016In another aspect, the present application provides a system for combining at least a portion of a microexpression into a facial animation. The system comprises a machine-readable repository of facial microexpression overlays each corresponding to a facial microexpression. An expression injection module is configured to combine at least a portion of one of the microexpression overlays into a facial animation. A computer system is operative to execute the expression injection module and access the repository.

p-0017In yet another aspect, the present application provides a method for combining at least a portion of a microexpression into a facial animation. The method comprises combining a facial microexpression overlay into a facial animation to form a combined animation, by mapping at least a portion of the microexpression overlay to at least a portion of a digital mesh of the facial animation. The method also comprises providing the combined animation for display. The method is performed by one or more computing devices.

p-0018In still another aspect, the present application provides a method for receiving a facial animation with a combined microexpression. The method comprises sending an emotion identifier to an expression injection module and receiving a combined animation from the expression injection module. The combined animation comprises the combination of at least a portion of a facial microexpression overlay corresponding to the emotion identifier with at least a portion of a facial animation, in which the microexpression overlay is mapped to a digital mesh of the facial animation. The method is performed by one or more computing devices.

p-0019In still another aspect, the present application provides a computer-readable storage medium storing a computer-readable program code adapted to be executed by hardware to perform a method comprising: accessing a facial microexpression overlay from a machine-readable repository of facial microexpression overlays each corresponding to a facial microexpression; combining the accessed microexpression overlay into a facial animation to form a combined animation, by mapping at least a portion of the accessed microexpression overlay to at least a portion of the facial animation; and providing the combined animation for display in an interface.

p-0020The present application also provides a computer-readable storage medium having computer-executable modules, executed by hardware, for combining a facial microexpression into a facial animation. The storage medium comprises a user interface and an expression injection module. The user interface is operative to present a set of emotions and receive a selection of one of the emotions by a user. The expression injection module is operative to select a facial microexpression overlay corresponding to the user-selected emotion from a machine-readable repository of facial microexpression overlays each corresponding to an emotion. The expression injection module is also operative to combine the selected microexpression overlay into a facial animation to form a combined animation, by mapping at least a portion of the microexpression overlay to at least a portion of a digital mesh of the facial animation. The expression injection module is further operative to provide the combined animation for display substantially immediately after the user interface receives the user-selected emotion.

p-0021The present application further provides, in another aspect, a method of inserting microexpression information into a facial animation, comprising: converting an actor's performance of a facial microexpression into a digital animation, subtracting a digital image of a substantially non-expressive facial pose of the actor from the animation to create an overlay of the microexpression, and mapping the overlay to a digital mesh of a facial animation sequence. The method is performed by one or more computing devices.

p-0022In yet another aspect, the present application provides a method comprising displaying an avatar in a virtual environment accessible to a plurality of users, the avatar corresponding to a first user. An instruction from the first user is received for controlling a facial animation of the avatar in the virtual environment. An emotion associated with the instruction is determined, and a facial microexpression overlay corresponding to the determined emotion is accessed. The microexpression overlay is combined into the facial animation of the avatar to form a combined animation, by mapping the microexpression overlay to the facial animation. At least a portion of the method is performed by one or more computing devices.

p-0023For purposes of summarizing the invention and the advantages achieved over the prior art, certain objects and advantages of the invention have been described above and as further described below. Of course, it is to be understood that not necessarily all such objects or advantages may be achieved in accordance with any particular embodiment of the invention. Thus, for example, those skilled in the art will recognize that the invention may be embodied or carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other objects or advantages as may be taught or suggested herein.

p-0024All of these embodiments are intended to be within the scope of the invention herein disclosed. These and other embodiments of the present invention will become readily apparent to those skilled in the art from the following detailed description of the preferred embodiments having reference to the attached figures, the invention not being limited to any particular preferred embodiment(s) disclosed.

BRIEF DESCRIPTION OF THE DRAWINGS

p-0025The features, objects, and advantages of the disclosed embodiments will become more apparent from the detailed description set forth below when taken in conjunction with the drawings, in which like reference characters identify correspondingly throughout and wherein:

p-0026<figref idrefs="DRAWINGS">FIG. 1</figref> is functional block diagram of an embodiment of a system for creating a repository of microexpression overlays and generating animations depicting microexpressions and other features.

p-0027<figref idrefs="DRAWINGS">FIG. 2</figref> is a functional block diagram of an embodiment of a system for inserting microexpression overlays into synthetic or actor-based computer generated heads.

p-0028<figref idrefs="DRAWINGS">FIG. 3</figref> is a high-level diagram of an embodiment of a training system to teach and evaluate trainees to recognize microexpressions.

p-0029<figref idrefs="DRAWINGS">FIG. 4</figref> illustrates an embodiment of a user interface to develop animations by combining expression or microexpression overlays in conjunction with interview trees and voice sequences.

p-0030<figref idrefs="DRAWINGS">FIG. 5</figref> illustrates an embodiment of a user interface to develop animations by combining expression or microexpression overlays in conjunction with interview trees and voice sequences.

p-0031<figref idrefs="DRAWINGS">FIG. 6</figref> is a system-block diagram of an embodiment of an integrated microexpression training system.

p-0032<figref idrefs="DRAWINGS">FIG. 7</figref> is a flow diagram illustrating in greater detail certain of the steps involved in the microexpression overlay generation system of <figref idrefs="DRAWINGS">FIG. 1</figref>, in accordance with one embodiment.

p-0033<figref idrefs="DRAWINGS">FIG. 8</figref> is a flow diagram of an embodiment of a method associated with the overlay insertion system shown in <figref idrefs="DRAWINGS">FIG. 2</figref>.

p-0034<figref idrefs="DRAWINGS">FIG. 9</figref> is a flow diagram illustrating in greater detail certain of the steps involved in the training system of <figref idrefs="DRAWINGS">FIG. 3</figref>, in accordance with one embodiment.

p-0035<figref idrefs="DRAWINGS">FIG. 10</figref> is a flow diagram of an embodiment of a method for incorporating microexpression overlays into an avatar display in a virtual environment.

p-0036While, for the purpose of simplicity of explanation, the methodologies shown in the various Figures are shown and described as a series of acts, it is to be understood and appreciated that the present invention is not limited by the order of acts, as some acts may, in accordance with the present invention, occur in different orders and/or concurrently with other acts from that shown and described herein.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

p-0037The present embodiments contemplate techniques to integrate not only high-fidelity expressions into facial animations, but microexpression performances as well. In some embodiments, these altered animations are subsequently used to facilitate training a trainee to recognize particular expressions or microexpressions in a variety of contexts.

p-0038In a preferred embodiment, raw facial microexpressions performed by an actor are captured using high-fidelity motion capture, such as markerless high-resolution surface capture (i.e., Mova® CONTOUR) and used to create animation overlays which may then be adapted to fit a variety of different facial meshes. Traditional techniques, such as cyberscanning or marker tracking may be used as well.

p-0039While FACS is a specific system for emotion classification, which may be used by performing actors in embodiments of the present invention, the present invention is not limited to the FACS system. In other words, preferred embodiments are FACS agnostic, having the capability of extracting and using microexpressions that were, or were not, identified using the FACS. Stated in another way, certain embodiments are “FACS-compatible”, but not “FACS-based.”

p-0040Certain embodiments provide a system to apply genuine and universal human emotional microexpressions onto realistic computer avatars. The system preferably permits both skilled trainers and casual users to author a series of synthetic conversations and to apply micro-expressions with imbued significance upon the faces of the speakers. These digital micro-expression motions can be authored into spoken sentences to seamlessly combine with existing conversational facial animation. The resulting face motion, including subtle real human emotional markers, can be played back as complete conversations composed with these sentence building blocks. Certain embodiments employ a universal facial structure that allows any such expressions and spoken language animations to operate upon thousands of unique human faces with consistent accuracy and fidelity, while running upon a personal computer in real time.

p-0041In certain embodiments, the present invention includes a software package that permits researchers and trainers in deception detection to create a variety of in-person interview scenarios using realistic computer generated human subjects via easy-to-use computer interfaces. For example, embodiments of the program can use this expert-created content to create a series of training materials that are to be monitored by a learning management system, with the ultimate goal of facilitating the rapid training of personnel in immigration, customs enforcement, and other security systems.

p-0042<figref idrefs="DRAWINGS">FIG. 1</figref> is a functional block diagram of a system and methodology <b>100</b> according to an embodiment for acquiring microexpression overlays from an actor's live performance <b>104</b> and subsequently using the overlays to generate a new animation <b>118</b>. The methodology preferably involves taking three dimensional captures of live actor performances using, e.g., a markerless high-resolution surface capture technique such as Mova® CONTOUR. Any of different various surface capture techniques will suffice, including both markerless and marker-based, so long as they are of adequate resolution. These motion captures preferably comprise “very-high-resolution” (VHR) geometries of each of the actor's performances <b>104</b>. In some embodiments, vertex densities of these geometries are on the order of 10,000-20,000 polygons. In addition to the high density geometric models, photographs and audio recordings may be taken and stored in conjunction with each performance. Each of the actor's performances <b>104</b> generally fall within one of two categories: microexpression performances and non-microexpression performances.

Non-Microexpression Performances

p-0043The non-microexpression performances comprise gestural and verbal responses <b>106</b> and archetype information <b>105</b>. The archetype information <b>105</b> comprises at least one neutral face <b>110</b>. This neutral face <b>110</b> is simply the capture of the actor's expressionless face, which is stored as part of an archetype actor repository <b>111</b>. This neutral pose provides information regarding the actor's personal features <b>119</b><i>a</i>, <b>119</b><i>b </i>such as the relative positioning of the eyes and mouth. This information is referred to herein as archetypes <b>122</b>.

p-0044The gestural and interactive performances <b>106</b> can be stored in their own repository <b>121</b>, also referred to herein as an “interaction fragment store”. These performances are not required for microexpression overlay generation, but are useful for the creation of training materials.

p-0045In some embodiments, the performances are retargeted to a generalized mesh having a lower resolution to facilitate deployment on personal computers and to generalize the performance so that it might more easily be applied to other models. A generic face model, referred to herein as a “universal rig” or “rig” <b>101</b> is used to reduce the high-density performance data <b>104</b> to a lower density form which is more manageable for real-time rendering on a personal computer. The rig <b>101</b> thus comprises a “neutral” or “average” base facial mesh, of a lower density than the geometries captured from live actors. In some embodiments, the rig resolution comprises a model having roughly 1600 vertices. In many embodiments, the rig density is chosen to satisfy rendering requirements on personal computer systems (typically the raw performance geometries are too large for such real-time rendering).

p-0046By retargeting the high-density performance <b>104</b> to the lower density rig <b>101</b>, a more general representation requiring less storage space can be placed in each of the repositories <b>111</b>, <b>121</b>, and <b>113</b> (discussed below), which are preferably machine readable computer storages. Rig retargeting may be generally performed for all the performances, both microexpression and non-microexpression. Thus, the stored archetypes <b>122</b>, gestures and interactions <b>108</b>, and overlays <b>107</b> retrieved <b>109</b> from the repositories are typically of lower, rig-based resolution than the original performances <b>104</b> from which they are derived.

p-0047In addition to the stored meshes, audio and photographs of the actors may be stored in each repository <b>111</b>, <b>121</b>, <b>113</b>. In some embodiments, an artist may use photographs taken of the actor to generate textures compatible with the archetypical geometric mesh (for example, using tools such as Zbrush®). In some embodiments, the photos themselves may be used from a performance <b>104</b> to identify dermal displacements to depict in a bump map. From these textures, animated bump maps may be generated and used to increase the apparent resolution of the model without increasing the polygon count. A bump map is a file associated with a texture map, wherein each bump map pixel describes a perturbation of the surface normal of the object to which the texture map is to be applied. By modifying the lighting of the texture based on the perturbation in the bump map, the illusion of a higher vertex resolution may be given to the object. This is especially useful where retargeting of performances to the rig <b>101</b> may have reduced vertex resolution, omitting minute features of the actor's face such as details of the microexpression in the form of dermal wrinkles, etc.

Microexpression Performances

p-0048Microexpression performances typically comprise the actor's controlled execution of a microexpression beginning from a neutral, i.e. expressionless, face <b>110</b>. The controlled execution of a microexpression typically requires that the actor be specially trained in the expression technique. In some embodiments a single frame is stored <b>113</b> from the performance when the microexpression is maximally expressed (a “static” performance). In other embodiments a succession of frames are stored <b>113</b> to capture the microexpression as it is expressed over time (a “dynamic” performance). While controlled executions of microexpression performances are typical, some embodiments involve capturing uncontrolled or inadvertent microexpression performances.

p-0049The archetype and overlay extraction process <b>103</b> generates microexpression overlays <b>107</b> by subtracting the actor's neutral face <b>110</b> from the performance <b>104</b> of the microexpression. In this context, “subtracting” may mean identifying the offset necessary to translate each pixel from its position in the neutral mesh to its position in the performance mesh. The facial animation and each of the resulting microexpression overlays may comprise vertex displacements over time, in which case the displacement in each frame is used to generate each frame of the overlay. The resulting overlay <b>107</b> will consist of a collection of vertex offsets in one or more frames, which are stored in repository <b>113</b> in association with each of the microexpression types <b>120</b><i>a</i>, <b>120</b><i>b </i>which were performed. In some embodiments, the overlay offsets may also comprise bone information. Overlay <b>107</b> creation may be performed either before or after the VHR geometries have been reduced to the rig <b>101</b> resolution via retargeting. Typically, however, the subtraction is performed after mapping to the rig <b>101</b> so as to generalize the overlays and to optimize storage space.

p-0050Computer manipulation and generation <b>116</b> of the head <b>117</b> may be performed using the universal rig <b>101</b> and the archetypes <b>122</b>. In some embodiments, however, the head <b>117</b> is taken from a collection <b>116</b> that is not derived from the archetypes <b>122</b> and rig <b>101</b> (such as a cartoon rendering, facial mesh from a different dataset, etc.). In some embodiments, the selected or generated head <b>117</b> will serve as the foundation upon which gestures, microexpressions, etc. <b>115</b> are to be depicted. A module for interaction sequence automation or manual editing <b>114</b> retrieves interactions <b>108</b> from the interaction fragment store <b>121</b> or microexpression overlays <b>107</b> by accessing <b>109</b> the expression overlay repository <b>113</b>. These are combined with the head <b>117</b> to generate animation <b>118</b>. This combination is accomplished via the operations of an expression injection module described below. One skilled in the art will recognize that a “module” may be computer hardware, computer software, or a combination of both.

p-0051<figref idrefs="DRAWINGS">FIG. 2</figref> is a functional block diagram of a system <b>200</b> for inserting microexpression overlays <b>107</b> (<figref idrefs="DRAWINGS">FIG. 1</figref>) and actions <b>108</b> into synthetic or actor-based computer generated heads <b>117</b>. The system inserts a microexpression performance from the repository <b>113</b> into a computer-represented facial geometry <b>201</b>. In some embodiments, the geometry <b>201</b> is synthetically generated (an animated character), or may be taken from a repository of facial meshes (i.e., repository <b>111</b>). The mesh may be animated, in some embodiments with a live actor performance (i.e. from repository <b>121</b>). Facial geometry <b>201</b> may comprise a static frame or a succession of frames to create an animation. A succession of frames may constitute a representation of a speech performance. In this case, an accompanying audio excerpt <b>206</b> may be included. Certain of the performances <b>104</b> (e.g., the non-microexpression performances) captured from the live actors and stored in repository <b>121</b>, for example, may comprise spoken portions.

p-0052A repository of facial microexpression overlays <b>113</b>, as created by the methods described in <figref idrefs="DRAWINGS">FIG. 1</figref>, is provided. Expression injection module <b>202</b> preferably combines the geometry <b>201</b> with the overlay <b>107</b> from repository <b>113</b>. Various methods may be used to facilitate the combination. In one embodiment, the offsets of the overlays <b>107</b> are added to the vertex positions in each frame of animation <b>201</b>. The percentage of the overlay offset may vary over time, or at the direction of a user as discussed below. If a single static frame of a microexpression performance was used to create the overlay <b>107</b>, then vertices in the geometry <b>201</b> may be modified to positions interpolated between their original point and the points of the overlay by a variety of interpolation techniques (linear, polynomial, spline, etc.). The resulting geometry <b>203</b> comprises microexpressions <b>204</b> in addition to the original geometry <b>201</b>. More than one overlay <b>107</b> may be simultaneously combined with the geometry <b>201</b> by averaging or taking percentages of each overlay offset value. In some embodiments, overlays may be added to the face in realtime, substantially immediately upon selection by a user.

p-0053<figref idrefs="DRAWINGS">FIG. 3</figref> is a high-level diagram of an embodiment of a training system <b>300</b> to teach trainees how to recognize microexpressions, and to evaluate the trainees' performance. The system includes a trainee computing device <b>303</b> presenting a trainee interface <b>301</b> with various training materials <b>302</b>. The system also includes a trainer computing device <b>308</b>, which comprises various facial animations <b>201</b>, a repository of facial microexpression overlays <b>113</b>, expression injection module <b>202</b>, trainee evaluation module <b>309</b> and trainer interface <b>305</b> comprising tools for generating training material <b>304</b> for the trainee.

p-0054In certain embodiments, a trainee views the display interface <b>301</b> to recognize facial microexpressions from animations of computer-simulated faces <b>302</b>. The animations may be created automatically, in which case the trainer interface <b>305</b> may not be necessary. In some embodiments, a trainer interface may be used to initially create all the training materials, but the human trainer need not be involved in the actual training process itself. Thus, it may be possible that the trainee computing device <b>303</b> and trainer computing device <b>308</b> are a single machine or two separate machines. When training materials are created, the expression injection module <b>202</b> is preferably configured to combine one or more of the facial microexpression overlays <b>107</b> (<figref idrefs="DRAWINGS">FIG. 1</figref>) from repository <b>113</b> into a facial animation <b>201</b> to form a combined animation <b>203</b> (<figref idrefs="DRAWINGS">FIG. 2</figref>), by mapping at least a portion of the expression overlay <b>107</b> to at least a portion of the facial animation <b>201</b>. In some embodiments, the trainer selects the overlays <b>107</b> from repository <b>113</b> to apply to a particular animation <b>201</b>. After the user is presented with the facial animation <b>203</b> (<figref idrefs="DRAWINGS">FIG. 2</figref>) with microexpressions, the trainee evaluation module <b>309</b> may be used to assess the trainee's skill. In some embodiments, trainees provide responses to the training materials via trainee interface <b>301</b>. Trainees may attempt to interpret or discern emotional state conveyed by the facial animations based on microexpressions, or to recognize simply that a microexpression has been performed. Trainers may simulate real-world microexpresson behavior by limiting the microexpression duration to approximately 1/25-⅕ of a second. In various implementations, a microexpression can be less than 0.25 seconds, less than 0.2 second, or less than 0.1 seconds. It will be appreciated that the ability to insert ever shorter microexpressions is useful because those may be the most difficult microexpressions to detect, and the ability to train users to detect them is therefore advantageous. In some embodiments, the trainee evaluation module <b>309</b> is configured to use the received response to determine a performance level indicative of an extent to which the trainee has correctly interpreted the one or more combined animations. In some embodiments, the evaluation module <b>309</b> will report the performance level by displaying it in the trainee or another interface. While shown here as residing on the trainer computing device <b>308</b>, one skilled in the art will readily recognize that the trainee evaluation module <b>309</b> may reside on the trainee computing device <b>303</b> or a third device not shown. Trainee evaluation may or may not occur in real time. Advantageously, this system does not require that the trainer be familiar with the FACS coding system or with the recognition of microexpressions to create and administer a training program

p-0055<figref idrefs="DRAWINGS">FIG. 4</figref> is a display of an embodiment of one possible computer-implemented trainer interface <b>305</b>. The illustrated trainer interface <b>305</b> displays a preliminary rendering <b>401</b> of the animation being created. The trainer may select from a variety of models <b>400</b>, having different neutral (e.g., substantially expressionless) geometries as a result of the gender and different individual characteristics of the actor or synthetic character. Once a model is selected, the gender <b>404</b>, age <b>405</b>, ethnicity <b>406</b>, various facial features <b>408</b>, body mass <b>409</b>, and/or voice <b>410</b> may be modified. Other embodiments may include additional similar features.

p-0056In addition to head generation, a CG head animator <b>411</b> is provided with a variety of pre-programmed animations that can be selectively combined to generate new animations. A head sequencer <b>412</b> may permit joining of multiple animations sequentially. Head animations <b>415</b>, eye animations <b>414</b>, and body animations <b>413</b> are all provided to enable more realistic portrayals of the various emotions. In addition to these general parameters are the microexpresssion specific variables, such as the selected microexpression <b>419</b> (in some embodiments referred to as an emotion identifier) and its intensity <b>418</b>. In some embodiments, animations having higher microexpression intensity are expressed for a longer time and/or with greater amplitude.

p-0057The interface may also comprise a visual rendering of a voice sequence associated with an animation. An audio sequencer <b>407</b> may be used to select and modify the speech segments <b>206</b>. Each speech segment <b>206</b> comprises spoken text, but may be additionally provided with facial cues, <b>421</b><i>a</i>, <b>421</b><i>b</i>, <b>421</b><i>c</i>, etc., which the trainer may manipulate or supplement. At intervals between speech segments, the interface permits microexpression overlay insertions <b>420</b><i>a</i>, <b>420</b><i>b</i>, <b>420</b><i>c</i>, etc. In the illustrated embodiment, a microexpression insertion <b>420</b> can have a trainer-selected intensity <b>418</b> (e.g., “happiness 2”, where “2” represents intensity). In certain embodiments, the expression injection module <b>202</b> is configured to limit an intensity of at least a portion of the inserted microexpression in the combined animation based on the speech data. The overlays may be referenced by terms or symbols indicating their location on the face. In some embodiments, the emotions commonly associated with the expression (“happy”, “sad”, etc.) may refer to a given expression. As shown in <b>420</b><i>c</i>, which has both “fear” and “happy”, simultaneous insertion of multiples microexpressions is possible. In these instances, the overlays may be summed and averaged before being inserted.

p-0058Advantageously, it has been found that certain difficulties inherent to microepxression insertion (e.g., discontinuities in the facial animation, cancellation of microexpression effects by the facial animation, etc.) can be overcome by inserting microexpression overlays between speech segments. The longer the time between speech segments, the more refined the overlay interpolation may be. Though overlays are shown here as inserted between speech segments to avoid interference with the speech animation, in other embodiments they may be inserted at any point throughout the animation.

p-0059Microexpression overlays can preferably be applied dynamically at runtime to a particular facial animation, to create a modified animation containing the microexpression. Alternatively, the overlays may be applied and stored for subsequent replay. They may preferably also be stored as “training loops” (e.g., single sentences of facially animated spoken dialogue that can be replayed) or as “interview trees”, (e.g. conversations between an animated, interviewed character and the trainee. An animated interview tree can be rendered from the point of view of the interviewer, so that the trainee views the animation as though the trainee is the interviewer. In certain embodiments, a microexpression overlay can be applied even to a facial animation of a face that is different from that of the actor from whom the overlay was derived. For example, if the bone layout (and/or other applicable retargeting constraints of the Rig <b>111</b>) of the new facial model is substantially the same as that of the model created for the original actor, then the overlay can be applied to the model to generate the same microexpression. Thus, the trainer interface <b>305</b> can permit expert weighting of all content (e.g., inserted emotions or microexpressions) and control of difficulty settings for detection of highly subtle facial clues when the animations play in real time or are shown as static frames.

p-0060<figref idrefs="DRAWINGS">FIG. 5</figref> is another display of an embodiment of the trainer user interface <b>305</b>, which specifies the sequence of animations that will comprise the training material. Here the trainer interface specifies the interview tree structure provided to the trainee. In this example an interview <b>501</b> comprises a succession of questions and answers with accompanying animations for those responses provided by a synthetic character <b>505</b>. The trainer may insert additional animations via dropdown menu <b>503</b>. Animations of selections may be previewed in a preview frame <b>504</b> to ensure that the desired end result has been achieved. Interview trees could be instead constructed by numerous other interfaces, which may or may not permit immediate visualization of the constructed response.

p-0061<figref idrefs="DRAWINGS">FIG. 6</figref> is a system-block diagram of an embodiment of an integrated microexpression training system <b>600</b> employing methods and techniques such as those previously described. The first portion <b>621</b> of the system comprises the “session capture” of actor performances <b>104</b>. In this portion the VHR geometries <b>602</b> are generated for each performance <b>104</b>. For each captured sentence <b>603</b>, the head, face, body, and microexpressions (if present) of the actor are captured. These captures are saved <b>605</b> and stored for subsequent use in the Learning Management System (LMS) <b>606</b>. The LMS <b>606</b> provides these captures to the trainer interface <b>305</b> (<figref idrefs="DRAWINGS">FIG. 3</figref>) to create training sessions and training materials. The training materials may comprise slide shows <b>606</b><i>a</i>, training loops <b>606</b><i>b</i>, and/or interview trees <b>606</b><i>c</i>. The interview trees <b>606</b><i>c </i>may be created as described above with respect to <figref idrefs="DRAWINGS">FIG. 5</figref>.

p-0062Three-dimensional renderings of the actor heads can also be captured statically <b>620</b> as part of actor head creation <b>622</b>. Photography accompanying the capture <b>619</b> may be used to texture the model to create a more lifelike appearance. Hair and shoulders may similarly be added <b>618</b> to suggest realism. The face mesh <b>620</b> may be used to facilitate this process by properly aligning one or more photographs to the model. Alternatively, the head may be synthetically created <b>623</b> by manipulating the user-selected elements <b>610</b> (age), <b>611</b> (race), <b>612</b> (features/attractiveness), <b>613</b> (body mass), and/or <b>614</b> (voice type) of synthetic and actor models to generate <b>615</b> a new synthetic model. Once the model has been generated, the desired microexpressions and gestures inserted, and the accompanying training materials prepared, the resulting facial motion sequences <b>604</b> and accompanying audio <b>206</b>, if available, are presented to the trainee.

p-0063Once the head and overlay materials are assembled, the trainer can modify the animation to the desired specifications. Head animation <b>608</b><i>a</i>, eye animation <b>608</b><i>b</i>, body animation <b>608</b><i>c</i>, and microexpressions <b>107</b> can be specified in the LMS <b>606</b> using the various control features such as <b>413</b>-<b>417</b> of <figref idrefs="DRAWINGS">FIG. 4</figref>. As mentioned, in some embodiments, the emotions commonly associated with the expression or microexpression <b>609</b> (“disgust”, “fear”, etc.) may be used to refer to a particular overlay that may be selected <b>624</b>. The trainer may employ these overlays while establishing the deceit settings <b>607</b> of the animation to be used in conjunction with the training materials (i.e. the degree and nature of the microexpressions displayed). The trainer may thus choose between raw actor performances <b>617</b> or modified, synthetic performances <b>616</b> when creating animations to supplement the training material.

p-0064<figref idrefs="DRAWINGS">FIG. 7</figref> depicts one possible embodiment of a method for acquiring microexpression overlays. The method <b>700</b> begins <b>701</b> by converting a trained expert's performance <b>104</b> of a facial microexpression into a digital animation <b>702</b>. In certain embodiments this is accomplished using Mova® CONTOUR technology, but any technology capable of highly granular motion capture would suffice. During a capture session, a digital image of a substantially non-expressive facial pose <b>110</b> is taken <b>703</b> as well. This “normalized” facial pose is then subtracted from subsequent microexpression captures—leaving only the microexpression overlay <b>107</b>. The resulting difference is stored <b>704</b> as an overlay before coming to an end <b>705</b>. The overlay may subsequently be calibrated (i.e.) retargeted to facial meshes other than the facial mesh of the expert who performed the microexpression as described above.

p-0065With reference to <figref idrefs="DRAWINGS">FIG. 8</figref>, an embodiment of a method for selecting and presenting microexpressions <b>107</b> is disclosed, which begins <b>801</b> by displaying <b>802</b> a set of emotions or expressions to a user at a user interface, such as the interface <b>305</b> shown in <figref idrefs="DRAWINGS">FIG. 3</figref>. For purposes of simplicity, in some embodiments microexpression selection is accomplished by selecting a desired “emotion” response rather than the actual FACS expression units. The user (e.g., a trainer creating training materials) selects at least one emotion, although more than one, or percentages of different emotions, may be chosen in certain embodiments. The system receives <b>803</b> the user's selection. The microexpression overlay(s) corresponding to the user-selected emotion(s) is(are) identified <b>804</b> and then combined or blended <b>805</b> (“combine” and “blend” are used synonymously herein and in U.S. Provisional Application No. 61/119,975) with the facial animation to create the desired combined animation. In certain embodiments, each of the microexpression overlays in the repository <b>113</b> (<figref idrefs="DRAWINGS">FIG. 1</figref>) corresponds to one of the emotions displayed <b>802</b>.

p-0066With reference to <figref idrefs="DRAWINGS">FIG. 9</figref>, an embodiment of a method for training an individual using several of the above described embodiments is provided. The method begins <b>901</b> by retrieving <b>902</b> a portion of a facial expression overlay from the repository <b>113</b> (<figref idrefs="DRAWINGS">FIG. 1</figref>) to form a combined animation. This overlay is then combined <b>903</b> into a facial animation as described above. This combined animation is then presented <b>904</b> to the trainee at the trainee interface <b>301</b>, possibly with the accompanying audio, so that the trainee may provide a response (e.g., an indication of one or more emotions or expressions believed by the trainee to be shown in the animation that was presented <b>904</b>). In the illustrated embodiment, the received response may be used to determine a performance level indicative of an extent to which the trainee has correctly interpreted the combined animation. The trainee's performance level may then be determined <b>906</b>, possibly by the trainee evaluation module <b>309</b> (<figref idrefs="DRAWINGS">FIG. 3</figref>).

p-0067In addition to the training methodologies described above, the present embodiments can facilitate various additional applications. One possible application is the incorporation of microexpressions into social exchanges in virtual worlds (e.g., in a virtual environment run locally or on the internet) to facilitate involuntary tacit communication of emotion, e.g., falsehoods, uneasiness, etc. <figref idrefs="DRAWINGS">FIG. 10</figref>, for example, depicts one embodiment of a method for using microexpressions in a virtual environment. The method begins <b>1001</b> by displaying <b>1002</b> an avatar, corresponding to a user, in a virtual environment, although one skilled in the art would readily recognize that a non-playable character (NPC) may also be designed to exhibit microexpressions. Where the avatar corresponds to the user, the system then receives <b>1003</b> instructions from the user for controlling the avatar. The instructions may cause the avatar to perform some action or function, or to express certain behavior or content. In either event, the system can determine <b>1004</b> an emotional response associated with the instruction and access <b>1005</b> a facial microexpression overlay corresponding to the determined emotion. In certain embodiments, an autonomous system may perform these functions to create emergent behavior in an avatar, so as to supplement an artificial intelligence and to appear more lifelike. Such a system may concurrently observe user actions and determine which will generate a microexpression response. The system can be configured to use this information to drive face, body and head gestures.

p-0068Microexpression leakage may be known to trained observers, but not to most members of the online virtual community. Using the methods described above, the system can then combine <b>1006</b> the microexpression overlay into the facial animation of the avatar to form a combined animation. In this manner, conversation between users or users and NPCs can include not only explicit, standardized-expression statements, but more realistically, the involuntary aspects of communication that occur in the real world. Abstracting the microexpression leakage as non-verbal communication of emotion, that emotion can be combined or swapped with other non-verbal emotive techniques, such as recognition or injection of head nod, body language, eye motion and vocal tone. Speech recognition may further be incorporated to facilitate recognition of vocalized incongruities and to detect emotion-rich phrases and expletives. In the converse, masking such non-verbal communication can itself be a feature of the system. In some embodiments, the involuntary response may arise from emergent behavior (i.e. a combination of several factors). In some embodiments, comparing the statement to the user's profile will suffice to determine if microexpression leakage arises. In some embodiments, a degree to which a microexpression overlay is combined with a facial animation is determined by emergent behavior arising from the virtual environment and/or the user.

p-0069Finally, other embodiments of facial emotion creation, authoring and activation can involve creation of conversation systems for learning the nuances of social interaction, dating, job screening interviews, post traumatic stress detection and even the detection of underlying emotions behind poker faces and deceitful public figures. Many other applications for learning underlying emotions associated with facial expressions in a variety of settings are possible.

p-0070Prototype systems have already demonstrated the effectiveness of the above-described microexpression overlay creation, insertion, and training methods. Tests were devised to capture and test data and validate it on a series of software platforms (from art tools to final software engines), and an expert trainer was utilized both as an actual capture subject and to coach other performers. Comprehensive R&amp;D prototyping was achieved using lower resolution tests and scaling up to the available resources while quick reference movies of results were instantly shared to resolve issues of image quality.

p-0071Numerous technical details have been set forth in this document in order to fully disclose, and to facilitate an understanding of, particular embodiments of the system. As will be recognized, the various inventive features set forth in this document may be implemented differently than described herein. Thus, nothing in this document or the appendices is to be construed as a requirement of, or as a limitation on the scope of, any of the disclosed inventions.

p-0072The techniques described in this disclosure may be implemented in hardware, software, firmware, or combinations thereof. If implemented in software, the techniques may be realized at least in part by a computer-readable medium comprising instructions that, when executed, performs one or more of the methods described above. The computer-readable medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a computer-readable communication medium that carries or communicates code in the form of instructions or data structures and that can be accessed, read, and/or executed by a computer.

p-0073Various embodiments of this disclosure have been described. These and other embodiments are within the scope of the following claims.