Multi-camera, multi-sensor panel data extraction system and method

MULTI-CAMERA, MULTI-SENSOR PANEL DATA EXTRACTION SYSTEM AND METHOD

This application is being filed as a PCT International Patent application on November 25, 2020 in the name of On Time Staffing Inc., a U.S. national corporation, applicant for the designation of all countries and Roman Olshansky, a U.S. Citizen, inventor for the designation of all countries, and claims priority to U.S. Patent Application No. 16/696,781, filed November 26, 2019, the contents of which are herein incorporated by reference in its/their entirety/entireties.

Field of the Invention

The present invention relates to a grouping of data from visual cameras, audio microphones, and behavior or depth sensors. More specifically, the present invention relates to the extraction of synched data for specific individuals from the created sensor data and to the identification of event data from that synched data.

Summary

Various embodiments provide a system for extracting data from a plurality of sensors. The system can include one or more video cameras configured to record video input of the at least first participant and second participant in the panel discussion. The system can include one or more behavioral data sensors spatially oriented to record behavioral data input of at least a first participant and a second participant. The behavioral data sensors can be depth sensors configured to measure a distance between the depth sensor and a first participant. The behavioral data sensors can sense electromagnetic waves in the non-visible electromagnetic spectrum. The system can include one or more microphones configured to record audio input of the at least first participant and second participant in the panel discussion. The system can include a non-transitory computer memory storing a participant database. The participant database can include participant profiles for the at least first participant and second participant in the panel discussion. The system can include one or more computer processors in communication with the non-transitory computer memory. The one or more computer processors can be configured to receiving recorded video input from the one or more video cameras from at least the first participant and second participant in the panel discussion. The one or more computer processors can be configured to receive recorded behavioral data input from the one or more behavioral data sensors from the at least first participant and second participant in the panel discussion. The one or more computer processors can be configured to receive recorded audio input from the one or more microphones of the at least first participant and second participant in the panel discussion. The one or more computer processors can be configured to time synchronize the received recorded video input, the recorded behavioral data input, and the recorded audio input. The one or more computer processors can be configured to identify a first portion and a second portion of the time synchronized video input, behavioral data input, and audio input. The first portion is identified with the first participant and the second portion is identified with the second participant. The one or more computer processors can be configured to identify a first event in the first portion of the time synchronized video input, behavioral data input, and audio input. The one or more computer processors can be configured to identify a first participant profile associated with the first event, the first participant profile being assigned to the first participant in the panel discussion. The one or more computer processors can be configured to extract biometric behavioral data from a portion of the recorded behavioral data input associated with the first participant, wherein the extracted biometric behavioral data is associated with the first event. The one or more computer processors can be configured to store the extracted biometric behavioral data in the first participant profile. In various embodiments, the one or more behavioral data sensors sense electromagnetic waves in the non- visible electromagnetic spectrum.

In various embodiments, the one or more behavioral data sensors are depth sensors, wherein each depth sensor is configured to measure a distance between the depth sensor and the first participant.

In various embodiments, the first participant is an employment candidate.

In various embodiments, the event is when the first participant is speaking, when the first participant is listening to another participant, when the first participant changes body posture, when a first participant is interrupted by another participant, or when a first participant interrupts another participant.

Various embodiments provide a system for evaluating participants in a panel discussion. The system can include one or more behavioral data sensors spatially oriented to capture behavioral data of participants in a panel discussion. The system can include one or more microphones configured to capture recorded audio of participant voices of the participants in the panel discussion. The system can include a non-transitory computer memory in communication with a computer processor; and computer instructions stored on the memory for instructing the processor. The computer instructions can instruct the processor to perform the step of receiving recorded audio input of the participant voices from the one or more microphones. The computer instructions can instruct the processor to perform the step of receiving first recorded behavioral data input from the one or more behavioral data sensors, the first recorded behavioral data input being associated with a first participant. The computer instructions can instruct the processor to perform the step of receiving second recorded behavioral data input from the one or more behavioral data sensors, the second recorded behavioral data input being associated with a second participant. The computer instructions can instruct the processor to perform the step of identifying a time synchronization between the first and second recorded behavioral data inputs and the recorded audio input. The computer instructions can instruct the processor to perform the step of identifying the first participant’s voice in the recorded audio input. The computer instructions can instruct the processor to perform the step of identifying a first time segment of the recorded audio input as containing spoken content by the first participant. The computer instructions can instruct the processor to perform the step of identifying the second participant’ s voice in the recorded audio input. The computer instructions can instruct the processor to perform the step of identifying a second time segment of the recorded audio input as containing spoken content by the second participant. The computer instructions can instruct the processor to perform the step of in response to identifying the first time segment, extracting biometric behavioral data captured during the first time segment from the second recorded behavioral data input. The computer instructions can instruct the processor to perform the step of in response to identifying the second time segment, extracting biometric behavioral data captured during the second time segment from the first recorded behavioral data input.

In various embodiments, the one or more behavioral data sensors which sense electromagnetic waves in the non- visible electromagnetic spectrum.

In various embodiments, the one or more behavioral data sensors are depth sensors, wherein each depth sensor is configured to measure a distance between the depth sensor and one or more of the participants.

In various embodiments, the first participant is an employment candidate.

In various embodiments, the system can further include computer instructions stored on the memory for instructing the processor to perform the steps of associating an anonymized participant profile with the first participant, the anonymized participant profile being stored in a participant database in a non-transitory computer memory; associating an identified participant profile with the second participant, the identified participant profile being stored in the participant database; storing the extracted biometric behavioral data captured during the second time segment in the anonymized participant profile; and storing the extracted extracting biometric behavioral data captured during the first time segment in the identified participant profile.

Various embodiments provide a system for evaluating participants in a panel discussion. The system can include one or more behavioral data sensors spatially oriented to record behavioral data input of two or more participants in a panel discussion. The system can include one or more microphones configured to record audio input of the two or more participants in the panel discussion. The system can include a non-transitory computer memory storing a participant database, the participant database including participant profiles for the two or more participants in the panel discussion. The system can include one or more computer processors in communication with a non-transitory computer memory. The one or more processors can be configured to receive recorded behavioral data input from the one or more behavioral data sensors of the two or more participants in the panel discussion. The one or more processors can be configured to receive recorded audio input from the one or more microphones of the two or more participants in the panel discussion. The one or more processors can be configured to time synchronize the received recorded behavioral data and the recorded audio input. The one or more processors can be configured to associate a first portion of the recorded audio input with a first participant in the panel discussion. The one or more processors can be configured to extract spoken word data from the first portion of the recorded audio input using speech to text. The one or more processors can be configured to store the spoken word data in a participant profile for the first participant in the participant database. The one or more processors can be configured to associate a second portion of the recorded audio input with a second participant in the panel discussion. The one or more processors can be configured to associate a first portion of the recorded behavioral data input with the first participant. The one or more processors can be configured to associate a second portion of the recorded behavioral data input with the second participant. The one or more processors can be configured to identify a first time segment in the time synchronization corresponding to the first portion of the recorded audio input. The one or more processors can be configured to in response to identifying the first time segment, store behavioral data extracted from the second portion of the recorded behavioral data input in a participant profile for the second participant in the participant database. The one or more processors can be configured to identify a second time segment in the time synchronization corresponding to the second portion of the recorded audio input. The one or more processors can be configured to in response to identifying the second time segment, store behavioral data extracted from the first portion of the recorded behavioral data input in the participant profile for the first participant.

In various embodiments, the one or more behavioral data sensors which sense electromagnetic waves in the non- visible electromagnetic spectrum.

In various embodiments, the one or more behavioral data sensors are depth sensors, wherein each depth sensor is configured to measure a distance between the depth sensor and one or more of the participants.

In various embodiments, the first participant is an employment candidate.

In various embodiments, the participant profile for the first participant is an identified profile, and the participant profile for the second participant is an anonymized profile.

This summary is an overview of some of the teachings of the present application and is not intended to be an exclusive or exhaustive treatment of the present subject matter. Further details are found in the detailed description and appended claims. Other aspects will be apparent to persons skilled in the art upon reading and understanding the following detailed description and viewing the drawings that form a part thereof, each of which is not to be taken in a limiting sense. The scope herein is defined by the appended claims and their legal equivalents. Brief Description of the Figures

FIG. 1 shows a schematic representation of depth sensors, visual cameras, and audio recording sensors linked with one or more servers for data extraction.

FIG. 2 shows an example room set up of a system for recording panel discussions.

FIG. 3 shows an example of a single frame of an image recorded by a digital video camera such as those in FIG. 1.

FIG. 4 is a flowchart showing a method of recording participant data for a participant in a panel discussion.

FIG. 5 is a flowchart showing a method of compiling a keyword set.

FIG. 6 is a schematic view of audio streams captured during recording panel discussions and their use in flagging interruptions.

FIG. 7 is a schematic view of audio streams captured during an alternate system for recording panel discussions and their use in flagging interruptions.

FIG. 8 is a schematic view of a panel discussion environment according to some examples.

Detailed Description

Overall System

FIG. 1 reveals the technical difficulties encountered and solved by the disclosed embodiments in the present application. A system 10 is designed to record and sense multiple individuals. These individuals can form a panel, meaning that the video and audio recordings and other sensor readings acquired by the system 10 relate to individuals interacting with each other. In one embodiment, the system 10 uses sensor modules 20 that incorporate multiple different types of sensors. In FIG. 1, module- 1 20 is shown with an audio sensor or microphone 30 to make sound recordings, a visual camera 40 to make visual recordings, and two behavioral or depth sensors 50, 52 to record and more easily identify the physical movements of the individuals. In system 10, two additional modules 22, 24 are also present. Although this is not shown in FIG. 1, these additional modules 22, 24 can contain the same cameras, recording devices, and sensors 30, 40, 50, 52 as module- 1 20.

The system 10 also includes an additional visual camera 42 and an additional audio sensor 32. These elements 32, 42 are not part of the modules 20, 22, 24. In one embodiment, each module 20, 22, 24 and be focused on a different individual. In other embodiments, there are two, three, four, or more individuals being recorded than the number of modules 20, 22, 24. The additional visual camera 40 and audio sensors 32 are designed to increase the coverage of the recorded audio and visual data. In still further embodiments, each audio sensor 30, 32, visual camera 40, 42, and behavioral or depth sensor 50, 52 monitors multiple individuals.

In FIG. 1, the camera, modules, and sensors 20-52 are in data communication with one or more servers 70 (which is generally referred to as a single server 70 in this description). This data communication can flow over a network 60, such as a wired or wireless local area network or even a wide area network such as the Internet.

In the embodiment shown in FIG. 1, the cameras, modules, and sensors 20-52 first communicate with an edge server 71, which in turn is responsible for communications with the servers 70 over the network 60. An edge server 71 provides local processing power and control over the cameras, modules, and sensors 20-52. In some circumstances, the edge server 71 can provide control interfaces to aim and adjust the settings of the cameras, modules, and sensors 20-52. In other circumstances, the edge server 71 can provide tracking capabilities for the cameras, modules, and sensors 20-52. For example, the visual cameras 40, 42 may include a motorized mount that allows for the identification and tracking of human faces, with the edge server 71 providing the processing programming and power necessary to both identify and track those faces. In still further embodiments, the edge server 71 is responsible taking input from the modules 20, 22, 24 and creating audiovisual output with a variety of camera angles. While FIG. 1 is shown with the edge server 71 providing communications over the network 60 with the server 70, in other embodiments the jobs and capabilities of the edge server 71 are provided by the server 70 and no edge server 71 is needed. Also, capabilities that are described herein as being performed by the server 70 can, alternatively, or in addition, be performed by the edge server 71.

The server 70 and the edge server 71 are both computing devices that each a processor 72 for processing computer programming instructions. In most cases, the processor 72 is a CPU, such as the CPU devices created by Intel Corporation (Santa Clara, CA),

Advanced Micro Devices, Inc (Santa Clara, CA), or a RISC processer produced according to the designs of Arm Holdings PLC (Cambridge, England). Furthermore, the server 70 and edge server 71 has memory 74 which generally takes the form of both temporary, random access memory (RAM) and more permanent storage such a magnetic disk storage, FLASH memory, or another non-transitory (also referred to as permanent) storage medium. The memory and storage component 74 (referred to as “memory” 74) contains both programming instructions and data. In practice, both programming and data will generally be stored permanently on non-transitory storage devices and transferred into RAM when needed for processing or analysis. In FIG. 1, data 80 is shown as existing outside of the server 70. This data 80 can be structured and stored as a traditional relational database, as an object-oriented database, or as a key- value data store. The data 80 can be directly accessed and managed by the server 70, or a separate database management computer system (not shown) can be relied upon to manage the data 80 on behalf of the server 70.

The separate camera recordings and sensed data from the sensors 22-52 are stored as sensor data 82. In one embodiment, the data acquired from each camera, microphone and sensor 22-52 is stored as separate data 82. As explained below, it is necessary to identify data from the sensors that relate to a single individual in a panel. To accomplished this, data associated from the sensors 22-52 must be synchronized and analyzed in order to extract that data that is associated for a single individual. The resulting data is shown as data for individuals 84 in FIG. 1. The transition from sensor data 82 to individual data 84 becomes more difficult when one sensor, such as additional visual camera 42, is used to record multiple individuals, or when multiple sensors, such as the two depth sensors 50, 52, record different portions of a single individual.

As is also described in further detail below, the data for individuals 84 must be analyzed in order to identify events that relate to multiple individuals. Data created for the identified events is shown as data for events 86 in FIG. 1. Each event might relate to audio, visual, or 3-D sensor data for multiple individuals. With proper analysis, data about the interaction between these multiple individuals can be saved as data 86. This event data 86 is associated with specifically identified individuals. As a result, the event data 86 can be separately stored with data that is maintained on each of the individuals being monitored by system 10, which is shown as individual data 88 in FIG. 1.

In other words, system 10 is designed to acquire sensor data 82 from a plurality of cameras, microphones, and sensors 20-52 concerning multiple individuals, and then use this sensor data 82 to identify data relating to specific individuals 84. This data for individuals 84 describes the actions of each individual as recorded by the multiple sensors 20-52. From this data 84, it is possible to generate data describing specific events 86 that involved multiple individuals. This event data 86 can then be used to create relevant additional data 88 describing the individuals being monitored.

Possible Use of System 10

The system 10 can be used, for example in a panel interview setting where more than one individual is present and actively participating in the interview. In one example, a group of individuals are seated in a room for a panel discussion. Multiple cameras 40, 42 are positioned to record video images of the individuals. Multiple behavioral sensors such as depth sensors 50, 52 are positioned to record quantitative behavioral data of the individuals. Multiple microphones 30, 32 are positioned to record the voices of the participants. In some examples, the group of individuals includes at least one candidate for a job opening. In some examples, the group of individuals includes one or more potential future co-workers of the candidate. In some examples, multiple panel interviews are performed. In some examples, a panel interview includes one job candidate and two or more potential future co-workers. In some examples, a job candidate participates in two or more panel interviews with different groups of future co-workers. In some examples, each panel interview includes two or more job candidates. Throughout the description, analysis of candidates and participants will be described, and the terms candidates and participant can be substituted for each other, except where there is an analysis needed of candidates vs. non-candidates.

In one example, a participant profile database 88 is provided. The database is pre populated with profiles for known participants before the interviews are conducted. For example, each participant can provide personally identifying information in advance, such as a name, resume, job title, and description. While the panel interview is conducted, each participant will be recorded with one or more sensors 20-52. For example, one or more cameras 40, 42 can focus on the facial expression of the participant. In addition, or alternatively, one or more sensors 20-52 can focus on the body posture of the participant. One or more sensors 20-52 can focus on the hands and arms of the participants. The system 10 can evaluate the behavior of one or more participants in the discussion. The system 10 can calculate a score for one or more participants in the discussion. If the participant is a job candidate, then the system 10 can calculate a score for the candidate to assess their suitability for an open job position. If the participant is an employee, the system 10 can calculate a participation score. The system 10 can assess the participant’s strengths and weaknesses and provide feedback on ways the participant can improve performance in discussions or in a skill. The system 10 can observe and describe personality traits that can be measured by physical movements.

The system 10 can compute a participation score for individuals based on an evaluation of the individual’s body posture, facial expression, voice characteristics, spoken word content, and contextual and domain knowledge data points.

In some examples, the system 10 can be described as a group discussion: A question is posed, and then multiple individuals answer the question in a roundtable discussion style. Audio, video, and behavioral data are recorded during the session. When a participant is talking, the system 10 extracts keywords from the participant’s speech using a speech to text software module. When the participant is listening to other people speaking, the system 10 monitors the participant’s behavioral data, such as eye gaze and body posture. The system 10 performs these features for each person involved in the discussion. Different opportunities for evaluating participants are available when the participants are part of multiple panel interviews. However, the system is capable of evaluating participants that only are present in one panel interview.

Individuals in the group are evaluated based on their interactions with other group members. For example, when one individual is speaking, a listener in the group can be evaluated using quantitative behavioral data recorded of the listener to determine whether the listener was being attentive to the speaker. This type of interaction between individuals can be considered an event, and data is stored in the data for events 86. As another example, two individuals engaged in back-and-forth conversation can be evaluated for whether they project a positive attitude or negative attitude toward the other speaker.

In some examples, each discussion participant is evaluated based on multiple types of behaviors and interactions. Participants can also be evaluated based on the quality of their input in the discussion, such as the depth of information communicated during the discussion. Natural language processing can be used to evaluate the substance of a speaker’s comments.

Each discussion participant can be evaluated while they are speaking. Each discussion participant can also be evaluated while they observe others speaking. This provides insight into the level of participation of each person in the discussion, the suitability of the individuals to work together as a team, and the engagement or attention of each person to the others in the room. Other insights include how well each individual participant fits into the group, and whether the participants are likely to work well together as a group. Based on the evaluations of each participant, or of each participant of a subgroup of the participants, the group as a whole can be evaluated. One example of a subgroup that might be analyzed is the group of current employees in the discussion. Analysis of this subgroup can shed light on the company’s culture, strengths, and challenges. Positive and negative actions by single individuals or interactions between multiple individuals can be flagged for evaluation. The system uses a keyword set compiled from text extracted from the audio input of each of the participants. The compiled keyword set enables an evaluation of the substance of the panel discussion. Each individual participant can be evaluated based on the keywords used by that particular participant in relation to the entire compiled keyword set across all individuals in the panel.

In some cases, a participant will only have a short answer, such as agreeing with another participant. In the context of the discussion, the system can flag this interaction/event as being a positive interaction in which the speaker plays well with the rest of the team. In other cases, a participant may have a very long answer in which the participant speaks at length about a topic. Very long answers and very short answers may potentially skew results of the analysis. To compensate for situations in which a participant has a very long answer or a very short answer, the system can evaluate each participant’s best responses. The best responses can be chosen based on many different criteria. For example, best responses can be selected based on the density of high-quality keywords the speaker produces while answering the questions.

The disclosure herein the individual data 88 forms part of a participant database system for analyzing and evaluating individual participants involved in a panel, and for evaluating the group as a whole based on positive or negative interpersonal events detected during the group discussion.

In some examples, one or more digital profiles are established in a profile database ahead of a group interaction. One or more participants in the panel may be associated with digital profiles in the profile database.

In some examples, the system 10 distinguishes between different individual voices in one or more recorded audio streams. 82 Each voice is associated with a profile in the individual data 88. In some examples, each individual participating in the panel can have a preexisting profile in the participant database. In alternative examples, the database can create new profiles for voices identified but not recognized in the audio tracks 82. In some examples, the profiles can be personally identifying for some or all of the participants in the panel discussion. In alternative examples, the profiles can be anonymized for some or all of the participants. In some examples, participants are evaluated based on a response to a stimulus. In these examples, the participant being evaluated can be personally identified, even when their behavior is being evaluated based on an interaction with a participant who is anonymized.

The system 10 is designed to evaluate participant behavior during a panel interaction (such as a panel interview). A panel discussion is an interaction between two or more participants. One example of a panel discussion is a job interview where a candidate for a company is being interviewed by two or more employees of the company. The term “panel interview” or “panel discussion” as it is used herein encompasses interactions between and discussions among two or more people, even if not in the context of a job interview. One such example of a panel interview is a group discussion of an issue of shared interest. Within the context of a workplace, a panel discussion could be a group discussion of a shared goal, a shared challenge, a facilities decision, or an information technology decision, such as a new software tool that is being considered for adoption.

The system provides evaluation modules that use recorded data as input. In the various examples herein, “recorded data” refers only to data that was recorded during the panel discussion, such as data 82. Recorded data can be recorded audio data, recorded video data, and/or recorded behavioral sensor data. Recorded data can mean the raw data received from a sensor 20-52, or it can be data converted into a file format that can be stored on a memory and later retrieved for analysis. The evaluation modules also use extracted data as input. As used herein, “extracted data” is information that is extracted from raw data of the recorded audio, recorded video, or recorded behavioral sensor data. For example, extracted data can include keywords extracted from the recorded audio using speech to text software. Extracted data can also include body posture data extracted from the behavioral sensor data, or eye-movement data extracted from the recorded video. Other examples are possible and are within the scope of the technology.

The evaluation modules can also use external data as input. “External data,” as used herein, is data other than that recorded during the panel discussion. External data can refer to audio data, video data, and/or behavioral sensor data that was recorded at some time other than during the panel discussion. External data also can refer to text data imported from sources external to the panel discussion. In the context of an interview, for example, the external data may include resumes, job descriptions, aptitude tests, government documents, company mission statements, and job advertisements. Other forms of external data are possible and are within the scope of the technology.

The system 10 is capable of storing data in a database structure. As used herein, “stored data” refers to data that is stored in at least one database structure in a non-volatile computer storage memory 74 such as a hard disk drive (HDD) or a solid-state drive (SSD). Recorded data, extracted data, and external data can each be stored data when converted into a format suitable for storage in a non-volatile memory 74.

In some examples, the system 10 analyzes recorded audio data, recorded video data, and recorded behavioral sensor data for each individual in the panel discussion (data 84) to identify events (data 86) that may be related to the evaluation modules. This analysis may be conducted as the raw data is being recorded, or the recorded data may be stored in a memory for later analysis. System for Recording Panel Discussions

FIG. 2 shows an example room set up of a system for recording panel discussions. A group of participants 101, 102, 103, 104, and 105 are situated in a room. In the example shown FIG. 2, the participants are seated at a table 111. In this example, each participant is provided a microphone 121, 122, 123, 124, 125. Multiple digital video cameras 131, 132,

133, 134, and 135 are positioned in the space to record video of each participant. Multiple additional sensors 151, 152, 153, 154, 155 can be positioned near or around the participants, such as attaching one additional sensor to each housing for the microphones 121, 122, 123, 124, 125. In an example, the additional sensors can take the form of an IR sensor, a second video camera, or a motion sensor.

In some examples, each of the digital video cameras 131, 132, 133, 134, and 135 includes a behavioral data sensor, which can be a sensor that senses electromagnetic waves in the non-visible electromagnetic spectrum. For example, the behavioral data sensors can be depth sensors that emit and detect electromagnetic waves in the infrared or near-infrared spectrum.

For example, an infrared sensor can be used to sense data corresponding to the individual’s body movements, gestures, or facial expressions. Behavioral data can be extracted from the sensor data input to determine additional information about the candidate’s body language when reacting to or interacting with other discussion participants. A microphone can provide behavioral data input, and the speech recorded by the microphone can be extracted for behavioral data, such as vocal pitch and vocal tone, word patterns, word frequencies, and other information conveyed in the speaker’s voice and speech.

In some examples, multiple sensors can monitor a single participant, such as at least two sensors for each individual. In some examples, the system can include at least three, four, five, six, seven, eight, nine, ten, or more sensors to monitor each participant. In some examples, the multiple sensors can be arranged at various positions relative to the participants, such as a sensor directly in front of a participant, a sensor behind a participant, a sensor below a participant, a sensor above a participant, a sensor behind a participant, and/or a sensor to one or both sides of a participant. In some examples the sensor can be adjacent to a microphone, such as on a microphone stand. Various sensor arrangements can include two or more of the listed possible sensor locations.

In some examples, two or more sensors can be located in the same housing. In some examples, two sensors can be arranged with one sensor above the other sensor. In some examples, two sensors can be located at different distances from the participant. In some examples, two sensors can be equal distances away from the participant. In some examples, the two sensors can be arranged with one sensor in front of the other sensor, such that one sensor is closer to the participant than the other sensor. In some examples, two sensors can be arranged side by side, such that the sensors are at different angles to the participant. In some examples, one sensor can be positioned in front of a participant and a second sensor can be positioned near the participant, such as shown in FIG. 2 with digital video cameras 131, 132, 133, 134, and 135 positioned in front of the participants, and additional sensors 151,

152, 153, 154, 155 on a microphone 121, 122, 123, 124, 125.

FIG. 3 shows an example of a single frame 227 of an image recorded by a digital video camera such as those in FIG. 2. For example, the frame 227 can show the individuals 102, 103, 104. The frame 227 also includes an image of the microphone 123. In some examples, the frame 227 can be linked to a grid of pixels of data sensed by a depth sensor. The pixel data can be parsed to extract data about the individual 103 in the frame. For example, the system can determine that the top of the individual’s head is approximately in the center of the frame. The system can identify that the microphone 123 is assigned to the individual 103. Data streams from the microphone 123, video camera 133, and behavioral data sensor 143 can all automatically be associated with a participant profile for the individual 103 in the participant database.

In some examples, at least one of the participants is a candidate for employment. In some examples, the other individuals participating in the panel interview are interviewers such as hiring managers, or the other individuals can be employees of the company that is hiring. In some examples, one or more of the participants is a potential future coworker of the candidate. In some examples, two or more of the participants are candidates for employment. Alternatively, the panel interview can be composed entirely of job candidates.

In some examples, the system is provided with a server for processing the inputs from the cameras, microphones, and behavioral data sensors. The server includes a processor and storage memory accessible by the processor. In some cases, the server can be located physically near the cameras, microphones, and behavioral data sensors. In alternative examples, the server can be a remote server to which the video, audio, and behavioral data can be sent over a network connection. In some examples, processing of the data input occurs at a local server. This could include extracting behavioral data from the raw sensor data received from the behavioral data sensors. In some alternative examples, the system transmits the data for analysis to a remote server.

In some examples, each individual participant is provided with one microphone, one video camera, and one or more behavioral data sensors, each of which is position primarily on the individual to record data of that single individual, as shown in FIG. 2. In these examples, the data streams output from the microphone, video camera, and behavioral data sensors for that individual can associated with the individual’s profile in the profile database during a set-up process for the discussion.

In some examples, even when participants are assigned to a single camera, microphone, or sensor, data from other participants may be recorded. For example, if an audio microphone is sufficiently sensitive, it will pick up voices from more than one person in the room, even if that microphone is intended to only record audio input from a single person. In these cases, system hardware or software can filter out the undesired audio portions, leaving only the desired data input for the single individual.

In some examples, each digital video camera has a sensor capable of recording body movement, gestures, or facial expression. In some examples, the sensors can be infrared sensors such as depth sensors for motion sensors. A system with multiple depth sensors can be used to generate 3D models of the individual’s movement. An infrared sensor emits infrared light and detects infrared light that is reflected. In some examples, the infrared sensor captures an image that is 1,024 pixels wide and 1,024 pixels high. Each pixel detected by the infrared sensor has an X, Y, and Z coordinate, but the pixel output is actually on a projection pane represented as a point (X, Y, 1). The value for Z (the depth, or distance from the sensor to the object reflecting light) can be calculated or mapped.

For example, the system can analyze the individual’s body posture by compiling data from three behavioral data sensors. This body posture data can then be used to extrapolate information about the individual’s behavior in reaction or interaction with other individuals during the panel discussion, such as whether the individual was reserved or outgoing, or whether the individual was speaking passionately about a particular subject.

FIG. 8 shows a schematic diagram of one example of a panel environment 700. The panel environment 700 includes an edge server 201 that has a computer processor 203, a system bus 207, a system clock 209, and a non-transitory computer memory 205. The edge server 201 is configured to receive input from the video and audio devices of the environment and process the received inputs.

In various examples, the panel environment 700 will be set up at a location conducive to a group discussion with sufficient room for the recording equipment. Examples of possible locations include a conference room at a company or meeting facility, or a lecture hall at a college campus.

The panel environment 700 can include an attendant user interface 233. The attendant user interface 233 can be used, for example, to check in users, or to enter data about the users. The attendant user interface 233 can be used to turn cameras, behavioral data sensor, or microphones on or off if there is a change in the expected attendees after physical set-up. The attendant user interface 233 can be provided with a user interface application program interface (API) 235 stored in the memory 205 and executed by the processor 203. The user interface API 235 can access particular data stored in the memory 205, such as interview questions 237. The interview questions 237 can be displayed to the panel members on an optional user interface, which is not illustrated.

The system includes multiple types of data inputs. FIGS. 2 and 8 show a physical set up for five individuals participating on the panel discussion, with a video camera, microphone, and behavioral data sensor provided for each participant. The system may have other numbers of video cameras, microphones, and behavioral data sensors, such as at least two, at least three, and at least four. In the example of FIG. 8, the five video camera devices 131, 132, 133, 134 and 135 each produce video input which is stored on memory 205 in the video files 261 in the edge server 201. Each video camera device 131, 132, 133, 134 and 135 can include a variety of cameras, such as cameras with a variety of scope or a particular direction, so that more information can be gathered about the participants. In one embodiment, each video camera device 131, 132, 133, 134 and 135 includes a wide-angle camera and a close-up camera. In one embodiment, each video camera device 131, 132, 133, 134 and 135 includes a high angle camera and a low angle camera.

In the example of FIG. 8, the five microphones 121, 122, 124, and 125 produce audio input 262 stored on memory 205 in the edge server 201. The system also receives behavioral data input from behavioral sensors 141, 142, 143, 144, and 145. The behavioral data input can be from a variety of different sources. In some examples, the behavioral data input is a portion of data received from one or more of the cameras 131, 132, 133, 134 and 135. In other words, the system receives video data and uses it as the behavioral data input. In some examples, the behavioral data input 267 is a portion of data received from one or more microphones. In some examples, the behavioral data input is sensor data from one or more infrared sensors provided on the cameras 131, 132, 133, 134 and 135. The system can also receive text data input that can include text related to a candidate on the panel, and candidate materials 223 that can include materials related to the individual’s job candidacy, such as a resume or text from questions that the candidate has answered. The system can also receive panel materials 221 to provide information about the make-up of the panel members. The panel materials 221 can include text with notes on company culture, keywords related to the company’s industry and business, the company’s mission statement, and special skills or certifications that are sought by the company. In some examples, the video inputs are stored in the memory 205 of the edge server

201 as video files 261. In alternative examples, the video inputs are processed by the processor 203, but are not stored separately. In some examples, the audio input is stored as audio files 262. In alternative examples, the audio input is not stored separately. The panel materials 221, candidate materials input 223, and behavioral data input 267 can also be optionally stored or not stored as desired.

In some examples, the edge server 201 further includes a network communication device 271 that enables the edge server 201 to communicate with a remote network 281. This enables data that is received and/or processed at the edge server 201 to be transferred over the network 281 to a candidate database server 291. The edge server 201 includes computer instructions stored on the memory 205 to perform particular methods. The computer instructions can be stored as software modules. The system can include an audiovisual file processing module 263 for processing received audio and video inputs and assembling the inputs into audiovisual files and storing the assembled audiovisual files 264. The system can include a data extraction module 266 that can receive one or more of the data inputs (video inputs, audio input, behavioral input, etc.) and extract behavior data 267 from the inputs and store the extracted behavior data 267 in the memory 205.

Additional hardware and software options for the system described herein are shown and described in commonly owned, co-pending U.S. Patent Application 16/366,746, titled Automatic Camera Angle Switching to Create Combined Audiovisual File, filed on March 27, 2019, which is incorporated herein by reference in its entirety.

Data Streams from Panel Recordings (FIG. 4) FIG. 4 is a flowchart showing a method of recording participant data for a participant in a panel discussion. The method can be performed for each participant in a panel discussion such as that conducted in relation to the system of FIG. 2. In step 301, recording devices are positioned in relation to a participant. For example, referring to FIG. 2, a microphone 121 and a video camera 131 having a video data sensor and a behavioral data sensor are positioned in relation to participant 101 such that each of the sensors can record audio, video, and behavioral data of the participant 101. Similarly, the microphones 122, 123, 124, and 125 and video cameras 132, 133, 134, and 135 are positioned in relation to the participants 102, 103,

104, and 105. The method of the flowchart in FIG. 4 will now be described in relation to participant 101. It should be understood that the method can be performed in a substantially similar manner for participants 102, 103, 104, and 105.

In step 301, multiple participants are positioned in front of recording devices, such as microphones, video cameras, and behavioral data sensors. In step 311, audio data of the individual 101 is recorded by a microphone 121. Behavioral sensor data of the individual 101 is recorded at step 312. The recording can be performed by a depth sensor that is part of the video camera 131. At step 313, video data of the individual 101 is recorded by video camera 131. The audio data recorded in step 311 goes through two flow processes. In one process, quantitative vocal data is extracted from the recorded audio data at step 321. The quantitative vocal data can be related to sounds recorded in the audio other than the actual content of the spoken words. For example, the quantitative vocal data can be related to voice pitch, voice tone, speech cadence, speech volume, a length of time speaking, or other similar qualities other than the content of the speech. In a second process, at step 331, the quantitative vocal data that was extracted from the recorded audio data is stored in a non-transitory computer memory for later processing. In some examples, the participant 101 has a participant profile in a profile database that is pre- associated with the audio data recorded in step 311. For example, if microphone 121 is dedicated exclusively to recording the voice of participant 101, the system can be configured such that a profile for participant 101 can be provided in a participant database and the recorded audio data 311 can be automatically associated with that profile.

In alternative examples, audio data recorded at step 311 is not associated with an identified individual. In this case, the system can create a new participant profile. This new participant profile can initially be anonymized. Optionally, an anonymized profile can later be linked to an identity for a participant.

In a second process flow, keywords are extracted from the recorded audio data at step 322. In some examples, the full text of an audio recording of a participant’s speech can be extracted using a speech to text software module. At step 332, stop words are removed from the extracted keywords. As used herein, a “stop word” is a part of speech that does not convey substance. For example, the words “the,” “a,” and “is” are necessary for grammar, but do not convey substantive meaning. Stop words are removed from the list of extracted keywords. In step 342, the remaining keywords are stored in the participant profile.

Turning now to the behavioral data process flow, after behavioral sensor data is recorded at step 312, quantitative behavioral data is extracted from the behavioral sensor data at step 323. For example, data related to a person’s head and torso movement can be extracted from raw sensor data of a depth sensor. Other types of quantitative behavioral data that can be extracted from behavioral sensor data are described in US patent application number 16/366,703, filed March 27, 2019 and entitled Employment Candidate Empathy Scoring System, the entirety of which is herein incorporated by reference. At step 333, extracted quantitative behavioral data is stored in the participant profile.

At a further process flow, video images are extracted at step 324 from the recorded video data. At step 334, the video data is combined with audio data recorded at step 311 to create a combined audiovisual file containing both video data and audio data. The video data and audio data can be time synchronized. In audiovisual file is stored in the participant profile, step 344.

In step 325, quantitative behavioral data is extracted from the recorded video data from step 313. For example, individual still video frames recorded by a digital video camera 131 may contain image data related to an individual’s eye movement, mouth movement, etc. This quantitative data can include measurements of distances or shapes of objects in the video frame. These measurements and shapes can be stored as quantitative behavioral data in step 335. Other examples are possible and will be described below.

Keyword Data Set Compilation Module (FIG. 5)

In some examples, participants are evaluated at least in part based on the substantive content of the panel discussion. The keyword data set compilation module compiles a set of keywords that the participants used during the panel discussion. In some examples, the keyword data set contains a compilation of keywords from one panel discussion. In alternative examples, the keyword data set can contain keywords compiled from across multiple panel discussions. The keyword data set can be used to evaluate individual candidates or groups of candidates based on the substance of the conversation by the group in the panel discussion.

The method of compiling a complete keyword data set receives as input text extracted from recorded audio data that was recorded during the panel discussion. This can be performed in a number of ways. The system can extract text from audio data using speech to text software. In some examples, the system extracts text in real time as the panel discussion is being recorded by one or more microphones. In alternative examples, the audio track is stored in a non-transitory memory, and speech to text software is later used to extract text from the audio.

The system can extract text from the audio of some or all conversation participants. In some examples, however, the system will extract text only from the audio recorded for certain targeted individuals. For example, in some implementations, a job candidate is a participant in the panel interview. The system can be configured such that speech to text is performed only on the audio track for the job candidate, and the extracted keywords are used for evaluation of the job candidate. In further examples, if there are multiple job candidates present in a single panel interview, or multiple panel interviews each having one or more job candidates, the system could be configured so that speech to text is performed only on the substance spoken by the job candidates. The resulting keyword data set would be comprised just of the substance spoken by job candidates during one or more panel interviews. Each job candidate’s individual performance in the substance of the interview can be compared to the keyword data set.

Turning to FIG. 5, in step 401, audio is recorded for one or more individuals present in the panel discussion. As noted above, the recorded audio input can be for one individual, such as a job candidate. The recorded audio input can also be for multiple individual participants in the panel discussion. Where the system uses multiple recorded audio inputs from multiple participants, each of the individuals’ substantive participation in the discussion can be evaluated against the substance of the conversation during a single panel discussion. Where the system records multiple panel discussions, each of the individuals’ substantive participation can also be evaluated against the substance of the multiple panel discussions.

In step 402, text is extracted from the recorded audio using speech to text software. This type of software is commercially available from a number of different companies, including but not limited to DRAGON® transcription software available from Nuance Communications, Inc., Burlington, Massachusetts, USA. In some examples, the extracted text can be stored as a full-text transcription of the panel discussion. In some examples, error correction can be performed on the extracted text to identify potential errors. In some examples, the extracted text can be compared against a dictionary of known words. In some examples, the system can automatically remove some or all unknown words from the extracted text. Many different examples of initial processing of the full text of the discussion are possible and are within the scope of the technology. In step 403, stop words are removed from the full text of the extracted audio. As used herein, a “stop word” is an insignificant word that does not provide statistically important information about the substance of the conversation. Short words, conjunctions, and similar words that are not related to the substantive content can be examples of stop words to remove from the full text. Particular stop words can be preselected for removal, or the system can use a statistical analysis on the full text to determine words that are unlikely to provide insight into the substance of the conversation.

In step 404, a complete list of keywords is compiled. The complete list contains keywords that can be used to evaluate an individual’s performance against the performance of one or more other people in the panel discussion. In some examples, the system moves on to step 405, in which the system performs a statistical analysis of the keywords to determine keywords that are most likely to be useful in distinguishing one participant against others. For example, the system could eliminate commonly used words or phrases.

At step 406, natural language processing is used to determine qualities of keywords used in the discussion. For example, natural language processing can be used to contextualize or disambiguate keywords. Natural language processing can also be used to estimate a person’s level of education, where they grew up, their level of familiarity with specific technical topics, and past experience with specific technical topics.

Optionally, at step 407, one or more high-value keywords can be identified. The high- value keywords can be words that are singled out as being especially relevant to the substance of the conversation, or of particular interest to the final evaluation of the panel participants. For example, if the keyword data set is related to one or more job candidates. Keywords that are highly relevant to the job opening can be selected as having a higher value than other keywords in the keyword set. Optionally, an algorithm can be used to determine which keywords are more important than others, so that the high-value keywords can be weighted more heavily when used as input into an evaluation of the participant’ s contribution to the conversation.

At step 408, the resulting keyword data set is stored in a non- transitory memory for later retrieval and processing during evaluation of the panelists. Note that the keyword data set is not simply a predetermined set of words that the system looks for in the text of the panel discussion. Instead, the keyword data set is particularly tailored to the specific conversation that was discussed in the panel discussion or multiple panel discussions. Thus, the system is not a one-size-fits-all approach, but instead contextualizes the substance of the discussion in the panel discussion and evaluates participants based on the actual conversation held between the multiple participants. This can be useful for situations in which the participants are speaking about a highly specialized topic, or where a particular word in the context of the conversation has a different meaning than its general meaning in the language.

A new keyword data set can be compiled when desired. For example, a keyword data set can be compiled for one or more panel interviews for a particular job opening. Alternatively, a keyword data set can be compiled for each company that uses the system, providing each company with a unique set of keywords that are specifically relevant to its own employees and operations. The system can start from scratch with a clean keyword data set each time the system is used.

Data Extraction Modules

The system uses a number of different software modules to extract information from recordings of the panel discussion. The data extraction modules take as input raw data input from the video, audio, and behavioral data sensors. The output of the data extraction modules is quantitative information about the individual participants’ behaviors during the panel discussion. Data extraction modules can be used to identify particular participants in the discussion, and optionally associate the participants with their corresponding profiles in the profile database. Data extraction modules can use an input of the particular timestamp during the recording of the panel discussion and then extract data related to an event taking place around the time of the timestamp.

Participant Identification and Profile Matching Module The participant identification module matches a participant’ s video input, audio input, and/or behavioral data sensor input to the identity of an individual so that the system associates all of these inputs with a specific person’ s profile in the database. If a name or other sufficient identifying information is not present in the system, a guest profile may be used. In an alternative arrangement, the participant identification module can associate an anonymized profile in the database with a participant’s video input, audio input, and behavioral data sensor input. In any case, video input, audio input, and behavioral data sensor input of a single individual being recorded are linked together with a participant profile, whether that individual is explicitly identified, anonymized, or treated as a guest. A participant can be identified and matched to a profile in a number of different ways. Predetermined Recording Devices Matched to Profile

In one example, each participant in the panel discussion is associated with a predetermined recorded stream of audio, video, and behavioral sensor data. Referring to the example of FIG. 2, the system can associate a preexisting participant profile in a participant database with individual 101. An administrator can initially assign the audio input of the microphone 121, the video input of the video camera 131, and the behavioral data sensor input of the behavioral data sensor 141 to the individual 101 having the preexisting participant profile. This can be done, for example, through a graphical user interface providing instructions to the processor for how to handle data input to the system. During later data extraction and analysis, data extracted from the audio input of the microphone 121, the video input of the video camera 131, and the behavioral data sensor input of the behavioral data sensor 141 will automatically be associated with the participant profile of the individual 101 based on this initial assignment by the administrator·

Differentiating Voice Prints and Associating Video and Behavioral Data with Audio In another example of identifying a participant and matching to a participant profile, one or more microphones record a plurality of individuals speaking during the panel discussion. The system can execute software to differentiate multiple voices within the audio stream or streams recorded by the one or more microphones. The system determines the total number of speakers. Using a voiceprint identification, each different voice can be temporarily assigned to a new profile for an unidentified participant. In cases in which all participants’ data is to be anonymized, and anonymized identification number and the assigned to each voiceprint. In alternative examples, voiceprint identification can be used to personally identify one or more particular participants. Alternatively, a participant’s voice can be manually identified and associated with an identified participant profile in the participant database.

Extracting Behavioral Data from Behavioral Data Sensor Input

Regardless of how many variables are measured, the data that is recorded is sequential data, which can be placed in an array in working memory. Each record in the array would correspond to an elapsed time interval, all together a sequential record set taken from the recording and read into memory as the data is being processed. Note that the bulk data is not necessarily saved in memory. Initially, the bulk data is read into working memory. After processing, the resulting observation and conclusion data can be written to a file. This method takes a lot of working memory but is more efficient than writing out raw files and later reading them into working memory again for processing. The array of data points provided by the depth sensor of the system can measure distances of movement in millimeters, for which the mode for each point can be calculated.

In one example, an assumed error rate is plus or minus 15 millimeters. The mode of a data set of distance measurements is the distance measurement that appears the most often. One goal of such a system is to distinguish minor fluctuations from actual posture changes. The mode, not an average, is used for each data point. The data set of distance measurements is associated with each unique body posture during the panel discussion. The mode of each data set, or unique body posture instance during the discussion, can be determined without saving the raw data.

As the recording progresses, and each moment of time elapses, the record from the virtual array would be written if the values therein represented a new mode for the body posture. This would thereby reduce the number of stored records quite drastically compared to recording data at small time intervals during the panel discussion. In one alternative method, the data is written out to a saved file and an algorithm is used to find the mode of all points at one time.

The system uses an analytical process to determine the most common postures for that individual and when the individual repeats those postures. In the method described here, the raw positional data could be discarded without affecting the accuracy of results.

The system can examine quantitative behavioral variables using numerical physical measurements. For example, the quantitative behavioral data may measure the changes in distance in millimeters. In some examples, a tolerance for movement, such as at least 15 millimeters or at least 10 millimeters, will be used before a movement is tracked. The use of a tolerance for movement will prevent false movement tracking due to respiration. For superior eye gaze and head movement tracking, multiple cameras are used, or alternatively a camera within a camera. One example of a camera within a camera is a camera with normal capabilities and with 3D scanning features. High camera resolution and high frame rate are desirable, such as 60 frames per second (fps) or 120 fps.

The system uses a specialized operating system or real-time operating system to increase the processing speed of the system to reduce frame loss. In one example, calculation delay is less than 50 milliseconds. The system assumes that human reaction time is between about 100-200 milliseconds. A large RAM memory is used to allow buffering and concurrent calculation from multiple servers.

Many types and qualities of data can be extracted from recordings of the panel discussion. The following are examples, though are not exhaustive:

Gaze Analysis: Eye movement is calculated in a two-dimensional image space with a number of key data points. A first data point is the maximum observed distance of movement within an elliptical or circular space of the eye shape. The analysis considers a wide variety of possible eye shapes when making this assessment. A second data point is the acceleration of eye movement based on the distance traveled and camera frame rate consistent with dispersions of the frame and network jitter. This value can be calculated in both the X and Y axes. Eye pixel positions will be captured at 100- 200 millisecond intervals to determine the pixel per second acceleration rate of eye movement.

In order to measure the gaze analysis data points, in one example, all frame sequences are recorded in a buffer, then frame sequences are analyzed to look for a frame sequence with an eye gaze movement. In one example, each frame is separated in time from an adjacent frame by 100 milliseconds and the frame analysis reviews an eye gaze position at points A,

B, C and D. If ray AB and AC are in the same direction, then ray ABC is a frame sequence with an eye gaze movement. Ray AD is examined to see if it is in the same direction as ray ABC. If yes, the frame showing point D is added to the frame sequence ABC to create frame sequence ABCD, to be the frame sequence with the eye gaze movement. If no, the system determines that the frame showing point D may be the beginning of a new frame sequence with an eye gaze movement. When a frame sequence with an eye gaze movement is identified, the timestamp of that frame sequence is recorded.

A blinking analysis will be performed, with a baseline calculation of average blink intervals. A total duration of the time that eyes are detected during the entire filming period is calibrated for network error and using the blinking analysis data.

For eye gaze tracking, both image detection and motion prediction can be used, with statistical modeling as a baseline. In some examples, it is assumed that a person is unlikely to look at extreme angles without turning their head because it is typically uncomfortable to do so. Thus, the system can narrow down where the gaze is likely to be.

A machine learning model can be created to consider additional personal/behavioral traits and contextual attributes to create additional eye gaze and head movement prediction. This is a model useful to calculate relevant participation scores, as well as empathy, engagement, organizational hierarchy, deference, culture, knowledge depth, emotional quotient, and intelligence quotient scores.

Attitude toward Speaker: Based on eye shape, head tilt, and/or upper body posture, individuals can also be evaluated for whether they are curious about what the speaker is saying or dismissive about the content. If the listener’s head or body posture is tilted forward with raised eyebrows, the listener is assumed to be curious and engaged. If the listener is leaning back with folded arms and a furrowed brow, the listener is assumed to be skeptical of the content.

Magnitude of Speaker’s Hand Gestures and Effect on Listener: The location of the speaker’s hand or hands can be measured as a distance H from the center of the speaker’s body. The focal point of the listeners can be observed, recorded, and measured for velocity of change of the focal point, V, during the time of the hand gesture (T_G) between Estimated Gaze Focus Beginning (EGFB) and Estimated Gaze Focus End (EGFE). EGFB and EGFE can be expressed as a distance as measured from the center of the speaker. The velocity of change of the focal point can be expressed as:

V = (EGFE - EGFB)/ T_G

For a particular listener, the system can calculate a coefficient of focus (CoF):

CoF = (H / Max(H)) * V

For a particular speaker time, the sum of the CoF values for each of the listeners can be calculated. The system can draw a preliminary conclusion that a listener with the largest sum for a particular speaker time is the panel member who is most affected by the speaker. The system can also make an estimate regarding whether the effect of the speaker on a listener was positive or negative. Using the center of the speaker as an axis, if the absolute value of EGFE is less than the absolute value of EGFB, then the listener looked toward the center of the speaker during the hand gesture. This is assumed to indicate a positive effect of the speaker on the listener. If the absolute value of EGFE is more than the absolute value of EGFB, then the listener looked away from the center of the speaker during the hand gesture.

The hand gesture values can be recorded to a temporary database as the system loops through the video file. The temporary database can then be evaluated to find a maximum.

Torso Motion in Relation to the Speaker: Torso motion indicators may also be analyzed to indicate if a listener turned towards or away from the speaker. A distance from the camera to the outer edges of the upper torso, commonly the shoulders, is assigned to a variable for each participant. From the perspective of the camera, the participant humans are identified and labeled as PHI, PH2, etc. An edge detection algorithm subroutine is run to identify the outer edge of the participant’s torso in a recording of the discussion, which could be a recording from a depth sensor such as an IR camera or a recording from a video camera. The locations for the outer edges of each participant are tracked in a database. For example, coordinate location values for the left shoulder of a first participant at a given frame can be recorded as PH1LSX, PH1LSY, and PH1LSZ, corresponding to the x-, y-, and z-axes. Coordinate location values for the right shoulder of a first participant at a given frame can be recorded as PH1RSX, PH1RSY, and PH1RSZ. The system can identify a line between the coordinates for the left shoulder and the coordinates for the right shoulder record, and then identify a twisting motion of the torso if the z-axis values change.

If the speaker is on the right of the listener, and if the left shoulder of the listener comes closer to the camera than the right shoulder, then the system can conclude that the listener is pivoting towards the speaker and is interested in what the speaker is saying. If the speaker is on the right of the listener, and if the left shoulder of the listener is farther away from the camera than the right shoulder, or the right shoulder comes closer to the camera, then the system can conclude that the listener is relaxing their attention towards the speaker. Further, the data regarding indication of interest in the speaker can be weighted as to whether a plurality of the listeners react similarly, and how often. Each data point can be measured and scaled for the absolute size of the participant’ bodies. The inverse position data can be graphed for the left and right of the speaker. Using these techniques, data conformity can be measured. From panel discussion to panel discussion, median ranges can be established by question, or by relative time period of the discussion. This data can be particularly measured, analyzed and compared for specific portions of the panel discussion, such as for the greeting or the farewell portions.

Emotional Analysis Using Eye Movements: The natural eye shape and eye size of individual participants will be calculated. Variations for individuals will be determined. Eye shape and gaze position is also analyzed in relation to other body parts and external inputs. One example of an external input to eye movement is what the participant is looking at. Eye movement will also be analyzed in relation to external noise, which may draw the participant’ s gaze and attention.

Emotional Analysis Using Lip Movements: Movement detection of the lips is similar to determining the shape transformation of the eyes. Lip movements will be analyzed and used as input for emotion analysis.

Calibration Using Neck Position: Neck position will be used as a reference for determining head movement and eye movement. Head and eye motion will be adjusted based on known neck position, e.g., adjusting the eye gaze detection when a participant looks toward or away from a computer screen or another participant.

Emotional Analysis and Calibration Using Shoulder Position and Motion: A participant’ s shoulder position is analyzed in relationship to other body parts in a three- dimensional space using multiple depth sensors.

Emotional Analysis and Calibration Using Spinal Position: Spinal position affects shoulder and hip movement which affects head and foot position. Analysis of these other measurements will be performed and all emotional and engagement states.

Calibration Using Hip Position: Hip position affects the location of the feet, similar to how shoulder position affects the location of the head and neck. Upper body movement tracking will be performed down to the hip and chair.

Analysis of Leg and Knee Position: Leg and knee position will be analyzed to detect whether legs are open or closed. An analysis will also be performed to determine whether the legs are crossed or uncrossed. Changes in the position of the legs and knees will also be timed for duration.

Analysis of Foot Position: When the participant is in a seated position, image and motion data from the waist down will be obfuscated, with the exception of foot movement and position. When the participant is sitting, foot position will be analyzed to determine whether the feet are far apart or close together. The position and direction of the participant’s toes will be determined.

The position of the feet affects posture because it translates to torso lean. Shifting the feet to support the weight is an indicator of forward torso lean. This is a confirming indicator that reinforces the magnitude of body posture measurements. For example, a participant might tuck their feet under the chair. The observed range of motion for torso lean in such a circumstance is less than the maximum that the system could expect to see when a candidate’s feet are planted forward.

Analysis of Arm and Elbow Position: Arm and elbow position will be analyzed to detect whether arms are open or closed. An analysis will also be performed to determine whether the arms are crossed or uncrossed. Changes in the position of the arms and elbows can be timed for duration of crossing and uncrossing.

Analysis of Hand Position: Hand motions and gestures, including sign language, can be recorded and analyzed. Emotional Analysis and Calibration Using Chair Movement Combined with

Seating Position: When the individual is seated, the system analyzes chair movement. For chairs that swivel, for example, the system measures how often the chair is swiveled, and the degree of movement. The chair position is also used in relation to the hip, foot, leg, arm, hand, and head position analysis. Extraction of Data from Recorded Audio

Speech to text and audio analysis are used to detect the substantive content of the panel discussion, as well as the emotional investment of the participants. The speaker's words can be analyzed for overall verbal complexity. For example, verbal complexity can be measured by calculating a reading level and an average number of syllables per minute. Extracted text can be used to determine relevance to open job positions, if the speaker is a candidate for employment.

Event Modules

In some examples, the system uses event modules to identify events recorded during the panel discussion that will be analyzed in the evaluation modules. The event modules are run on recorded data to look for events known to be related to inputs for the evaluation modules. One example of an event is an answer from an individual participant to a question. Another example of an event is when two participants are speaking at the same time, which may be characterized as an interruption. Another example of an event is a change in an individual’s body posture.

An interaction is identified as being possibly relevant to a participant evaluation for a particular participant. Later, a quantification of this interaction can be used as part of the input for participant evaluation. There can also be intermediate algorithms for deciding whether the “possibly relevant” interactions should be used as input. For example, the system may be structured to use the best four answers from a participant as input to its behavioral analysis. In this scenario, the system will input all of the answers into a first algorithm to choose which answers count as the “best.” The selected best answers are then used as input into the evaluation algorithm (e.g., candidate score, candidate ranking, employee score, group interaction score). There are predetermined criteria for whether a participant’s interaction is “good,” “bad,” or “other.”

Event Identification Module

In some examples, the system can include an event identification module. The event identification module can be configured to analyze, identify, and detect specific acoustic events as they happen. In some examples, the system can use distributed intelligence, such as a probability matrix that can assign and identify events for processing. Sequence-based modeling can build out the matrix. In some examples, a plurality (such as five or more) microphones and the sensor array are collecting data, and parallel processing of different algorithms can be used to speed up the processing time.

Given the plurality of microphones, the approximate location of the speaker can be reliably determined by ordering which microphone is providing the strongest input.

In a first step, the system can determine which speaker corresponds to which audio sample. The audio samples can be identified as starting and stopping when the audio input from the same microphone is dominant or strongest. In some examples, to prevent attributes with great numeric range from skewing the data away from the smaller numeric ranges, the system can average the input of the training set for each attribute. In effect, this removes outliers that would skew the data. For example, in some examples, the system can receive an input of an audio file. The audio file can be converted to a text file, such as by using a natural language processor or a speech to text processor. The system can remove stop words, as discussed herein, resulting in a plurality of words for further analysis, in some cases commonly referred to as a “bag of words.” The “bag of words” can be analyzed for an attribute, such as sentiment. In various examples, each of the words in the “bag of words” can be assigned a value or a vector. Similar words (e.g. princess and queen) can be assigned similar values, whereas very different words (e.g. jump and valve) can be assigned very different values. The words can be shown visually, such as using t-distributed stochastic neighbor embedding (t-SNE), which is a localized hashing mechanism to map the keywords graphically and calculate distances between words in the “bag of words”. Other high- dimension mapping algorithms can be used. In some examples, words can be grouped and averaged. In some examples, words that would skew the average in an undesirable way can be removed. In some examples, technical words, which can have large values or vectors assigned to them, can be removed from the analysis, such that they do not overly impact the results.

In a second step, the system can confirm the identification of the speaker in the first step. In some examples, according to facial biometric analogies, it can be known that there is a ratio between the inter-ocular distance and the distance between the eyes and mouth. Thus, once the system knows the position of the eyes from the gaze detection method, which can be recorded as (xl, yl) and (x2, y2), the system can locate the mouth center when the speakers are looking straight ahead. Further, upon rotation, the system can calculate where the mouth is via the head rotation from the gaze detector. For example, head rotation can be + or - Q degrees, the mouth at X, Y where X = (xl + x2)/2 +/- (distance x sin(Q)), Y = (yl + y2)/2 +/- (distance x cos(Q)). If there is no confirmation, the sample can be excluded from further processing.

In a third step, the system can identify what type of event is being identified. For example, the system can determine if one of the three scenarios is occurring:

A. Two-Speaker Dialog

B. Multi- Speaker Dialog

C. Hybrid event - such as audio silence, unintelligible speech, environmental sounds, multiple dialog misclassified as two-speaker dialog. In some examples when hybrid events are identified, the event is not pursued, except for silence.

In a fourth step, the end of sample or segment is identified, such as by identifying a pause in the audio. In some examples, the cycle of steps one through four can be repeated, if applicable. In various examples, another issue related to audio event identification can be facing environment changes from studio to site. In situation where the system is properly trained with real data recorded, various real-world environments can result in some situations beyond the ones observed in previous experiences. In some instances, the occurrence of sound events from the surrounding environment, if used to train the system, may lead to lower accuracy. Variability in the Signal to Noise Ratio (SNR) in the raw acoustic signal recorded by the sensors can be a major issue. Therefore, in some examples, a threshold can be included to limit the samples processed in order to avoid false positives. In various examples, during testing, the threshold can be when more than one person is moving and/or speaking. As the system progresses, the threshold of more than one person moving or speaking can be removed or lowered to cacophony levels, such that fewer events are blocked from consideration. In some examples, this can also help reduce the demand on the CPU and other parts of the system.

The universe of audio events that can be detected in order to observe potential overlaps between different events. In some examples, the system can use a classifier able to identify more than one coexisting event at a time. In this regard, algorithms providing more than one output that rely on the audio event probability of occurrence can be included.

Reliability of the audio event estimation can be key. Finally, it can be assured that the data algorithm is trained using enough audio that ensures enough diversity for each category of sound events to be identified.

Speaker Interruption Engine

There may be many times throughout the discussion that two participants will talk simultaneously. This data can be used as part of the analysis, even if the voices of the two participants are not individually distinguishable. In practice, it can be difficult to contextually analyze two voices speaking over each other in an audio data input stream. In some examples, it may be more practical to do one of the following. Where the voices of two participants are overlapping, the system creates a decision point, with these options:

1. Exclude that portion of the file from statistical analysis.

2. Define a variable, INTERRUPTION, and assign it a +1 for each time a targeted individual was speaking, and a different individual also spoke and continued speaking for three seconds beyond the cessation of speech by the targeted individual.

3. Define a variable, ACCOMODATION, and assign it a +1 for each time the targeted individual was speaking and another individual also spoke and continued to speak, and the targeted individual did not speak for a minimum period of time, such as at least 30 seconds.

4. Assign the variable in #3 a more positive connotation for teamwork and/or willingness to listen.

5. Define a variable, ASSERTIVENESS, and assign it a +1 for each time the targeted individual was speaking and another individual also spoke, and then the targeted individual continued speaking for a minimum period of time, such as at least 30 seconds.

6. Look exclusively at the body language during that section. Look for evidence from the body language data that the candidate is inviting an interruption, is collaborating, or is sharing the floor.

In some examples, visual data can be used to determine which individual is speaking or which individuals are speaking. For example, lip movement recorded by a video camera or depth sensor can be used instead of parsing the audio track to determine when one person is speaking over another.

In any case, the system still analyzes the body language of both individuals if there was an overlap to speech to determine how each participant reacts to the interruption. Tone of voice and body language contribute significantly to personal communication, in some cases much more than the content of the words. The system can mark and identify interruptions and generate a reference file. The reference file would stop and start video playback at a time interval such as 20 seconds before and 20 seconds after an interruption.

To generate a model for the interruption engine, a sample of interruptions are evaluated by individuals. The interruptions are presented in a random order. The individuals each generate a qualitative description of the interruption to determine a consensus for each individual sample interruption. For interruptions in which the evaluators disagree, the interruptions are presented for evaluation by a second set of individuals known to have high emotional intelligence. After this process is finished, the samples size can be, for example, on the order of 1,000 sample interruptions. The samples can be shrunk down to clips of 2-3 seconds before and after the time stamp of the event. The samples are each fed into a machine learning engine. The machine learning engine can generate a model to assign a qualitative assessment of the individual who was the source of the interruption, and the individual who was interrupted while speaking.

FIG. 6 shows a schematic view of audio streams captured during recording panel discussions and their use in flagging interruptions. In various examples, a plurality of participants can participate in a group interview or discussion simultaneously. The participants 502 are shown on the left side of FIG. 6 as 1, 2, 3, 4, and 5. The participants 1, 2, 3, 4, 5 can be asked a first question Q1 and a second question Q2. A group interview or discussion can involve just one question, two questions, three questions, four questions, and other number of questions. The participants 502 can provide responses or answers to the questions. The answers can be uniquely attributed to the participant that provides the answer. For example, FIG. 2 reflects that A 1 is an answer that was provided by participant 1, and A3 is an answer that was provided by participant 3. The system can record who provides an answer. In some examples, each participant can have an assigned microphone. If the microphone assigned to participant 1 provides the input audio to the system, it can be assumed that participant 1 answered the question.

If an answer provided by one participant overlaps in time with answer provided by a second participant, an interruption I can be recorded. The interruption I can be attributed to the participant that provides an answer while a previous answer is still being provided. For example, FIG. 6 attributes the first interruption 12504 to participant 2, because participant 2 did not let participant 3 finish answer A3 506 prior to providing answer A2 508. It can further be seen in FIG. 6 that participant 2 interrupts participant 4 providing answer A4510 at interruption 12512. Participant 4 also interrupts participant 2 at interruption 14514. As discussed above, the number of interruptions for each participant can be counted and tracked. Further, other types of behavior can also be tracked, such as the possible assertiveness of participant 4 interrupting participant 2 after participant 2 interrupted participant 4.

In some examples, the number of interruptions by a participant can be counted and tracked. If the number of interruptions exceeds a threshold the participant can be removed from the discussion, such as by disabling the microphone assigned to that participant. In some examples, the threshold can be a designated number of interruptions, such as two, three, four, or five interruptions. In some examples, the threshold can include a ratio of interruptions and a time period, such as two interruptions in five minutes or two interruptions in three questions.

FIG. 7 shows a schematic view of audio streams captured during an alternate system for recording panel discussions and their use in flagging interruptions. In some examples, a single microphone can be used by multiple participants. Each horizontal row in FIG. 7 represents a single microphone shared by two participants. As such, it can be known which participant the answer (or interruption) should be attributed to based solely on which microphone provided the audio stream. In the example shown in FIG. 7, one microphone is used for a pair of participants, such as one microphone for participants 1 and 2, one microphone for participants 3 and 4, one microphone for participants 5 and 6, one microphone for participants 7 and 8, and one microphone for participants 9 and 10. In contrast to FIG. 6, the system cannot attribute the input from one microphone to a specific participant, since more than one participant can be using the same microphone. In some examples, two participants that are using the same microphone can engage in an interruption event. For example, as shown in FIG. 7, participants 3 and 4 are sharing a microphone. Participant 4 can be providing an answer A4602 to a question Ql, when participant 3 interrupts 1604 participant 4 with answer A3 606.

In some examples, audio analysis of the input audio can be used to determine which participant is talking during a specific segment. In some examples, gesture recognition analysis can be used to determine which participant is talking.

Flagging Events

The system can monitor for interaction events on-the-fly, or data can be stored and later analyzed to determine events. When scoring participants, the system uses the context of an event to evaluate a participant’ s response. The system searches for particular predetermined types of events in the panel discussion recordings. Events are associated with particular time segments in the recording. When an event is found, participant data that was recorded around the time of that event is analyzed. The system looks for particular signatures of events. The relevant data to be analyzed can be simultaneous with the event, or it can occur just before or just after the event, depending on which type of event is being evaluated and how it is evaluated. Evaluation Modules

Conversation Contribution Score Module

In various examples, the system can determine the depth and extent of a candidate’s substantive knowledge of the subject matter that they talked about during the discussion. In various examples, the system can determine the depth and extent of the candidate’s knowledge based on keywords, and in some example, the system can score or rank candidates, such as to compare candidates to each other or to a threshold.

In some examples, the system can determine the candidate substantive knowledge by extracting keywords from spoken word data, such as discussed above in relation to FIG. 5. In various examples, the system can optionally use behavioral data relevant to the degree to which the candidate was actively participating in conversation, e.g. length of time speaking, speed of speaking, and/or timeliness of responses.

In some examples, the system extracts list of keywords from the candidate’s audio using speech to text processing. The system can remove stop- words from the candidate’s list of key words, as discussed above. Using the keyword data set (in relation to FIG. 5, above), the system can evaluate the candidate’s extracted keywords. For example, the system can perform statistical analysis of how many keywords were used. The system can analyze how many high-quality keywords are used.

In some examples, the system can analyze a portion of the questions that candidates have answered. For example, if a candidate has answered a first number of questions, a user can select the top second number of answers, or a subset of the questions. If a candidate spoke 10 times, the system could first analyze each of the response against each other to determine which four questions were answered the best, such as by assessing the answers based on the most high-quality or relevant keywords. The system could use each candidate’s top four answers to score candidates relative to each other, rather than using all answers that were collected.

In some examples, the system can produce a score for a candidate. The score can be representative of how much substance the candidate brought to the discussion. For example, if the candidate used many relevant keywords in answers, the candidate will receive a better score. In contrast, if there are fewer high-quality or relevant keywords used, the candidate will receive a worse score. In some examples, the score can be numerical, such as a number between 1 and 10, or a number between 1 and 100.

The system can further rank or organize the candidates against each other. In some examples, the ranking can be based solely on the score the candidates receive from the Conversation Contribution analysis. Ranking the candidates can allow a reviewer, such as a hiring employer, to easily identify which candidate had the highest quality of input to the conversation. In some examples, ranking the candidates can allow the reviewer to easily identify which candidates are above a threshold. <img file="WO2021108564A1_D0001.tif" />

In various examples, the system can include a participation score module. The participation score module can be configured to provide an employer a score relating to the candidate’s participation in the group dialog or in response to a question proposed to a candidate. The employer can use the participation score to determine which employees or candidates did not do well in the group dialog or in response to a question. The individuals that did not do well can be targeted for education or other options to improve the individuals.

The participation score module can be adapted to the employer’s preferences. For example, if the employer is hoping to see lots of participation, physical evidence of engagement, minimal interruptions, or responses to more substantive questions, than the participation module can be configured to output scores related to the employer’s desires.

In some examples, an individual’s time- synchronized biometric data from during the discussion can be input into the module. The module can determine events that happen during an interview in which the employer wants to gauge the individual’s participation, such as interruptions or when someone else is speaking. The system can analyze the behavioral data of the individual while the event was occurring. In some examples, this can be repeated for each of the individuals in the interview.

The system can output a score for each individual. The score can represent how well the individual participated in the discussion. In some examples, a threshold can be set for a minimum number of interactions. For example, an individual must interact at least three times to get a meaningful score. In some examples, the system can predict different outcomes of an event and create a model where, if a specific individual does a defined action/response, then the individual would get a higher or better score than if the individual did an alternative action/response. The system can flag or note positive or negative interactions in a database.

In some examples, an audio recording of the group dialog or of a response to a question can be processed to determine a participation score. In an example, the audio file can go through a two-step process to evaluate the content of the audio. In a first step the audio can be converted from audio to text. The stop words in the text file can be removed, such as described above. This step results in a text file without any of the stop words. In the second step, the text file can be analyzed for content. In some examples, the text file can be analyzed to measure average number of syllables and/or measure the complexity of words used. The analysis can provide a rating based on the language used. In some examples, the system can output a reading level in addition to a participation score. Database Server for a Recorded Panel Discussion System

The system records multiple types of data inputs. In one example, a digital video camera produces a video input, a microphone produces an audio input, and one or more behavioral data sensors produce behavioral data input.

The system can also receive text data input that can include text related to the panel participants. And as described above, the system can extract keywords from the audio recorded of the individuals in the panel discussion.

Participant Database for a Recorded Panel Discussion System The system can include a participant database stored on the participant database server. Data streams from the microphone 123, video camera 133, and behavioral data sensor 143 can all automatically be associated with a participant profile for the individual 103 in the participant database. In some examples, the participant database can have profiles for identified individuals. In some examples, the participant database can create anonymized profiles for individuals in the panel discussion.

As used in this specification and the appended claims, the singular forms include the plural unless the context clearly dictates otherwise. The term “or” is generally employed in the sense of “and/or” unless the content clearly dictates otherwise. The phrase “configured” describes a system, apparatus, or other structure that is constructed or configured to perform a particular task or adopt a particular configuration. The term “configured” can be used interchangeably with other similar terms such as arranged, constructed, manufactured, and the like.

All publications and patent applications referenced in this specification are herein incorporated by reference for all purposes. While examples of the technology described herein are susceptible to various modifications and alternative forms, specifics thereof have been shown by way of example and drawings. It should be understood, however, that the scope herein is not limited to the particular examples described. On the contrary, the intention is to cover modifications, equivalents, and alternatives falling within the spirit and scope herein.