Brain computer interface (bci) system based on gathered temporal and spatial patterns of biophysical signals

BACKGROUND

0001Communication and control of the environment is important to everyday life. In particular, disabled people have gone to extraordinary lengths to communicate. A Brain Computer Interface (BCI) enables direct communication between the brain and a computer or electronic device. BCI could also be applied in applications where existing communication methods exhibit shortcomings, e.g. in noisy industrial applications, military environments where stealth and movement are constrained, etc. In the consumer market. BCI may provide advantages as a gaming or entertainment interface or may speed existing or enable entirely new computer-user interactions.

0002Regardless of function, each part of the brain is made up of nerve cells called neurons. As a whole, the brain is a dense network that includes about 100 billion neurons. Each of these neurons communicates with thousands of others in order to regulate physical processes and to produce thought. Neurons communicate either by sending electrical signals to other neurons through physical connections or by exchanging chemicals called neurotransmitters. When they communicate, neurons consume oxygen and glucose that is replenished through increased blood flow to the active regions of the brain.

0003Advances in brain monitoring technologies allow observations of the electric, chemical, fluidic, magnetic, etc. changes as the brain processes information or responds to various stimuli. Research continues in brain computer interface (BCI) systems that could provide new communication and control options for a wide variety of users and applications. Through brain activity monitoring, a database of characteristic mental profiles may be gathered.

0004Device and data security threats are ubiquitous and a very high value has been assigned to highly accurate and precise authentication systems and methods. While several forms of biologically distinct signatures or biometrics are employed in authentication systems (fingerprints, retinal patterns, voice characteristics, etc.), there is very little exploitation involving the uniqueness in brains as an authentication technique.

0005Some BCI systems rely on electroencephalography (EEG), which is characterized by relatively high temporal resolution, but also by relatively low spatial resolution. Further research in BCI systems is underway addressing the many questions and challenges involving their reliable use.

BRIEF DESCRIPTION OF THE DRAWINGS

0006<figref idref="DRAWINGS">FIG. 1</figref> illustrates a system for providing psychological and sociological matching through correlation of brain activity similarities according to an embodiment;

0007<figref idref="DRAWINGS">FIG. 2</figref> illustrates a system that compares the brain anatomy and activity to known information and prerecorded signatures to identify and assign a probability of uniqueness to an individual, according to an embodiment;

0008<figref idref="DRAWINGS">FIG. 3</figref> illustrates a consumer grade wearable system for gathering BCI input, according to an embodiment;

0009<figref idref="DRAWINGS">FIGS. 4<i>a</i>-<i>c </i></figref>illustrate components of a neuroimaging device, according to an embodiment;

0010<figref idref="DRAWINGS">FIG. 5</figref> is a flowchart of a method for providing control of computing experiences according to an embodiment:

0011<figref idref="DRAWINGS">FIGS. 6<i>a</i>-<i>b </i></figref>illustrate a visual search according to an embodiment;

0012<figref idref="DRAWINGS">FIG. 7</figref> illustrates a BCI system for providing telepathic search according to an embodiment;

0013<figref idref="DRAWINGS">FIG. 8</figref> illustrates wireless telepathic communication using a BCI system according to an embodiment;

0014<figref idref="DRAWINGS">FIG. 9</figref> is a networked system for performing telepathic contextual search according to an embodiment;

0015<figref idref="DRAWINGS">FIG. 10</figref> is a flowchart for providing telepathic augmented reality using a BCI system according to an embodiment;

0016<figref idref="DRAWINGS">FIGS. 11</figref> a-d show an example of AR presentation and control according to an embodiment;

0017<figref idref="DRAWINGS">FIG. 12</figref> illustrates a system that may provide telepathic augmented reality according to an embodiment;

0018<figref idref="DRAWINGS">FIG. 13</figref> illustrates a cortical representation of visual space used to represent mental desktop space provided by a BCI system according to an embodiment;

0019<figref idref="DRAWINGS">FIG. 14</figref> is a diagram of the optic system according to an embodiment;

0020<figref idref="DRAWINGS">FIG. 15</figref> shows the physiologically segregated sections of visual space according to an embodiment;

0021<figref idref="DRAWINGS">FIG. 16</figref> is a model of a human cortex showing the primary motor cortex and the primary sensory cortex as utilized by a BCI system according to an embodiment;

0022<figref idref="DRAWINGS">FIG. 17</figref> is a model demonstrating the topographical organization of the primary motor cortex on the pre-central gyrus of the human cortex;

0023<figref idref="DRAWINGS">FIG. 18</figref> illustrates a user interface for assigning BCI measures and other modalities to applications according to an embodiment;

0024<figref idref="DRAWINGS">FIG. 19</figref> shows a BCI system that accepts BCI inputs and other modality inputs according to an embodiment:

0025<figref idref="DRAWINGS">FIG. 20</figref> illustrates a flowchart of a method for determining user intent according to an embodiment;

0026<figref idref="DRAWINGS">FIG. 21</figref> is a flowchart of a method for assigning BCI input for controlling an application according to an embodiment;

0027<figref idref="DRAWINGS">FIG. 22</figref> is a flowchart of a method for adjusting contextual factors by the BCI system according to an embodiment; and

0028<figref idref="DRAWINGS">FIG. 23</figref> illustrates a block diagram of an example machine for providing a brain computer interface (BCI) system based on gathered temporal and spatial patterns of biophysical signals.

DETAILED DESCRIPTION

0029According to embodiments described herein, brain/skull anatomical characteristics, such as gyrification, cortical thickness, scalp thickness, etc., may be used for identification/authentication purposes. Measured stimuli/response brain characteristics, e.g., anatomic and physiologic, may be translated into specific patterns to categorize a brain for identification and/or authentication purposes. People may be correlated according to similarities in brain activity in response to stimuli. Information on other brain signatures, e.g., anatomic and physiologic, and comparisons to similar brains may be used to predict a brain response to a new stimulus and for identification and/or authentication purposes. Brain identification and/or authentication techniques in combination with other identification and/or authentication techniques, e.g., password, other biometric parameters, may be used to increase the identity/authentication sensitivity and specificity.

0030More recent technological advances in sensing brain or neuronal activity signal present opportunities for the creation of more sophisticated BCI usages and systems. Based on gathered temporal and spatial patterns of biophysical signals, it is now possible to measure and identify psychological states or mental representations of a person. For example, temporal and spatial patterns of biophysical signals may be obtained through, but not limited to, electrical, fluidic, chemical, magnetic sensors.

0031Examples of devices that gather electrical signals include electroencephalography (EEG). EEG uses electrodes placed directly on the scalp to measure the weak (5-100 μV) electrical potentials generated by activity in the brain. Devices that measure and sense fluidic signals include Doppler ultrasound and devices that measure chemical signals include functional near-infrared spectroscopy (fNIRS). Doppler ultrasound measures cerebral blood flow velocity (CBFV) in the network of arteries that supply the brain. Cognitive activation produces increases in CBFV within these arteries that may be detected using Doppler ultrasound. fNIRS technology works by projecting near infrared light into the brain from the surface of the scalp and measuring optical changes at various wavelengths as the light is refracted and reflected back to the surface. The fNIRS effectively measures cerebral hemodynamics and detects localized blood volume and oxygenation changes. Since changes in tissue oxygenation associated with brain activity modulate the absorption and scattering of the near infrared light photons to varying amounts, fNIRS may be used to build functional maps of brain activity. Devices that measure magnetic signals include magnetoencephalography (MEG). A MEG measures magnetic fields generated by the electrical activity of the brain. MEG enables much deeper imaging and is much more sensitive than EEG because the skull is substantially transparent to magnetic waves.

0032Through the use of biophysical sensor devices, as described above and others, temporal and spatial patterns of biophysical signals may be used to measure and identify psychological states or mental representations of a person to reveal information, such as cognitive workload, attention/distraction, mood, sociological dynamics, memories, and others. Utilization of these data unlocks a new frontier for opportunities in human-machine interaction.

0033Businesses, political institutions, and society place a high value on finding like-minded people for purposes that include marketing, messaging, and social networking. Systems and techniques that may assist in the identification of like-minded (or conversely, unlike-minded) people would accordingly be highly valued. Information on regarding how brains respond to similar stimuli may be used to ascertain and score the “mental similarity” between different people or groups of people. This may be used in conjunction with other measures of mental, personality, or sociological traits to create more sophistication to the matching assessment.

0034By instrumenting people with brain monitoring devices and then taking them through a series of imagined or thought-provoking experiences (through any imagined sensory channel, visual, auditory, tactile, etc.), the resulting spatial and temporal brain activity patterns may be captured and characterized. The degree to which the brain activity of different people responds similarly would provide a measure of mental similarity. Conversely, the degree to which the brain activity of people responds dissimilarly to the same stimuli would provide measures of mental dissimilarity. The collection of responses to the set of stimuli may be used to build characteristic mental profiles and serve to establish models of mental predilections that would be equivalent to concepts, such as the Meyers-Briggs® or Five Factor Model (FFM), for characterizing personality traits.

0035The political, mental, or social compatibility (or incompatibility) or people may be predicted using temporal and spatial patterns of biophysical signals and stimuli response data based on the theory that certain similarities or correlations of mental responses make for better pairings. This comparison to others could happen through a web-based system and infrastructure to be used as part of dating services, e.g., Match.com®.

0036Specific examples where this idea may prove useful include a comparison of mental profiles with job satisfaction information to help identify and predict potential career matches (or mismatches), with relationship satisfaction to help identify and predict potential social compatibility (or incompatibility), with political inclinations to help identify and predict political party alignment (or misalignment), with product usage, satisfaction, or interest to help identify marketing targets, etc.

0037<figref idref="DRAWINGS">FIG. 1</figref> illustrates a system <b>100</b> for providing psychological and sociological matching through correlation of brain activity similarities according to an embodiment. The system collects spatial and hemispherical information to define inputs to a BCI system. A library of stimuli <b>110</b> is provided to elicit brain activity in subjects <b>112</b>, <b>114</b>. The library of stimuli <b>110</b> include sets of stimuli involving any of various media designed to engage the brain, e.g., pictures, films, audio tracks, written or spoken questions or problems. The stimuli options are numerous and a library of specific stimuli <b>110</b> foci is imagined that could range from the general, e.g., designed to test emotional sensitivity, to the very specific, e.g., orientations on family and child-rearing. The brain activity is recorded by a data collection and recording system <b>120</b>, <b>122</b> as the stimuli are presented to the subjects <b>112</b>, <b>114</b>. Neuroimaging devices that measure the resulting brain activation patterns may include EEG, fNIRS, MEG, MRI (magnetic resonance imaging), ultrasound, etc. However, embodiments are not meant to be limited to measurement systems specifically mentioned herein.

0038The recorded brain activity is then processed using a pattern recognition and classification system <b>130</b>. A pattern recognition system <b>130</b> processes the recorded brain activity to characterize and classify the brain activation pattern. There are numerous techniques and algorithms from the field of pattern recognition that may be applied including classification, clustering, regression, categorical sequence labeling, real-valued sequence labeling, parsing, Bayesian networks, Markov random fields, ensemble learning, etc. Further methods for obtaining or analyzing brain activity: modified Beer-Lambert Law, event-related components, multi-voxel pattern analysis, spectral analysis, the use of MVPA (multi-voxel pattern analysis) on fNIRS, etc. Simple brain EEG signal characterization may be used for identification purposes. General purpose pattern recognition techniques and algorithms may be implemented. The pattern and classification results <b>132</b> are combined, at a mental profile modeling system <b>140</b>, with personal data and other traits from a database <b>150</b> to develop mental profile models of the subjects <b>112</b>, <b>114</b>. The mental profile modeling system <b>140</b> thus creates a model that combines the brain pattern recognition results with other personal data and traits, such as gender, age, geographic location, genetic information, etc., to build a mental profile as a function of the specific stimuli.

0039The personal data and other traits from database <b>150</b> may be obtained through questionnaires, observation, etc. and maintained in a personality trait database. The mental profile modeling system <b>140</b> produces a mental profile match of a subject by comparing the mental profile of a subject with a database of other mental profiles. A mental profile analysis system <b>160</b> correlates probabilities between subjects. The mental profile analysis system <b>160</b> calculates the statistics and probability of mental match for any of a range of topics, e.g., social, problem solving, music genre affinity, financial orientation, etc. General purpose statistical techniques may be used to translate pattern recognition into probabilistic relationships given known conditions.

0040Accordingly, the system <b>100</b> translates recorded brain activity patterns in response to stimulus <b>110</b> into a characteristic mental profile for that stimulus. A library of stimuli <b>110</b> is translated into a library of mental profiles for each individual. The mental profiles also include the integration of personal data and traits from database <b>150</b>. The mental profile analysis system <b>160</b> derives the similarity or dissimilarity of mental profiles based on the degree of similarity or dissimilarity of pattern matching results between two people for a stimulus or set of stimuli. This result, a mental profile match result <b>170</b> represents a probabilistic score of a “mental match.”

0041BCI systems may also be used to provide user identification and authentication using brain signatures. Everyone has unique brain anatomical model and physiological characteristics that are a function of their genetic, environmental, and situational influences that can be leveraged for identification and authentication purposes.

0042<figref idref="DRAWINGS">FIG. 2</figref> illustrates a system <b>200</b> that compares the brain anatomy and activity to known information and prerecorded signatures to identify and assign a probability of uniqueness to an individual, according to an embodiment. Accordingly, the brain anatomical model and physiological uniqueness may be used for security and/or authentication purposes. In <figref idref="DRAWINGS">FIG. 2</figref>, a calibration process <b>202</b> and an authentication process <b>204</b> based on brain responses to activity are shown. In the calibration process <b>202</b>, stimuli <b>210</b> are provided to a first subject <b>212</b>. The stimuli <b>210</b> may include images, statements, or questions that are presented to a user that would incite certain characteristic brain activity responses. The response of the subject is measure by data gathering and recording system <b>220</b>. The data gathering and recording system <b>220</b> also measure activity associated with brain anatomy and activity. Neuroimaging devices that measure the resulting brain activation patterns may include EEG, fNIRS, MEG, MRI, ultrasound, etc.

0043The data associated with brain anatomy and activity is provided to a pattern recognition device <b>230</b> that analyzes the associated with brain anatomy and activity to identify patterns. Anatomic characteristics of the brain. e.g., gyrification, cortical thickness, etc., are identified. Again, there are numerous techniques and algorithms from the field of pattern recognition that may be applied including classification, clustering, regression, categorical sequence labeling, real-valued sequence labeling, parsing, Bayesian networks, Markov random fields, ensemble learning, etc. Further methods for obtaining or analyzing brain activity: modified Beer-Lambert Law, event-related components, multi-voxel pattern analysis, spectral analysis, the use of MVPA on fNIRS, etc. Simple brain EEG signal characterization may be used for identification purposes. General purpose pattern recognition techniques and algorithms may be implemented.

0044The brain measurements are stored in a memory profile memory system <b>240</b>. A database <b>250</b> maintains a collection of a population's brain anatomy and activity signatures. During the authentication process <b>204</b>, stimuli <b>270</b> are provided to a second subject <b>272</b>. The second subject <b>272</b> may be the first subject <b>212</b> or another subject having data maintained in the database <b>250</b>. The response of the subject is measured by data collection and recording system <b>274</b>. The response may include data associated with brain anatomy and activity. Again, brain anatomy may be obtained using technology such as ultrasound, and/or EEG, fNIRS, MEG, MRI, etc. The data associated with brain anatomy and activity is provided to a pattern recognition device <b>276</b> that analyzes the associated with brain anatomy and activity to identify patterns. Again, anatomic characteristics of the brain, e.g., gyrification, cortical thickness, etc., are identified.

0045An analysis device <b>260</b> receives the results from the pattern recognition device, the previously processed brain measurements and prediction data associated with the subject from the database that maintains a collection of a population's brain anatomy and activity signatures. The analysis device <b>260</b> determines whether the subject being authenticated correlates to the subject's previously processed brain measurements and prediction data. The brain anatomy and activity patterns are thus compared to known or predicted signatures collected during calibration sessions, previous signatures collections, or predicted a priori from a library of ‘similar’ brain signatures. The analysis device <b>260</b> assigns a confidence of authenticity to authentication. The confidence of authenticity to the authentication may be based on statistical techniques to translate pattern recognition into probabilistic relationships given known conditions.

0046If the analysis device <b>260</b> determines that the response from the subject <b>272</b> is not acceptable, the subject may be rejected. However, if the brain measurements of the subject <b>272</b> being authenticated correlates with the subject's previously processed brain measurements and prediction data, the subject may be accepted. These brain signature identification techniques may be used in combination with other indeterminate authentication methods, e.g., handwriting recognition, to improve the sensitivity and specificity of the authentication method.

0047Accordingly, the system may be used to compare the brain anatomy and activity to known information and prerecorded signatures to identify and assign a probability of uniqueness. Alternatively, the system may use a potentially more discriminating and secure approach that may involve a series of memorized thoughts (e.g., child, car, waterfall), patterns of muscle activation (e.g., jump, serve a tennis ball, play a song on a piano), or imagined activities (e.g., pet a cat, solve the equation 13×14, eat banana) that a user would run through mentally to incite certain characteristic brain activities.

0048In another embodiment, a BCI system may be used to allow users to control a processing device, such as a computer, laptop, mobile phone, tablet computer, smart television, remote controls, microwaves, etc. A BCI system may be used to direct devices to carry out activities by associating mental conditions with media and search services. Current BCI systems rely on EEG, which is characterized by relatively fine temporal resolution but also relatively low spatial resolution. The low spatial resolution limits is not compatible with certain analysis techniques that have been shown to be useful for extracting high-level information. While a product created today based on existing technologies may not be good enough for precision applications, the technology is already available to provide entertainment-level implementations.

0049<figref idref="DRAWINGS">FIG. 3</figref> illustrates an example of a wearable system <b>300</b> for gathering BCI input according to an embodiment. A wearable brain system <b>300</b> may utilize an EEG sensor <b>310</b> that is held in place against the forehead with a headband <b>312</b>. Alternatively or in addition to EEG sensor <b>310</b>, the wearable system <b>300</b> may include several electrodes <b>316</b> that may be attached to the user's head to detect EEG signals. The wearable brain imaging device <b>300</b> may allow a user to grossly control one factor, e.g., the level of air pressure that a small fan outputs, by measuring electrical activity in the brain. The nature of EEG limits the spatial resolution to gross statistical estimates of scalp distributions that typically lead BCI-EEG devices to utilize spectral analysis to analyze and tease apart the unique frequency bands contained within the EEG signal.

0050<figref idref="DRAWINGS">FIGS. 4<i>a</i>-<i>c </i></figref>illustrate components of a neuroimaging device <b>400</b>, according to an embodiment. Far more sophisticated BCI input sensors move away from EEG to neuroimaging approaches that provide higher spatial resolution that is similar to MRI or PET scans. Optical brain imaging or even a combination of optical and EEG provide a system with the spatial and temporal resolution used for distinguishing hundreds or even thousands of unique activation patterns. In <figref idref="DRAWINGS">FIG. 4<i>a</i></figref>, a hat or helmet <b>410</b> is provided for the subject that includes a plurality of sensors and sources <b>412</b>. The sensors and sources <b>412</b> are provided to a processing device <b>420</b> that may be coupled to the subject. <figref idref="DRAWINGS">FIG. 4<i>b </i></figref>shows a detector <b>430</b> and a source <b>432</b> that is used in the neuroimaging device <b>400</b>. The sensors <b>432</b> may include EEG. A near-infrared spectroscopy (NIRS) detector <b>430</b> may also be used. <figref idref="DRAWINGS">FIG. 4<i>c </i></figref>illustrates the control module <b>440</b>.

0051One embodiment for using a BCI system to provide control computing experiences involves telepathic search. By monitoring BCI patterns while a user is exposed to various media, e.g., music or images, the BCI system may create a database of associations. Subsequently, when the user is in a search mode, mental imagery may recreate those brain activity patterns to help make the search more efficient. Another embodiment providing control computing experiences involves telepathic communication. By training the two or more users on the same set of media, while monitoring brain activity patterns, the system could create a common mental vocabulary that users could use to communicate with each other. Another embodiment providing control computing experiences involves telepathic augmented reality. Users may train mental imagery that is paired with 3D models and/or animation of those models to perform specific actions. Thus, by thinking about the model, the user may cause the 3D model or animation to appear while viewing through a device with AR capability.

0052<figref idref="DRAWINGS">FIG. 5</figref> is a flowchart of a method <b>500</b> for providing control of computing experiences according to an embodiment. Stimuli are presented to a user while measures of brain activity of that user are recorded <b>510</b>. Stimuli may be compound-stimuli, such as an image paired with a sound, to allow more reliable correlation. One or more brain activity measures that have reliable correlation with specific stimuli are identified using guided testing <b>520</b>. The brain activity measures may be one or more types, e.g., fNIRS and EEG. The candidate brain activity-stimuli pairings are stored <b>530</b>. Brain activity-stimuli pairings that have reliable correlations when the user is imagining the stimuli as compared to actually seeing, hearing, touching, etc. are identified through guided testing <b>540</b>. The list of brain activity-imagined-stimuli pairings is stored <b>550</b>. The stimuli are retrieved and displayed when correlated brain activity measures are detected to allow telepathic search, telepathic communication, and telepathic AR <b>560</b>. The strength of the correlations is refreshed by retesting brain activity-stimuli pairings using guided testing <b>570</b>.

0053With regard to a telepathic search, users typically know what content they are searching for and provide input search terms into a search tool. For example, to search for the song “Song 1” in a library, the user provides input search terms that overlap with song title, artist, album title, genre or others. However, users may have varying levels of search literacy for complex searches or may have fuzzy concepts of searches requests that too poorly define an effective search. This consequently produces poor search results. However, a telepathic search performed according to an embodiment allows users to perform a hands-free search against an image or music database by using a user's mental visualization. A telepathic search according to an embodiment allows for searches such as image searches, video searches, music searches, or web searches. A telepathic search may also allow a user to perform a search without knowing the actual word.

0054The BCI system builds on the concept of matching the unique patterns of thought to a database of content that is categorized to a user's brain patterns that emerge in response to basic elements of a thought, e.g., movement, light/dark patterns, attentional settings, etc. Once the user's brain patterns are recorded and correlated, the BCI system reconstructs thoughts from the brain patterns alone. When the users initiate a search, the new thought would be matched to known elements from previous thoughts and content stored in the database. Search results may be weighted based on the number of elements in the new thought that match with elements known to be associated with content in the database. The search results would be seemingly telepathic in the way that a user could think a thought and have the BCI system perform a telepathic search that return results matching the thought.

0055One example may include a user that is searching for an image. The memory is stored as a mental representation of the image, which may or may not be easily translated into words. Perhaps the image is a picture of a white dove followed by a black dove <b>610</b> as represented in <figref idref="DRAWINGS">FIG. 6</figref><i>a. </i>

0056Internet searches for the aforementioned picture may verbally be searched for with, “white dove followed black dove.” Translating a purely visual concept into a text search yields spurious results <b>620</b> as shown in <figref idref="DRAWINGS">FIG. 6<i>b</i></figref>. Translating a visualization into a visual search is not only intuitive, but holds the potential for more effective searches where verbal information does not easily translate from the output into a text-based search.

0057Another example where a telepathic search would yield superior results for a non-verbal search than a text-based search involves a search for music. For example, a user may want Animation's <b>1984</b> masterpiece “Obsession,” but cannot remember the artist, song title, album or lyrics. The user could think of the sounds of the song and a BCI system performing a telepathic search provides results of music that matches brain activations to the user's thoughts of the sound of “Obsession” without the user providing text inputs. A BCI system may perform such a search by matching those patterns of brain activity from the learning phase with brain activations produced by thinking of the song.

0058Cognitive psychology provide strong support for the neural network model, which proposes that representations in the brain are stored as patterns of distributed brain activity co-occurring in a particular temporal and spatial relationship. For example, a response to a particular input, such as a picture, results in a distribution of neuronal activity that is distributed across the brain in a specific pattern in time and cortical spatial location in your brain, which produces as an output the visual representation of the input.

0059Along these same lines, the psychophysical process of stimulus perception begins in the brain with the individual components signaled in the brain, then reassembled based on the elements that fall within attention. For example, when a viewer perceives an object, the color information, shape information, movement information, etc. initially enters the brain as individual components and attention or another mechanism binds the elements together to form a coherent percept. These concepts are important because a stimulus is not represented as a whole object or in a single, unified portion of the brain.

0060A telepathic search according to an embodiment may be implemented using techniques like multi-voxel pattern analysis (MVPA). MVPA builds on the knowledge that stimuli are represented in a distributed manner and perceived as a reconstruction of their individual elements. MVPA is a quantitative neuroimaging methodology that identifies patterns of distributed brain activity that are correlated with a particular thought such as perceiving a visual stimulus, perceiving an auditory stimulus, remembering three items simultaneously, attending to one dimension of an object while not focusing on another, etc. MVPA identifies the spatial and temporal patterns of activity distributed throughout the brain that identify complex mental representations or states. The mental representations may be cognitive activities such as memory activities, such as retrieving a long-term memory or representations of perceptual inputs including auditory stimulus. MVPA traditionally utilizes the temporal correlations between brain activity measured in volumetric pixels, i.e., voxels, that become active at a given moment in response to a stimulus or as part of a narrowly defined cognitive activity, e.g., long-term memory retrieval. Temporal and spatial patterns of biophysical signals may also be used to measure and identify psychological states or mental representations of a person to reveal information, such as cognitive workload, attention/distraction, mood, sociological dynamics, memories, and others.

0061MVPA may identify a person's unique patterns of activations in response to particular stimulus, then reconstruct that stimulus the patterns of brain activation alone. For example, video from a users' brain activations have may be reconstructed after the MVPA had been trained to learn the brain responses from the video. First, users may be shown video clips, and then each user's idiosyncratic pattern of activity in response to each video may be analyzed using MVPA to identify brain activity associated with elements of the video. Following the learning episode, the brain activity alone may identify enough elements from the video to reconstruct it by matching the brain activity to elements of videos stored in a database.

0062However, MVPA is mainly applied to MRI neuroimaging. MRI is a powerful neuroimaging technique, but it relies on large super-conducting magnets that make it an impractical brain imaging device in mobile settings. Optical imaging techniques such as fNIRS are relatively nascent but provide the potential for low-cost, wearable solutions that may be extensible to a wide variety of usages and applications.

0063According to an embodiment, MVPA and fNIRS may be combined to offer a viable analysis approach in MVPA with a viable wearable device to provide novel BCI-software interactions and functionality that is able to distinguish among dozens to potentially hundreds of brain activity patterns.

0064For a BCI system providing telepathic search, a learning phase is used to learn brain activation patterns in response to stimuli and a search phase is used to match mental representations to searchable content. In the learning phase, the system identifies stable patterns of brain activity in response to a given type of content, e.g., video, music, etc. The patterns are categorized in ways relevant for the type of content, e.g., image properties for pictures or video. Again, neuroimaging devices that may be used include INIRS, EEG, MEG, MRI, etc. Methodologies for obtaining or analyzing brain activity may again include modified Beer-Lambert Law, event-related components, multi-voxel pattern analysis, spectral analysis, use of MVPA on fNIRS.

0065<figref idref="DRAWINGS">FIG. 7</figref> illustrates a BCI system <b>700</b> for providing telepathic search according to an embodiment. In <figref idref="DRAWINGS">FIG. 7</figref>, a BCI Tracking module <b>710</b> monitors brain activity readings in relation to images or words on a display <b>720</b>. A BCI search coordination module <b>730</b> retrieves earlier BCI-word associations while the user is engaged with performing a search. Those associations are used to weight or order the search results. A training system <b>740</b> displays stimuli, e.g., sight, sound, smell, tactile, etc., or a combination, while measuring a user's brain activity, thus allowing BCI measures to be associated with particular images or words.

0066<figref idref="DRAWINGS">FIG. 8</figref> illustrates wireless telepathic communication using a BCI system <b>800</b> according to an embodiment. In <figref idref="DRAWINGS">FIG. 8</figref>, wireless communication <b>802</b> is provided between two subjects, first user, user₁<b>810</b> and second user, user₂<b>820</b>. The BCI system according to an embodiment enables users to communicate words and symbols using brainwaves and other brain activity measures. Users first have their BCI systems trained on a common set of brain activity measures in reaction to the same stimuli, e.g., as explained above with regard to telepathic searching. Then, when one user uses thought patters related to that stimuli, the BCI system alert the other user by displaying that same stimuli, thus allowing a sort of“mind reading.” The user interface may appear like a text chat UI with simple words and symbols. The user interface may be an audio-based or tactile-based system.

0067The wireless communication may include visual <b>830</b> and/or sound <b>850</b>. For visuals, users view the same images while brain activity measures of each are taken <b>832</b>. A first user, user₁<b>810</b>, thinks to elicit image X <b>834</b>. A second user, user₂<b>812</b>, sees the image X that user₁was thinking of displayed <b>836</b>.

0068For sound, users hear the same sounds while brain activity measures of each are taken <b>852</b>. A first user, user₁<b>810</b>, thinks to elicit sound X <b>854</b>. A second user, user₂<b>812</b>, hears through headphones the sound X that user1 <b>810</b> was thinking <b>856</b>. The sending user may be identified on a UI. The user could think to choose a recipient of the message.

0069Again, neuroimaging devices that may be used include fNRS, EEG, MEG, MRI, etc. Methodologies for obtaining or analyzing brain activity may again include modified Beer-Lambert Law, event-related components, multi-voxel pattern analysis, spectral analysis, etc.

0070<figref idref="DRAWINGS">FIG. 9</figref> is a networked system <b>900</b> for performing telepathic contextual search according to an embodiment. In <figref idref="DRAWINGS">FIG. 9</figref>, a networked BCI module <b>910</b> monitors wireless transmissions from other user's BCI systems <b>920</b>. A UI <b>930</b>, such as a chat interface, displays stimuli associated with brain activity measures from another person. The stimuli may also include other senses or combinations, such as sound, words, tactile, etc. A training system <b>940</b> displays stimuli, such as sight, sound, smell, tactile, etc., or a combination, to a user the user's brain activity is measured. This allows brain activity measures to be associated with particular images. A biometric and environmental sensor array <b>950</b> may be used to provide stimuli and obtain brain activity measurements. Contextual building blocks <b>960</b> may be developed that determine user activity, such as walking, running, and talking with someone. The sensor array <b>950</b> may be mounted on the user's head.

0071<figref idref="DRAWINGS">FIG. 10</figref> is a flowchart <b>1000</b> for providing telepathic augmented reality using a BCI system according to an embodiment. The BCI system enables users to see, hear and feel augmented reality (AR) objects by consciously directing thought. AR objects may be presented by monitoring BCI inputs and presenting corresponding AR experiences that the user may not purposefully invoke. As explained in previous related disclosures, a BCI system is trained to associate brain activity measures with certain images, thus allowing the image to later be displayed if the user creates a matching brain activity pattern.

0072In <figref idref="DRAWINGS">FIG. 10</figref>, a BCI system monitors BCI input <b>1010</b>. A determination is made whether a pattern match is detected <b>1020</b>. If not, then at <b>1022</b> the system continues to monitor BCI input and analyze the BCI input for matches. If a pattern match is detected, then at <b>1024</b> the BCI system creates rendering that reflects AR that correlates with the pattern match <b>1030</b>. The system then plays the AR experience <b>1040</b>. Thereafter, the BCI process may return <b>1042</b> to continue to monitor BCI input and analyze the BCI input for matches.

0073The AR experience is launched as a result of the monitoring by the BCI system. The AR experience may be visual, audio, tactile, or any other sense-based experience. Further, the users may direct the movement of AR characters through thought. This allows users to play a game in which they control AR characters that move or race. Moreover, the BCI system may monitor the environment for cues that could interact with current BCI input. For example, the system could perform object recognition, and if the user produces a brain activity measure that relates to cartoons, a cartoon version of the identified object may be presented. In another embodiment, the user may invoke the 3D orienting of an object by thinking about its position. Simple systems, e.g., MINDBENDER® allow a user to move objects through the use of concentration. However, these simple systems do not involve AR presentation or control.

0074<figref idref="DRAWINGS">FIGS. 11</figref> a-d show an example of AR presentation and control according to an embodiment. The AR <b>1110</b> in <figref idref="DRAWINGS">FIG. 11<i>a </i></figref>may be presented when the user produced a brain activity measure that corresponded to a lizard <b>1112</b>. In this example, the user lays out a trail of markers <b>1120</b>, produces the brain activity match with the lizard <b>1112</b>, and then watches as the AR experience is displayed. In <figref idref="DRAWINGS">FIG. 11<i>b</i></figref>, the AR <b>1110</b> has moved from the tablet <b>1114</b> onto the trail of markers <b>1120</b>. In <figref idref="DRAWINGS">FIG. 11<i>c</i></figref>, the AR <b>1110</b> moves even farther along the trail of markers <b>1120</b> towards the laptop <b>1140</b>. In <figref idref="DRAWINGS">FIG. 11<i>d</i></figref>, the AR <b>1110</b> has reached the laptop <b>1140</b>. The display in <figref idref="DRAWINGS">FIGS. 11</figref> a-d may be through a phone, a head-mounted display or another sensory modality.

0075<figref idref="DRAWINGS">FIG. 12</figref> illustrates a system <b>1200</b> that may provide telepathic augmented reality according to an embodiment. Yet again sensors and detectors <b>1210</b>, such as fNIRS, EEG, MEG, modified Beer-Lambert Law, event-related components, etc. may be utilized along with a biometric and environmental sensor array <b>1212</b>. The brain activity sensor array may be mounted on the user's head. In an embodiment, an AR capable computing device <b>1220</b> may be used with media systems <b>1230</b> that may include dual facing cameras <b>1232</b>, of which one may be a top facing camera.

0076An AR rendering module <b>1240</b> may automatically make AR characters blend with the environment in convincing ways. A database <b>1250</b> of recognized sensor inputs is used and AR character and AR environmental content are implemented. A face detection subsystem <b>1260</b> may be provided to identify the face of a subject. Further, video analytics <b>1270</b> may include object recognition <b>1272</b>, projected beacon tracking <b>1274</b> and environmental characteristics recognition <b>1276</b>, e.g., recognizing horizontal surfaces to bound AR character actions so it does not goes through the floor. An RFID scanner system <b>1280</b> may be used for scanning objects with embedded tags. An integrated projector <b>1234</b> may also be utilized.

0077The BCI system <b>1200</b> may further include a BCI to AR mapping module <b>1282</b> that receives input from BCI middleware <b>1284</b> and maps it to an AR experience. A database <b>1286</b> of brain activity patterns provide matches to AR experiences. These matches may be general for users “out of the box,” or they may be created through a matching process. An AR presentations system <b>1288</b> may be connected to BCI sensors <b>1210</b>, and may be wireless or wired. Also, a game module <b>1290</b> that allows user to compete in “mind control” of AR characters may be implemented.

0078<figref idref="DRAWINGS">FIG. 13</figref> illustrates a cortical representation of visual space used to represent mental desktop space provided by a BCI system <b>1300</b> according to an embodiment. The BCI system <b>1300</b> enables use of the mental desktop for access, navigation, command, and control of digital devices and content functions. Human-computer interfaces now available rely on the physical senses, e.g., vision, and physical movements, e.g., hands controlling a mouse or keyboard, to provide an interactive platform for accessing and controlling digital devices and content. These physical and perceptual inputs and controls are limited and restrict the potential expression of more novel and efficient human-computer interfaces. Existing systems that provide brain-computer interfaces are based on electroencephalography (EEG) technology and control functions from a predetermined library that define the outputs and leave the inputs open-ended. Moreover, the inputs are provide using spectral analysis of electrical activity, which is fundamentally different than the use of spatial and hemispherical information that mental gestures used in a visual BCI workspace according to an embodiment.

0079According to the embodiment described with respect to <figref idref="DRAWINGS">FIG. 13</figref>, a BCI system <b>1300</b> enables users to operate computing devices by focusing mental attention on different sections of the visual field. This visual field is referred to as the “mental desktop.” When users mentally visualize or attend to the upper-left quadrant of their visual field, an associated command may be executed. However, this does not mean the user changes eye focus to that area, but may simply think about, i.e., visualize, that area. The areas of the visual field have particularly strong mapping to brain activity measures.

0080<figref idref="DRAWINGS">FIG. 13</figref> shows the mental desktop includes a left field <b>1310</b> and right field <b>1320</b>. The left field <b>1310</b> is arranged to include a left-left upper quadrant <b>1312</b>, a left-right upper quadrant <b>1314</b>, a left-left lower quadrant <b>1316</b> and a left-right lower quadrant <b>1318</b>. The right field <b>1320</b> is arranged to include a right-left upper quadrant <b>1322</b>, a right-right upper quadrant <b>1324</b>, a right-left lower quadrant <b>1326</b> and a right-right lower quadrant <b>1328</b>.

0081Visual signals from the retina reach the optic chiasma <b>1330</b> where the optic nerves partially cross <b>1332</b>. Images on the sides of each retina cross over to the opposite side of the brain via the optic nerve at the optic chiasma <b>1330</b>. The temporal images, on the other hand, stay on the same side. This allows the images from either side of the field from both eyes to be transmitted to the appropriate side of the brain, combining the sides together. This allows for parts of both eyes that attend to the right visual field to be processed in the left visual system in the brain, and vice versa. The optic tract terminates in the left geniculate nucleus <b>1340</b> and right geniculate nucleus <b>1360</b>. The left geniculate nucleus <b>1340</b> and right geniculate nucleus <b>1360</b> are the primary relay center for visual information received from the retina of the eye. The left <b>1340</b> and right <b>1360</b> geniculate nuclei receive information from the optic chiasma <b>1330</b> via the optic tract and from the reticular activating system. Signals from the left and right geniculate nucleus are sent through the optic radiations <b>1370</b>, <b>1372</b>, which act as a direct pathway to the primary visual cortex <b>1390</b>. In addition, the left <b>1340</b> and right <b>1360</b> geniculate nuclei receive many strong feedback connections from the primary visual cortex. Meyer's loops <b>1380</b>, <b>1382</b> are part of the optic radiation that exit the left lateral geniculate nucleus <b>1340</b> and right lateral geniculate nucleus <b>1360</b>, respectively, and project to the primary visual cortex <b>1390</b>. The visual cortex <b>1390</b> is responsible for processing the visual information.

0082For a “mental desktop,” eye tracking technology may be used to effect navigation and command and control of computer interfaces. However, eye tracking technology is constrained to the physical space and suffers from the same limitations as archetypal operating system models. For example, looking up and to the left has been used to translate a mouse pointer to that region.

0083Computer interfaces utilize a digital desktop space represented by physical space on a computer display. In contrast, a mental desktop according to an embodiment detaches the physical desktop into mental workspace, i.e., mental desktop that divides workspace into visuospatial regions based on regions of an individual's field of view referred to as visual hemifields.

0084Visual information is naturally segregated by the left and right eye as well as upper and lower, and left and right divisions, within the left and right eyes. These divisions create hemifields that are each represented in corresponding brain regions. The organization of the brain around the hemifields is referred to as retinotopic organization because regions of the retina are represented in corresponding brain regions.

0085The mental desktop's workspace facilitates access of assigned information, e.g., application shortcut, file, or menu, by looking or mentally visualizing a region in visual space. In summary, the mental desktop creates an imaginary desktop space for a user to use in a similar manner as a digital desktop in current operating systems.

0086By extracting the eight visual hemifields through the retinotopic organization in the human primary visual cortex, a mental desktop may be implemented by the BCI system. <figref idref="DRAWINGS">FIG. 14</figref> is a diagram of the optic system <b>1400</b> according to an embodiment. In <figref idref="DRAWINGS">FIG. 14</figref>, the optical nerve <b>1410</b> tracks from eyes <b>1412</b> to the human primary visual cortex <b>1420</b> in the occipital lobe of the brain.

0087The visual signals pass from the eyes <b>1412</b> and through the optic nerve <b>1410</b> to the optic chiasm. For example, looking at or visualizing the upper right visual field to the right of the midline produces concomitant brain activity in visual cortex corresponding to the same hemifield in the upper right of the right eye. The retinotopic organization of the visual cortex allows for the use of visuospatial information decoded from brain activity into usable information for mental desktop to identify the region a user wishes to access. As described above, images on the sides of each retina cross over to the opposite side of the brain via the optic nerve <b>1410</b> at the optic chiasma <b>1430</b>. The temporal images, on the other hand, stay on the same side. The lateral geniculate nuclei <b>1440</b> (left and right) are the primary relay center for visual information received from the eye <b>1412</b>.

0088The signals are sent from the geniculate nuclei <b>1440</b> through the optic radiations <b>1450</b>. The optic radiations <b>1450</b> are the direct pathway to the primary visual cortex <b>1420</b>. Unconscious visual input goes directly from the retina to the superior colliculus <b>1460</b>.

0089Table 1 illustrates the mapping of the left and right field quadrants to the visual cortex through the optic chiasma, left and right geniculate nucleus, and the optic radiations including the Meyer's loop.

0000<tables id="TABLE-US-00001" num="00001"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="4"><colspec colname="1" colwidth="49pt" align="left" /><colspec colname="2" colwidth="63pt" align="center" /><colspec colname="3" colwidth="42pt" align="center" /><colspec colname="4" colwidth="63pt" align="center" /><thead><row><entry namest="1" nameend="4" rowsep="1">TABLE 1</entry></row><row><entry namest="1" nameend="4" align="center" rowsep="1" /></row><row><entry /><entry>Directly Through</entry><entry /><entry>Optic Radiation to</entry></row><row><entry /><entry>Optic Chiasma or</entry><entry>Left or Right</entry><entry>Visual Cortex or</entry></row><row><entry>Field</entry><entry>Cross-Over in Optic</entry><entry>Geniculate</entry><entry>Meyer's Loop to</entry></row><row><entry>Quadrant</entry><entry>Chiasma</entry><entry>Nucleus</entry><entry>Visual Cortex</entry></row><row><entry namest="1" nameend="4" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry>Left Field</entry><entry /><entry /><entry /></row><row><entry>L-LUQ</entry><entry>OC</entry><entry>LGN</entry><entry>OR2VC</entry></row><row><entry>L-RUQ</entry><entry>OCX</entry><entry>RGN</entry><entry>OR2VC</entry></row><row><entry>L-LRLQ</entry><entry>OC</entry><entry>LGN</entry><entry>ML2VC</entry></row><row><entry>L-RLQ</entry><entry>OCX</entry><entry>RGN</entry><entry>ML2VC</entry></row><row><entry>Right Field</entry></row><row><entry>R-LUQ</entry><entry>OCX</entry><entry>LGN</entry><entry>OR2VC</entry></row><row><entry>R-RUQ</entry><entry>OC</entry><entry>RGN</entry><entry>OR2VC</entry></row><row><entry>R-LLQ</entry><entry>OCX</entry><entry>LGN</entry><entry>ML2VC</entry></row><row><entry>R-RLQ</entry><entry>OC</entry><entry>RGN</entry><entry>ML2VC</entry></row><row><entry namest="1" nameend="4" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>

0090<figref idref="DRAWINGS">FIG. 15</figref> shows the physiologically segregated sections of visual space <b>1500</b> according to an embodiment. The BCI system assigns content to physiologically segregated sections of visual space. In <figref idref="DRAWINGS">FIG. 15</figref>, the visual space <b>1500</b> is organized into a left hemifield <b>1510</b> and right hemifield <b>1520</b>. The visual space <b>1500</b> is also arranged into an upper hemifield <b>1530</b> and a lower hemifield <b>1540</b>. Thus, according to an embodiment, the visual space <b>1500</b> may be assigned content, which the user may implement based on visualization of the appropriate area in the visual space <b>1500</b>. Content may be anything, such as application shortcuts, file pointers, files, etc. The user would simply look at or visualize a particular hemifields to access whatever content is assigned to the location that is seen or visualized according to that physiologically segregated section of visual space. Each of the eight hemifields of the visual space <b>1500</b> may have content assigned to the region in space accessed through visualization or any activity in the hemifield space. For example, a first application shortcut <b>1550</b> may be assigned to the left-left upper hemifield. A file pointer <b>1552</b> may be assigned to the left-left lower hemifield.

0091A training system may be utilized to map each individual's visual field to define the regions of visual space <b>1500</b> in <figref idref="DRAWINGS">FIG. 15</figref> to correspond to regions of the visual field. The acuity of the mental desktop may be refined with spatial boundaries in the visual field. Further, users may use a head mounted display to facilitate the visualization of the division of visual space <b>1500</b> available on the mental desktop. An individual's mental desktop may be remotely stored, e.g., cloud storage, to enable the mental desktop workspace on any device.

0092<figref idref="DRAWINGS">FIG. 16</figref> is a model of a human cortex <b>1600</b> showing the primary motor cortex <b>1610</b> and the primary sensory cortex <b>1612</b> as utilized by a BCI system according to an embodiment. The primary motor cortex <b>1610</b> and the primary sensory cortex <b>1612</b> are anterior and posterior to the central sulcus <b>1630</b>, respectively, indicated by the vertical slice. The central sulcus <b>1630</b> is a fold in the cerebral cortex, which represents a prominent landmark of the brain that separates the parietal lobe <b>1640</b> from the frontal lobe <b>1650</b> and the primary motor cortex <b>1610</b> from the primary somatosensory cortex <b>1660</b>.

0093According to an embodiment, mental gestures are mapped to brain activity emerging from topographically organized brain regions, such as the primary motor and/or somatosensory cortices found in the human brain. These two brain areas each have regions that are divided into discrete areas that are dedicated to controlling corresponding body locations. In <figref idref="DRAWINGS">FIG. 16</figref>, the supplementary motor cortex <b>1660</b> is shown generally midline of the frontal lobe <b>1650</b>. The supplementary motor cortex <b>1660</b> is the part of the cerebral cortex <b>1600</b> that contributes to the control of movement. The primary motor cortex <b>1610</b> is located in the posterior portion of the frontal lobe <b>1650</b>. The primary motor cortex <b>1610</b> is involved with the planning and execution of movements. The posterior parietal cortex <b>1640</b> receives input from the three sensory systems that play roles in the localization of the body and external objects in space: the visual system, the auditory system, and the somatosensory system. The somatosensory system provides information about objects in our external environment through touch, i.e., physical contact with skin, and about the position and movement of our body parts through the stimulation of muscle and joints. In turn, much of the output of the posterior parietal cortex <b>1640</b> goes to areas of frontal motor cortex <b>1670</b>. The premotor cortex <b>1670</b> lies within the frontal lobe <b>1650</b> just anterior to the primary motor cortex <b>1610</b>. The premotor cortex <b>1670</b> is involved in preparing and executing limb movements and coordinates with other regions to select appropriate movements.

0094User interfaces that utilize physical or perceptual inputs use a set of protocol to perform pre-determined actions. For example, a keyboard uses keys that have characters assigned to each or a mouse uses X-Y locations and clicks to indicate a response. Similarly, a BCI system needs a foundation to establish a widespread, practical use of BCI inputs. According to an embodiment, a BCI system implements and processes mental gestures to perform functions or provide other types of input. Mental gestures are a library of thought gestures interpreted from brain activity to be used as a computer input in the same way a keys on a keyboard provide pre-determined input for flexible control over their output.

0095For example, touch-enabled surfaces have pre-set gestures such as pinching, squeezing, and swiping. These touch gestures serve as a foundation to build touch interfaces and usages across tasks. Similarly, mental gestures follow the same principle of establishing a foundation for BCI input through a library of BCI gestures, i.e., mental gestures, to enable usages across tasks and even platforms.

0096Mental gestures are executable through thought and recorded directly from brain activity. In contrast to touch gestures that are based on actual movement, mental gestures are imagined motor movements. The combination of a library of mental gestures and the flexibility of using a wide number of imagined movements rather than a single modality such as touch present the potential for an extremely powerful interface to a BCI system. Benefit to mental gestures over traditional inputs include (1) the user doesn't need to physically input any information, which would allow people without limbs or control of those limbs to perform the actions, (2) the mental gestures may emerge from any imagined motor movements that would not be practical as physical inputs, e.g., kicking, (3) the range of possible mental gestures expands the flexibility and utility over traditional inputs such as mice, keyboards, and trackpads that rely on manual inputs, and (4) mental gestures may be hemisphere specific because many brains have a left and right lateralized brain hemispheres that may create independent motor signals.

0097Examples of mental gestures include but are not limited to single digit movement, digit movement of different numbers, e.g., 1, 2 or 3-finger movement), hand waving, kicking, toe movement, blinks, head turning, hand nodding, bending at the waist, etc. The movements represented by mental gestures are purely imagined movements that may be associated with a variety of computer inputs. For example, an operating system may assign functionality to single-digit movement and a different functionality to two-digit movement. Alternatively, a media player could assign each of its functions, e.g., play/pause, reverse, shuffle, etc., to different mental gestures.

0098One possible implementation of mental gestures would be a software development kit (SDK) with a library of mental gestures for developers to assign to proprietary functions within their software. A SDK is a set of software development tools that allows for the creation of applications for a system. The SDK would enable developers to access mental gestures that may be used in a flexible, open-ended way. For example, a videogame developer could use the mental gestures SDK to develop BCI control over aspects of a videogame or a mobile original equipment manufacturer (OEM) could use the mental gestures SDK to develop mental gesture control over proprietary functions on their mobile device.

0099Mental gestures could also be used with another system that could combine multiple sources of inputs. If a cross-modal perceptual computing solution existed, mental gestures may be an additional source of input to be combined with other perceptual inputs. For example, air gestures could combine with mental gestures to code for left or right-handed air gestures based on left-lateral or right-lateral mental gesture input.

0100<figref idref="DRAWINGS">FIG. 17</figref> is a model <b>1700</b> demonstrating the topographical organization of the primary motor cortex <b>1710</b> on the pre-central gyrus of the human cortex. Each part of the body is represented by distinct areas of the cortex with the amount of cortex associated with the control of its respective body part. <figref idref="DRAWINGS">FIG. 17</figref> shows the areas responsible for preparation and execution of movement of the foot <b>1720</b>, the hip <b>1722</b>, the trunk <b>1724</b>, the arm <b>1726</b>, the hand <b>1728</b>, the face <b>1730</b>, the tongue <b>1732</b> and the larynx <b>1734</b>.

0101Any brain imaging device with spatial resolution high enough to extract the signals from a segment of cortex narrow enough to distinguish between neighboring areas may be used to implement the mental desktop. Some examples of currently available devices include dense electrode EEG, fNIRS, MRI, or MEG.

0102For each signal from the brain, the hemisphere (left or right), spatial location, and area are responsible for codes for the source of the motor signal. For example, activity or imagined activity of the left index finger would produce activity in the finger area in the right hemisphere. Mental gestures would code for left, single digit movement and the location and amount of area active would code for the precise finger and the number of digits, i.e., 1, 2, 3, or 4-digit gestures.

0103Thus, a system for implement a mental desktop that uses mental gestures for input according to an embodiment may include neuroimaging devices, such as fNIRS, EEG. MEG. MRI, ultrasound, etc. Methodologies for obtaining or analyzing brain activity may also include modified Beer-Lambert Law, event related components, multi-voxel pattern analysis, spectral analysis, use of MVPA on fNIRS.

0104According to an embodiment, a BCI system provides a mental desktop that maps computer content and functions to different sections of the visual field. The BCI system allows users to be trained in the application of the above-referenced system. A library of thought gestures that are interpreted from brain activity may be used to affect computer navigation, command, and control. Further, development systems may be provided to allow software developers to utilize mental gestures.

0105<figref idref="DRAWINGS">FIG. 18</figref> illustrates a user interface <b>1800</b> for assigning BCI measures and other modalities to applications according to an embodiment. In <figref idref="DRAWINGS">FIG. 18</figref>, a column of applications <b>1810</b> are shown on the left. BCI measures and other modalities <b>1820</b> are shown on the right. Other applications <b>1810</b> as well as other BCI measures and other modalities <b>1820</b> may be implemented. The BCI measures and other modalities <b>1820</b> may be selected for assignment to at least one of the applications <b>1810</b> on the left. Thus, a BCI system according to an embodiment may be used in multimodal systems.

0106By combining BCI with other modalities, e.g., gesture, voice, eye tracking, and face/facial expression tracking, new user experiences and ways for users to control electronic devices may be provided. Thus, the BCI system according to an embodiment recognizes both BCI types of input as well as other modalities. In addition, some approaches to feedback loops with brain activity elicitation may be implemented, and contextual sensing may alter the use of BCI input.

0107<figref idref="DRAWINGS">FIG. 19</figref> shows a BCI system <b>1900</b> that accepts BCI inputs <b>1910</b> and other modality inputs <b>1912</b> according to an embodiment. In <figref idref="DRAWINGS">FIG. 19</figref>, BCI inputs <b>1910</b> are provided to the BCI system <b>1900</b> for implementation with applications <b>1920</b>. Additional modalities <b>1912</b> are also provided to the BCI system <b>1900</b> for implementation with applications <b>1920</b>. However, some of the BCI inputs <b>1910</b> and the additional modalities <b>1912</b> may be mutually exclusive while some may be used together. A perceptual computing to BCI database <b>1930</b> may be used to hold heuristics on how natural UI inputs and BCI inputs work together. A coordination module <b>1940</b> receives input from BCI inputs <b>1912</b>, the additional input modalities <b>1912</b> and perceptual computing inputs from the database <b>1930</b> and then makes determinations on the determined user intent. The coordination module <b>1940</b> makes a final determination of user intent based on results from the BCI application coordination module <b>1970</b> and initiates a final command. A UI <b>1960</b>, as shown in <figref idref="DRAWINGS">FIG. 19</figref>, may be used for assigning the BCI <b>1910</b> and additional modality <b>1912</b> inputs to applications <b>1920</b>. An application associations database <b>1932</b> may be used to store BCI/application associations. A BCI application coordination module <b>1970</b> monitors whether assigned applications are running and initiates BCI control for relevant applications.

0108A BCI input quality module <b>1972</b> monitors environmental signals that degrade sensor input. The BCI system further includes a factor database of factor conditions <b>1934</b>, which includes the variables described above and their levels that inhibit particular forms of input <b>1910</b>, <b>1912</b>. A director module <b>1980</b> receives the inputs <b>1910</b>, <b>1912</b>, weighs them against the factor database <b>1934</b>, and sends commands to the applications <b>1920</b> to control how the inputs <b>1910</b>, <b>1912</b> are used, e.g., turned off, turned on, some measures weighed more than others, etc. A contextual building block subsystem <b>1982</b> measures environmental and user factors. A determination is made by the director module <b>1980</b> whether possible interference is occurring. If interference is detected, the director module <b>1980</b> adjusts the BCI input <b>1910</b>.

0109With input modalities such as hand gestures, voice, and eye-tracking, one challenge is alerting the system to an imminent command through one of those modalities, e.g., a voice command. The system may interpret inadvertent noise or movements, before or after the actual command, as a command. A BCI pattern from the user immediately before the command could signal that the next major sensor-detected event may be interpreted as the command.

0110When a system is subject to input from one or more modalities at the same time, BCI input could indicate which modality is to have precedence. One example of the use of cross-modal BCI input would be the use of BCI input to determine whether it is a gesture is a right or left handed gesture base. Alternatively, BCI input may be used simultaneously with another modality to reinforce the input command. For example, a brain activity pattern may be measured at the same time as a voice command. The brain activity pattern may be used to help the system differentiate between 2 similar sounding commands.

0111BCI systems according to an embodiment that include life blogging and “total recall” systems that records audio and video from the wearer's point of view may be used to aid people with cognitive deficits. Software algorithms may be used determine aspects of the sensor input. For example, an elderly person with memory loss could wear such a device, and when the BCI detects a confused state, through electrical patterns and/or blood flow, the system could give audio information in the earpiece that reminds the user of the names of people and objects in view.

0112See and think commands and tracking may be provided. A user could use eye tracking input to select a target, and then use the BCI system to provide input to act on the target that is being focused upon. e.g., the object the user is looking at changes color based on the brain activity pattern. This example could also be applied to visual media, e.g., the user could focus on a character, and the user's brain activity pattern could mark that character as being more interesting. Further, as a user reads, confusion may be detects to indicate the text was not understood, which may be helpful in teaching.

0113BCI input may be used to address cross modality interruption. The BCI input may be used to interrupt a system that is responding to another modality. For example, in a game, a user may use an air gesture to move a character in a direction, then use a change in BCI input to stop the character. UI feedback may also be used with BCI input. For example, a BCI system may provide various feedback to users when the system identifies BCI input, allowing the user to know the input has been received and confirmed. The BCI feedback could occur with other modality feedback, such as gesture. Further, a UI may be used for user mapping of BCI input. A user interface allows a user to map brain activity patterns to a given modality so that the system activates a command window-of-opportunity for that modality when the corresponding BCI pattern occurs. A user may map brain activity patterns to a given modality, so that the system has higher reliability in recognizing a command because the sensed inputs correlate to a brain activity pattern plus another modality. A user may also map different modalities to different brain activity patterns so that one pattern will mean that a correlating modality may be active, while another pattern activates a different modality.

0114BCI input may also be used to activate system resources. For example, a system may be alerted to come out of power state when the user becomes more alert. This may be used when a user is doing visual design. The BCI system could allow a processor to go into a sleep state as the user is in browsing mode. When brain activity patterns indicate that the user is about to take action, such as making an edit, the system could power up the processor so the processor is more responsive when the user starts the action.

0115According to an embodiment, the BCI system <b>1900</b> enables users to assign BCI input to one application that may not have focus, wherein focus refers to the application that currently has the attention of the OS. An application would then respond to BCI input even though the user is doing something else. A UI enables the user to assign the application.

0116Other examples of embodiments may include music and audio implementations where BCI input is accepted for control while the user is editing a document. Communication channels may show the user's status, e.g., busy thinking, through an instant messaging (IM) client while the user is being productive. Particular brain regions facilitate switching between tasks and BCI input to change the music may facilitate switching between tasks. The BCI system may be mapped to a music player so that whenever the task-switching portion of the brain becomes active, the music player skips to the next track to facilitate switching to a new task. In addition, autonomous vehicles will allow drivers to escape the demands of driving to enjoy non-driving activities in a vehicle. However, when the duties of driving return to the driver, the non-driving activities withdraw. The BCI system may map entertainment features of an in-vehicle infotainment system to cognitive workload to switch off entertainment features when a certain workload level is reached.

0117The BCI system could also make determinations about user context in order to allow various BCI inputs to be used at a given time. A status indicator could show the user when BCI input is available as an input. Other contextual determinations may be provided by the BCI system according to an embodiment. For example, the activity of the user may be determined by biometric sensors measuring as heart rate, respiration, movement, by accelerometers and gyroscopes, and user position, e.g., standing up versus lying down. At certain activity levels, unreliable BCI input may be prevented from being used by the system, or the system could adjust to the varying circumstances. The BCI system may determine whether the user is engaged in conversation and that information may be used as BCI input. BCI input for making contextual determinations may also include environmental conditions inhibiting reliable BCI input that causes user distraction, including sounds, visual stimuli, unpredictable noise, odor, media being played, and other factors and environmental conditions that could inhibit accurate measures due to electrical interference, such as magnetic fields, ambient temperature, and other environmental factors.

0118Different types of brain activity sensors have different strengths and benefits for a given task that the user is doing. For example, in instances where higher spatial resolution desired, the system may select fNIRS input rather than EEG which has lower spatial resolution. In other instances, rapid feedback may be desired so the system may select EEG or another technology that also has higher temporal resolution. Environmental sensors could determine user activities to influence which BCI input is best. Environmental factors such as electromagnetic energy are known to be detectable by EEG. In instances where electromagnetic (EM) energy would interfere with EEG recording, the BCI system may switch to a superior input source.

0119<figref idref="DRAWINGS">FIG. 20</figref> illustrates a flowchart of a method <b>2000</b> for determining user intent according to an embodiment. A BCI system determines user intent <b>2010</b>. A perceptual computing system interprets the user input <b>2020</b>. A coordination module then makes a final determination of user intent and initiates final command <b>2030</b>.

0120<figref idref="DRAWINGS">FIG. 21</figref> is a flowchart of a method <b>2100</b> for assigning BCI input for controlling an application according to an embodiment. A user matches a BCI input with an application <b>2110</b>. The BCI application coordination module monitors for application use <b>2120</b>. A determination is made whether the application is in use <b>2130</b>. If the application is not in use <b>2132</b>, the process returns to match BCI input with an application. If the application is in use <b>2134</b>, the assigned BCI input is used to control the application <b>2140</b>.

0121<figref idref="DRAWINGS">FIG. 22</figref> is a flowchart of a method <b>2200</b> for adjusting contextual factors by the BCI system according to an embodiment. A BCI input subsystem is running <b>2210</b>. The contextual building block subsystem measures environmental and user factors <b>2220</b>. A determination is made by the director module whether possible interference is occurring <b>2230</b>. If no, then at <b>2232</b> the process returns to the beginning of the process. If possible interference is detected <b>2234</b>, the director module adjusts the BCI input <b>2240</b>. The process may return to the beginning <b>2242</b>.

0122Thus, according to embodiments described herein, brain/skull anatomical characteristics, such as gyrification, cortical thickness, scalp thickness, etc., may be used for identification/authentication purposes. Measured stimuli/response brain characteristics, e.g., anatomic and physiologic, may be translated into specific patterns to categorize a brain for identify/authentication purposes. The anatomical and physiologic brain data may be coupled to determine identity and authenticity of a user. Information on other brain signatures, e.g., anatomic and physiologic, and comparisons to similar brains may be used to predict a brain response to a new stimulus and for identification and/or authentication purposes. Brain identification and/or authentication techniques in combination with other identification and/or authentication techniques, e.g., password, other biometric parameters, may be used to increase the identity/authentication sensitivity and specificity.

0123<figref idref="DRAWINGS">FIG. 23</figref> illustrates a block diagram of an example machine <b>2300</b> for providing a brain computer interface (BCI) system based on gathered temporal and spatial patterns of biophysical signals according to an embodiment upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform. In alternative embodiments, the machine <b>2300</b> may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine <b>2300</b> may operate in the capacity of a server machine and/or a client machine in server-client network environments. In an example, the machine <b>2300</b> may act as a peer machine in peer-to-peer (P2P) (or other distributed) network environment. The machine <b>2300</b> may be a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a mobile telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations.

0124Examples, as described herein, may include, or may operate on, logic or a number of components, modules, or mechanisms. Modules are tangible entities (e.g., hardware) capable of performing specified operations and may be configured or arranged in a certain manner. In an example, circuits may be arranged (e.g., internally or with respect to external entities such as other circuits) in a specified manner as a module. In an example, at least a part of one or more computer systems (e.g., a standalone, client or server computer system) or one or more hardware processors <b>2302</b> may be configured by firmware or software (e.g., instructions, an application portion, or an application) as a module that operates to perform specified operations. In an example, the software may reside on at least one machine readable medium. In an example, the software, when executed by the underlying hardware of the module, causes the hardware to perform the specified operations.

0125Accordingly, the term “module” is understood to encompass a tangible entity, be that an entity that is physically constructed, specifically configured (e.g., hardwired), or temporarily (e.g., transitorily) configured (e.g., programmed) to operate in a specified manner or to perform at least part of any operation described herein. Considering examples in which modules are temporarily configured, a module need not be instantiated at any one moment in time. For example, where the modules comprise a general-purpose hardware processor <b>2302</b> configured using software; the general-purpose hardware processor may be configured as respective different modules at different times. Software may accordingly configure a hardware processor, for example, to constitute a particular module at one instance of time and to constitute a different module at a different instance of time. The term “application,” or variants thereof, is used expansively herein to include routines, program modules, programs, components, and the like, and may be implemented on various system configurations, including single-processor or multiprocessor systems, microprocessor-based electronics, single-core or multi-core systems, combinations thereof, and the like. Thus, the term application may be used to refer to an embodiment of software or to hardware arranged to perform at least part of any operation described herein.

0126Machine (e.g., computer system) <b>2300</b> may include a hardware processor <b>2302</b> (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory <b>2304</b> and a static memory <b>2306</b>, at least some of which may communicate with others via an interlink (e.g., bus) <b>2308</b>. The machine <b>2300</b> may further include a display unit <b>2310</b>, an alphanumeric input device <b>2312</b> (e.g., a keyboard), and a user interface (UI) navigation device <b>2314</b> (e.g., a mouse). In an example, the display unit <b>2310</b>, input device <b>2312</b> and UI navigation device <b>2314</b> may be a touch screen display. The machine <b>2300</b> may additionally include a storage device (e.g., drive unit) <b>2316</b>, a signal generation device <b>2318</b> (e.g., a speaker), a network interface device <b>2320</b>, and one or more sensors <b>2321</b>, such as a global positioning system (GPS) sensor, compass, accelerometer, or other sensor. The machine <b>2300</b> may include an output controller <b>2328</b>, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR)) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

0127The storage device <b>2316</b> may include at least one machine readable medium <b>2322</b> on which is stored one or more sets of data structures or instructions <b>2324</b> (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. The instructions <b>2324</b> may also reside, at least partially, additional machine readable memories such as main memory <b>2304</b>, static memory <b>2306</b>, or within the hardware processor <b>2302</b> during execution thereof by the machine <b>2300</b>. In an example, one or any combination of the hardware processor <b>2302</b>, the main memory <b>2304</b>, the static memory <b>2306</b>, or the storage device <b>2316</b> may constitute machine readable media.

0128While the machine readable medium <b>2322</b> is illustrated as a single medium, the term “machine readable medium” may include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that configured to store the one or more instructions <b>2324</b>.

0129The term “machine readable medium” may include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine <b>2300</b> and that cause the machine <b>2300</b> to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Non-limiting machine readable medium examples may include solid-state memories, and optical and magnetic media. In an example, a massed machine readable medium comprises a machine readable medium with a plurality of particles having resting mass. Specific examples of massed machine readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

0130The instructions <b>2324</b> may further be transmitted or received over a communications network <b>2326</b> using a transmission medium via the network interface device <b>2320</b> utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile telephone networks ((e.g., channel access methods including Code Division Multiple Access (CDMA), Time-division multiple access (TDMA), Frequency-division multiple access (FDMA), and Orthogonal Frequency Division Multiple Access (OFDMA) and cellular networks such as Global System for Mobile Communications (GSM), Universal Mobile Telecommunications System (UMTS), CDMA 2000 1×*standards and Long Term Evolution (LTE)), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) 802 family of standards including IEEE 802.11 standards (WiFi), IEEE 802.16 standards (WiMax®) and others), peer-to-peer (P2P) networks, or other protocols now known or later developed.

0131For example, the network interface device <b>2320</b> may include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas to connect to the communications network <b>2326</b>. In an example, the network interface device <b>2320</b> may include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by the machine <b>2300</b>, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.

Additional Notes & Examples

0132Example 1 may include subject matter (such as a device, apparatus, client or system) including a library of stimuli for provisioning to a user, a data collection device for gathering temporal and spatial patterns of biophysical signals associated with brain activity in response to provisioning stimuli from the library of stimuli to the user and a processing device for correlating the gathered temporal and spatial patterns of biophysical signals associated with brain activity to identify a brain signature of the user and performing a processor controlled function based on the brain signature of the user identified through correlating the gathered temporal and spatial patterns of biophysical signals associated with brain activity.

0133Example 2 may optionally include the subject matter of Example 1, wherein the processing device compares mental profile of the user derived from the brain signature of the user with mental profiles from a database of mental profiles of a predetermined population.

0134Example 3 may optionally include the subject matter of any one or more of Examples 1-2, wherein the processing device calculates statistics and probability of a match of the mental profile of the user for any of a range of topics.

0135Example 4 may optionally include the subject matter of any one or more of Examples 1-3, wherein the processing device builds a mental profile of the user as a function of the stimuli based on the temporal and spatial patterns of biophysical signals associated with brain activity of the user.

0136Example 5 may optionally include the subject matter of any one or more of Examples 1-4, wherein the processing device combines the brain signature of the user with personal data and other traits obtained from a database to develop a mental profile model of the user.

0137Example 6 may optionally include the subject matter of any one or more of Examples 1-5, wherein the processing device correlates probabilities between subjects and calculates statistics and probability of a mental match between the mental profile model of the user and mental profile models of at least one other user.

0138Example 7 may optionally include the subject matter of any one or more of Examples 1-6, wherein the processing device provides identification and authentication of a user, wherein a mental profile of a user is created by the processing device during a calibrating stage based on presentation of stimuli from the library of stimuli to the user, the processing device further determining whether a mental profile of a user being authenticated correlates to the mental profile of the user created during the calibration stage.

0139Example 8 may optionally include the subject matter of any one or more of Examples 1-7, wherein the processing device is arranged to perform telepathic contextual search by monitoring transmissions from a brain-computer interface system of a user, displaying stimuli associated with brain activity measurements from the user, searching for the brain activity measurements to locate a search object associated with the brain activity measurements and returning search results based on a match between the brain activity measurements and search objects having the associated brain activity measurements correlated therewith.

0140Example 9 may optionally include the subject matter of any one or more of Examples 1-8, wherein the processing device provides telepathic augmented reality by receiving input from brain-computer interface (BCI) sensors and detectors and a biometric and environmental sensor array, the processing device arranged to map input and data obtained a database of recognized sensor inputs, AR character and AR environmental content to an AR experience, the processing device blending AR characters with the environment and presenting the AR experience to a user based on the user intent derived from the input from the brain-computer interface (BCI) sensors and detectors and the biometric and environmental sensor array.

0141Example 10 may optionally include the subject matter of any one or more of Examples 1-9, wherein the processing device creates a mental desktop representing a left and right hemifield for each of a left and right eyes of a user, the processing device further segregating each eye into an upper division and a lower division, wherein the mental desktop includes eight areas of a visual field of the user having information assigned thereto, the processing device detecting mentally visualization of a region in the mental desktop and implementing a function according to the information assigned to the mentally visualized region.

0142Example 11 may optionally include the subject matter of any one or more of Examples 1-10, wherein the processing device is arranged to analyze receive inputs including temporal and spatial patterns of biophysical signals associated with brain activity of the user and additional input modalities received for implementation with applications, and perceptual computing inputs from the perceptual computing to BCI database, the processing device further arranged to determine an intent of the user based on the inputs and interrelatedness data associated with the inputs obtained from a perceptual computing database and factors obtained from a factor database, wherein the processing device initiates a command based on the determined user intent.

0143Example 12 may optionally include the subject matter of any one or more of Examples 1-11, wherein the processing device is arranged to determine whether interference is occurring and to adjust the wherein the temporal and spatial patterns of biophysical signals of the user to account for the interference.

0144Example 13 may optionally include the subject matter of any one or more of Examples 1-12, further includes a user interface for assigning temporal and spatial patterns of biophysical signals associated with brain activity of the user and additional modality inputs to applications.

0145Example 14 may include or may optionally be combined with the subject matter of any one of Examples 1-13 to include subject matter (such as a method or means for performing acts) for providing stimuli to a user, gathering temporal and spatial patterns of biophysical signals associated with brain activity in response to providing the stimuli to the user, correlating the gathered temporal and spatial patterns of biophysical signals associated with brain activity to identify user brain signatures and performing a processor controlled function based on the user brain signatures identified through correlating the gathered temporal and spatial patterns of biophysical signals associated with brain activity.

0146Example 15 may optionally include the subject matter of Example 14, wherein the processor controlled function includes determining at least one similarity between identified patterns of the user and patterns common to a group of users.

0147Example 16 may optionally include the subject matter of any one or more of Examples 14-15, further includes providing a user with a brain monitoring device and running the user through a series of experiences associated with the stimuli, wherein the correlating the gathered temporal and spatial patterns includes characterizing the gather spatial and temporal brain activity patterns to identify the user brain signatures.

0148Example 17 may optionally include the subject matter of any one or more of Examples 14-16, wherein the performing a processor controlled function further includes building a characteristic mental profile of the user based on the user brain signatures, establishing models of mental predilections and personality traits and using the established models to predict an affinity of the user with an association of people.

0149Example 18 may optionally include the subject matter of any one or more of Examples 14-17, wherein the correlating the gathered temporal and spatial patterns of biophysical signals further includes translating recorded brain activity patterns in response stimuli into a characteristic mental profile associated with the stimuli, maintaining mental profiles to the stimuli for each individual in a database, integrating personal data and traits into the mental profiles, identifying a mental match between the mental profile of the user in response to the stimuli and at least one mental profile of other users associated with the stimuli and providing a probabilistic or percentage score of a mental match.

0150Example 19 may optionally include the subject matter of any one or more of Examples 14-18, wherein the providing stimuli to a user, gathering temporal and spatial patterns of biophysical signals associated with brain activity in response to providing the stimuli to the user and correlating the gathered temporal and spatial patterns of biophysical signals further includes calibrating a brain signature of a user based on the stimuli and authenticating the user by comparing a currently measured brain signature and the calibrated brain signature.

0151Example 20 may optionally include the subject matter of any one or more of Examples 14-19, wherein the calibrating the brain signature of the user includes presenting a set of stimuli to a user to incite brain activity responses, measuring brain anatomy and activity in response to the presented set of stimuli, performing pattern recognition of the measurements of brain anatomy and activity to produce a brain signature of the user, storing the produced brain signature and adding the stored brain signature to a database of anatomical and physiologic brain signatures of a predetermined population.

0152Example 21 may optionally include the subject matter of any one or more of Examples 14-20, wherein the presenting a set of stimuli further includes running the user through thoughts to incite certain characteristic brain activities.

0153Example 22 may optionally include the subject matter of any one or more of Examples 14-21, wherein the running the user through thoughts includes running the user through one selected from a group consisting of a series of memorized thoughts, patterns of muscle activation and imagined activities.

0154Example 23 may optionally include the subject matter of any one or more of Examples 14-22, wherein the measuring brain anatomy and activity includes measuring brain anatomy and activity using at least one of functional near infrared spectroscopy, electroencephalography, magnetoencephalography, magnetic resonance imaging and ultrasound.

0155Example 24 may optionally include the subject matter of any one or more of Examples 14-23, wherein the measuring brain anatomy and activity includes measuring anatomical characteristics.

0156Example 25 may optionally include the subject matter of any one or more of Examples 14-24, wherein the measuring anatomical characteristics includes measuring at least one of gyrification, cortical thickness and scalp thickness.

0157Example 26 may optionally include the subject matter of any one or more of Examples 14-25, wherein the performing pattern recognition of the measurements of brain anatomy and activity further includes performing pattern recognition based on at least one of modified Beer-Lambert Law, event-related components, multi-voxel pattern analysis (MVPA), spectral analysis, and MVPA on fNIRS.

0158Example 27 may optionally include the subject matter of any one or more of Examples 14-26, wherein the performing pattern recognition of the measurements of brain anatomy and activity further includes translating anatomic and physiologic measurements into specific patterns that can be used to categorize a brain for identification and authentication.

0159Example 28 may optionally include the subject matter of any one or more of Examples 14-27, wherein the authenticating the user includes presenting a previously applied set of stimuli to a user to incite brain activity responses, measuring brain anatomy and activity of the user based on the previously applied set of stimuli, performing pattern recognition of the measurements of brain anatomy and activity to produce a brain signature of the user and analyzing the brain signature of the user obtained through the performing the pattern recognition by comparing the brain signature with the calibrated brain signature of the user.

0160Example 29 may optionally include the subject matter of any one or more of Examples 14-28, wherein the analyzing the brain signature of the user includes comparing the brain signature with anatomical and physiologic brain signatures of a predetermined population.

0161Example 30 may optionally include the subject matter of any one or more of Examples 14-29, wherein the analyzing the brain signature of the user includes comparing the brain signature with additional identification and authentication techniques to increase sensitivity and specificity of the identification and authentication techniques.

0162Example 31 may optionally include the subject matter of any one or more of Examples 14-30, wherein the comparing the brain signature with additional identification and authentication techniques includes comparing the brain signature with at least one of handwriting recognition results, a password query and an additional biometric parameter.

0163Example 32 may optionally include the subject matter of any one or more of Examples 14-31, wherein the performing a processor controlled function based on the user brain signatures includes directing a device to perform a function in response to the gathered temporal and spatial patterns of biophysical signals associated with the brain activity.

0164Example 33 may optionally include the subject matter of any one or more of Examples 14-32, wherein the providing stimuli to a user, the gathering temporal and spatial patterns of biophysical signals associated with brain activity in response to providing the stimuli to the user, the correlating the gathered temporal and spatial patterns of biophysical signals and the performing a processor controlled function based on the user brain signatures further includes presenting a set of stimuli to a user, obtaining brain-computer interface (BCI) measurements of the user, identifying candidate brain activity-stimuli pairings from the BCI measurement having reliable correlation with predetermined stimuli, storing candidate brain activity-stimuli pairings, determining brain activity-stimuli pairings having reliable correlation when the user is imagining the stimuli, storing the brain activity-stimuli pairings having reliable correlation when the user is imagining the stimuli and retrieving and displaying the stimuli when a correlated BCI measurement is detected to perform telepathic computer control.

0165Example 34 may optionally include the subject matter of any one or more of Examples 14-33, wherein the presenting a set of stimuli to a user further includes presenting compound stimuli to the user to increase correlation reliability.

0166Example 35 may optionally include the subject matter of any one or more of Examples 14-34, wherein the telepathic computer control includes a telepathic search performed by the user by recreating mental imagery of a stimuli paired with a BCI measure associated with a search object.

0167Example 36 may optionally include the subject matter of any one or more of Examples 14-35, wherein the telepathic search is performed by matching the patterns of thought of a user to a database of content that is categorized to brain patterns of the user developed in response to brain activity measurements associated with the pattern of thought to produce search results and weighting the search results based on a number of elements in the patterns of thought that match with elements known to be associated with content in the database.

0168Example 37 may optionally include the subject matter of any one or more of Examples 14-36, wherein the telepathic search includes a search for an image, wherein the user thinks of the image that is an object of the search and providing results of images that matches brain activity-stimuli pairings to the user's thoughts of the image.

0169Example 38 may optionally include the subject matter of any one or more of Examples 14-37, wherein the telepathic search includes a search for a work of music, wherein the user thinks of sounds associated with the work of music and providing results of music that matches brain activity-stimuli pairings to the user's thoughts of the sounds associated with the work of music.

0170Example 39 may optionally include the subject matter of any one or more of Examples 14-38, wherein the telepathic search includes a telepathic search performed using a combination of multi-voxel pattern analysis (MVPA) and functional near infrared spectroscopy (fNIRS) to identify patterns of distributed brain activity correlated with a particular thought.

0171Example 40 may optionally include the subject matter of any one or more of Examples 14-39, wherein the telepathic computer control includes a telepathic communication, wherein at least two users trained with a common mental vocabulary use the common mental vocabulary to communicate with each other based on brain activity-stimuli pairings.

0172Example 41 may optionally include the subject matter of any one or more of Examples 14-40, wherein a sending user is identified on a user interface of a receiving user, and wherein a sending user thinks of a receiving user to select a receiving user to send a message.

0173Example 42 may optionally include the subject matter of any one or more of Examples 14-41, wherein the telepathic computer control includes a telepathic augmented reality (AR) performed by thinking about a brain activity-stimuli pairing that associates mental imagery with a model to perform a predetermined action.

0174Example 43 may optionally include the subject matter of any one or more of Examples 14-42, wherein the predetermined action includes presenting the user with sensory signals produced by an AR object associated with the brain activity-stimuli pairing, wherein the sensory signals includes visual, audio and tactile signals.

0175Example 44 may optionally include the subject matter of any one or more of Examples 14-43, wherein the predetermined action includes presenting an AR experience not purposefully invoked by the user through monitoring of BCI inputs.

0176Example 45 may optionally include the subject matter of any one or more of Examples 14-44, wherein the predetermined action includes directing movement of AR characters by thinking about a brain activity-stimuli pairing.

0177Example 46 may optionally include the subject matter of any one or more of Examples 14-45, wherein the predetermined action includes an action initiated using the brain activity-stimuli pairing with monitored environmental cues.

0178Example 47 may optionally include the subject matter of any one or more of Examples 14-46, wherein the performing a processor controlled function based on the user brain signatures includes operating computing devices by focusing, by the user, mental attention on different sections of a visual field of the user.

0179Example 48 may optionally include the subject matter of any one or more of Examples 14-47, wherein the providing stimuli to a user, the gathering temporal and spatial patterns of biophysical signals associated with brain activity in response to providing the stimuli to the user, the correlating the gathered temporal and spatial patterns of biophysical signals and the performing a processor controlled function based on the user brain signatures further includes dividing a mental desktop workspace into visuospatial regions based on regions of a field of view of a user, training a user to map a visual field of the user to regions of a primary visual cortex of the user, wherein the regions of the primary visual cortex correspond to one of the visuospatial regions, assigning content to physiologically segregated sections of the visual field represented by the visuospatial regions and accessing assigned information by mentally visualizing one of the visuospatial regions to access content assigned to the visualized visuospatial region.

0180Example 49 may optionally include the subject matter of any one or more of Examples 14-48, wherein the visuospatial regions includes a left and right hemifield for each of a left eye and a right eye, and wherein each hemifield is divided into an upper and lower division.

0181Example 50 may optionally include the subject matter of any one or more of Examples 14-49, wherein the providing stimuli to a user, the gathering temporal and spatial patterns of biophysical signals associated with brain activity in response to providing the stimuli to the user, the correlating the gathered temporal and spatial patterns of biophysical signals and the performing a processor controlled function based on the user brain signatures further includes imagining, by a user, movements of a body location associated with providing a computer input, recording brain activity emerging from a topographically organized brain region dedicated to controlling movements of the corresponding body location, correlating the recorded brain activity in the topographically organized brain region with the movement of the corresponding body location, performing a mental gesture by visualizing movement of the body location to produce activity in the topographically organized brain region, detecting brain activity corresponding to the recorded brain activity and performing a computer input associated with the movement of the corresponding body location in response to detection of the brain activity corresponding to the recorded brain activity.

0182Example 51 may optionally include the subject matter of any one or more of Examples 14-50, further including receiving perceptual computing input, wherein the performing a processor controlled function based on the user brain signatures includes correlating the temporal and spatial patterns of biophysical signals associated with brain activity with the perceptual computing input to determine an intent of the user and initiating a command to control electronic devices based on the determined user intent.

0183Example 52 may optionally include the subject matter of any one or more of Examples 14-51, wherein the receiving perceptual computing input includes receiving gesture, voice, eye tracking and facial expression input.

0184Example 53 may optionally include the subject matter of any one or more of Examples 14-52, wherein the receiving perceptual computing input comprises receiving at least one of: gesture, voice, eye tracking or facial expression input.

0185Example 54 may optionally include the subject matter of any one or more of Examples 14-53, wherein the correlating the gathered temporal and spatial patterns of biophysical signals further includes identifying a pattern to the temporal and spatial patterns of biophysical signals associated with brain activity from the user prior to initiating a command indicating that a next sensor-detected event is a command.

0186Example 55 may optionally include the subject matter of any one or more of Examples 14-54, further including receiving perceptual computing input, wherein the performing the processor controlled function further includes indicating a modality from the brain activity and perceptual computing inputs having precedence.

0187Example 56 may optionally include the subject matter of any one or more of Examples 14-55, further including measuring the temporal and spatial patterns of biophysical signals associated with brain activity of the user contemporaneously with receiving perceptual computing input, and using the contemporaneous temporal and spatial patterns of biophysical signals associated with brain activity of the user and the received perceptual computing input to reinforce an input command.

0188Example 57 may optionally include the subject matter of any one or more of Examples 14-56, wherein the providing stimuli to a user, the gathering temporal and spatial patterns of biophysical signals associated with brain activity in response to providing the stimuli to the user, the correlating the gathered temporal and spatial patterns of biophysical signals and the performing a processor controlled function based on the user brain signatures further includes measuring temporal and spatial patterns of biophysical signals associated with brain activity of the user, determining a state of the user based on the measured temporal and spatial patterns of biophysical signals associated with brain activity of the user and providing a response to the user based on the determined state.

0189Example 58 may optionally include the subject matter of any one or more of Examples 14-57, further including receiving perceptual computing input, wherein the perceptual computing input includes eye tracking to select a target, and wherein the temporal and spatial patterns of biophysical signals of the user are used to act on the target.

0190Example 59 may optionally include the subject matter of any one or more of Examples 14-58, wherein the performing a processor controlled function based on the user brain signatures further includes using temporal and spatial patterns of biophysical signals associated with brain activity of the user to interrupt a system responding to another modality.

0191Example 60 may optionally include the subject matter of any one or more of Examples 14-59, wherein the performing a processor controlled function based on the user brain signatures further includes using temporal and spatial patterns of biophysical signals associated with brain activity of the user to provide feedback to the user when the temporal and spatial patterns of biophysical signals associated with brain activity of the user have been identified and received.

0192Example 61 may optionally include the subject matter of any one or more of Examples 14-60, wherein the performing a processor controlled function based on the user brain signatures further includes alerting a system to change states when a change in state of a user is determined to have changed based on the correlating the gathered temporal and spatial patterns of biophysical signals.

0193Example 62 may optionally include the subject matter of any one or more of Examples 14-61, further including mapping the brain activity of the user to activation of a command window-of-opportunity when the brain activity occurs.

0194Example 63 may optionally include the subject matter of any one or more of Examples 14-62, further including obtaining perceptual computing inputs, gathering data from a database arranged to maintain heuristics on how perceptual computing inputs and the temporal and spatial patterns of biophysical signals of the user interrelate, analyzing the temporal and spatial patterns of biophysical signals of the user, the perceptual computing inputs, and the input from the database to determine user intent and generating a command based on the determined user intent.

0195Example 64 may optionally include the subject matter of any one or more of Examples 14-63, further including measuring environmental and user factors, determining possible interference, and adjusting temporal and spatial patterns of biophysical signals of the user based on the determined possible interference.

0196Example 65 may optionally include the subject matter of any one or more of Examples 14-64, wherein the processor controlled function comprises one selected from a group of actions consisting of performing a telepathic augmented reality (AR) by thinking about a brain activity-stimuli pairing that associates mental imagery with a model to perform a predetermined action, presenting an AR experience not purposefully invoked by the user through monitoring of BCI inputs, directing movement of AR characters by thinking about a brain activity-stimuli pairing, and an action initiated using the brain activity-stimuli pairing with monitored environmental cues.

0197Example 66 may include or may optionally be combined with the subject matter of any one of Examples 1-65 to include subject matter (such as means for performing acts or machine readable medium including instructions that, when executed by the machine, cause the machine to perform acts) including providing stimuli to a user, gathering temporal and spatial patterns of biophysical signals associated with brain activity in response to providing the stimuli to the user, correlating the gathered temporal and spatial patterns of biophysical signals associated with brain activity to identify user brain signatures and performing a processor controlled function based on the user brain signatures identified through correlating the gathered temporal and spatial patterns of biophysical signals associated with brain activity.

0198Example 67 may optionally include the subject matter of Example 66, wherein the providing stimuli to a user, gathering temporal and spatial patterns of biophysical signals associated with brain activity in response to providing the stimuli to the user and correlating the gathered temporal and spatial patterns of biophysical signals further includes calibrating a brain signature of a user based on the stimuli and authenticating the user by comparing a currently measured brain signature and the calibrated brain signature.

0199Example 68 may optionally include the subject matter of any one or more of Examples 66-67, wherein the calibrating the brain signature of the user includes presenting a set of stimuli to a user to incite brain activity responses, measuring brain anatomy and activity in response to the presented set of stimuli, performing pattern recognition of the measurements of brain anatomy and activity to produce a brain signature of the user, storing the produced brain signature and adding the stored brain signature to a database of anatomical and physiologic brain signatures of a predetermined population.

0200Example 69 may optionally include the subject matter of any one or more of Examples 66-68, wherein the authenticating the user includes presenting a previously applied set of stimuli to a user to incite brain activity responses, measuring brain anatomy and activity of the user based on the previously applied set of stimuli, performing pattern recognition of the measurements of brain anatomy and activity to produce a brain signature of the user and analyzing the brain signature of the user obtained through the performing the pattern recognition by comparing the brain signature with the stored brain signature of the user.

0201Example 70 may optionally include the subject matter of any one or more of Examples 66-69, wherein the performing a processor controlled function based on the user brain signatures includes directing a device to perform a function in response to the gathered temporal and spatial patterns of biophysical signals associated with the brain activity.

0202Example 71 may optionally include the subject matter of any one or more of Examples 66-70, wherein the providing stimuli to a user, the gathering temporal and spatial patterns of biophysical signals associated with brain activity in response to providing the stimuli to the user, the correlating the gathered temporal and spatial patterns of biophysical signals and the performing a processor controlled function based on the user brain signatures further includes presenting a set of stimuli to a user, obtaining brain-computer interface (BCI) measurements of the user, identifying candidate brain activity-stimuli pairings from the brain activity measurement having reliable correlation with predetermined stimuli, storing candidate brain activity-stimuli pairings, determining brain activity-stimuli pairings having reliable correlation when the user is imagining the stimuli, storing the brain activity-stimuli pairings having reliable correlation when the user is imagining the stimuli and retrieving and displaying the stimuli when a correlated brain activity measurement is detected to perform telepathic computer control.

0203Example 72 may optionally include the subject matter of any one or more of Examples 66-71, wherein the telepathic computer control includes a telepathic search performed by the user by recreating mental imagery of a stimuli paired with a BCI measure associated with a search object.

0204Example 73 may optionally include the subject matter of any one or more of Examples 66-72, wherein the telepathic search is performed by matching the patterns of thought of a user to a database of content that is categorized to brain patterns of the user developed in response to brain activity measurements associated with the patterns of thought to produce search results and weighting the search results based on a number of elements in the patterns of thought that match with elements known to be associated with content in the database.

0205Example 74 may optionally include the subject matter of any one or more of Examples 66-73, wherein the telepathic computer control includes a telepathic communication, wherein at least two users trained with a common mental vocabulary use the common mental vocabulary to communicate with each other based on brain activity-stimuli pairings.

0206Example 75 may optionally include the subject matter of any one or more of Examples 66-74, wherein a sending user is identified on a user interface of a receiving user.

0207Example 76 may optionally include the subject matter of any one or more of Examples 66-75, wherein a sending user thinks of a receiving user to select a receiving user to send a message.

0208Example 77 may optionally include the subject matter of any one or more of Examples 66-76, wherein the telepathic computer control includes a telepathic augmented reality (AR) performed by thinking about a brain activity-stimuli pairing that associates mental imagery with a model to perform a predetermined action.

0209Example 78 may optionally include the subject matter of any one or more of Examples 66-77, wherein the performing a processor controlled function based on the user brain signatures includes operating computing devices by focusing, by the user, mental attention on different sections of a visual field of the user.

0210Example 79 may optionally include the subject matter of any one or more of Examples 66-78, wherein the providing stimuli to a user, the gathering temporal and spatial patterns of biophysical signals associated with brain activity in response to providing the stimuli to the user, the correlating the gathered temporal and spatial patterns of biophysical signals and the performing a processor controlled function based on the user brain signatures further includes dividing a mental desktop workspace into visuospatial regions based on regions of a field of view of a user, training a user to map a visual field of the user to regions of a primary visual cortex of the user, wherein the regions of the primary visual cortex correspond to one of the visuospatial regions, assigning content to physiologically segregated sections of the visual field represented by the visuospatial regions and accessing assigned information by mentally visualizing one of the visuospatial regions to access content assigned to the visualized visuospatial region.

0211Example 80 may optionally include the subject matter of any one or more of Examples 66-79 further including receiving perceptual computing input, wherein the performing a processor controlled function based on the user brain signatures includes correlating the temporal and spatial patterns of biophysical signals associated with brain activity with the perceptual computing input to determine an intent of the user and initiating a command to control electronic devices based on the determined user intent.

0212Example 81 may optionally include the subject matter of any one or more of Examples 66-80, wherein the receiving perceptual computing input includes receiving gesture, voice, eye tracking and facial expression input.

0213Example 82 may optionally include the subject matter of any one or more of Examples 66-81, wherein the receiving perceptual computing input comprises receiving at least one of: gesture, voice, eye tracking or facial expression input.

0214Example 83 may optionally include the subject matter of any one or more of Examples 66-82, wherein the correlating the gathered temporal and spatial patterns of biophysical signals further includes identifying a pattern to the temporal and spatial patterns of biophysical signals associated with brain activity from the user prior to initiating a command indicating that a next sensor-detected event is a command.

0215Example 84 may optionally include the subject matter of any one or more of Examples 66-83, further including receiving perceptual computing input, wherein the performing the processor controlled function further includes indicating a modality from the brain activity and perceptual computing inputs having precedence.

0216Example 85 may optionally include the subject matter of any one or more of Examples 66-84, further including measuring the temporal and spatial patterns of biophysical signals associated with brain activity of the user contemporaneously with receiving perceptual computing input, and using the contemporaneous temporal and spatial patterns of biophysical signals associated with brain activity of the user and the received perceptual computing input to reinforce an input command.

0217Example 86 may optionally include the subject matter of any one or more of Examples 66-85, wherein the providing stimuli to a user, the gathering temporal and spatial patterns of biophysical signals associated with brain activity in response to providing the stimuli to the user, the correlating the gathered temporal and spatial patterns of biophysical signals and the performing a processor controlled function based on the user brain signatures further includes measuring temporal and spatial patterns of biophysical signals associated with brain activity of the user, determining a state of the user based on the measured temporal and spatial patterns of biophysical signals associated with brain activity of the user and providing a response to the user based on the determined state.

0218Example 87 may optionally include the subject matter of any one or more of Examples 66-86, further including receiving perceptual computing input, wherein the perceptual computing input includes eye tracking to select a target, and wherein the temporal and spatial patterns of biophysical signals of the user are used to act on the target.

0219Example 88 may optionally include the subject matter of any one or more of Examples 66-87, wherein the performing a processor controlled function based on the user brain signatures further includes using temporal and spatial patterns of biophysical signals associated with brain activity of the user to interrupt a system responding to another modality.

0220Example 89 may optionally include the subject matter of any one or more of Examples 66-88, wherein the telepathic computer control comprises one selected from a group of controls consisting of a telepathic search performed by the user by recreating mental imagery of stimuli paired with a BCI measure associated with a search object; a telepathic communication, wherein at least two users trained with a common mental vocabulary use the common mental vocabulary to communicate with each other based on brain activity-stimuli pairings; and a telepathic augmented reality (AR) performed by thinking about a brain activity-stimuli pairing that associates mental imagery with a model to perform a predetermined action.

0221Example 90 may optionally include the subject matter of any one or more of Examples 66-89, wherein the performing a processor controlled function based on the user brain signatures comprises one selected from a group of functions consisting of operating computing devices by focusing detected mental attention on different sections of a visual field of the user; providing a mental desktop by dividing a mental desktop workspace into visuospatial regions based on regions of a field of view of a user, training a user to map a visual field of the user to regions of a primary visual cortex of the user, wherein the regions of the primary visual cortex correspond to one of the visuospatial regions, assigning content to physiologically segregated sections of the visual field represented by the visuospatial regions and accessing assigned information by mentally visualizing one of the visuospatial regions to access content assigned to the visualized visuospatial region; correlating the temporal and spatial patterns of biophysical signals associated with brain activity with the perceptual computing input to determine an intent of the user and initiating a command to control electronic devices based on the determined user intent; performing a processor controlled function based on the user brain signatures by measuring temporal and spatial patterns of biophysical signals associated with brain activity of the user, determining a state of the user based on the measured temporal and spatial patterns of biophysical signals associated with brain activity of the user and providing a response to the user based on the determined state; and using temporal and spatial patterns of biophysical signals associated with brain activity of the user to interrupt a system responding to another modality.

0222The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments that may be practiced. These embodiments are also referred to herein as “examples.” Such examples may include elements in addition to those shown or described. However, also contemplated are examples that include the elements shown or described. Moreover, also contemplate are examples using any combination or permutation of those elements shown or described (or one or more aspects thereof), either with respect to a particular example (or one or more aspects thereof), or with respect to other examples (or one or more aspects thereof) shown or described herein.

0223Publications, patents, and patent documents referred to in this document are incorporated by reference herein in their entirety, as though individually incorporated by reference. In the event of inconsistent usages between this document and those documents so incorporated by reference, the usage in the incorporated reference(s) are supplementary to that of this document; for irreconcilable inconsistencies, the usage in this document controls.

0224In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Also, in the following claims, the terms “including” and “comprising” are open-ended, that is, a system, device, article, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to suggest a numerical order for their objects.

0225The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more aspects thereof) may be used in combination with others. Other embodiments may be used, such as by one of ordinary skill in the art upon reviewing the above description. The Abstract is to allow the reader to quickly ascertain the nature of the technical disclosure, for example, to comply with 37 C.F.R. §1.72(b) in the United States of America. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features may be grouped together to streamline the disclosure. However, the claims may not set forth features disclosed herein because embodiments may include a subset of said features. Further, embodiments may include fewer features than those disclosed in a particular example. Thus, the following claims are hereby incorporated into the Detailed Description, with a claim standing on its own as a separate embodiment. The scope of the embodiments disclosed herein is to be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.