Emotion type classification for interactive dialog system

[0001] Artificial interactive dialog systems are an increasingly popular feature in state-of-the-art home electronic devices. For example, today's wireless smartphones incorporate voice recognition, interactive dialogs, and speech synthesis software for real-time interactive conversations with users, providing these services for information and news, remote device configuration and programming, and conversations. Deliver as an intimate relationship.

[0002] It is desirable to generate audio or other output with emotional content in addition to semantic content so that the user can experience a more natural and seamless conversation with the dialog system. For example, when delivering news, scheduling tasks, or otherwise interacting with the user, emotional features in the synthesized audio and / or other output to make the user more effective in the conversation. It is desirable to give.

<p>[0003] Therefore, in order to determine the emotions suitable for imparting to the semantic content delivered by the interactive dialog system, and the emotions thus determined are one of a plurality of predetermined emotion types. It is desirable to provide a technique for classifying according to one.</p>

<p>[0004] Means for solving the present problems are provided to introduce in a simplified form the selection of concepts further described in the form for carrying out the invention below. The means for solving this problem are not intended to identify the main or important function of the claimed subject matter, nor are they intended to be used to limit the scope of the claimed subject matter. Absent.</p><p>[0005] Simply put, the various aspects of the subject matter described herein are directed to techniques for providing devices for interactive dialog systems. In some embodiments, the fact or profile input available to the mobile communication device may be combined with previous or current user input to select the appropriate emotion type code to associate with the output statement generated by the interactive dialog system. it can. For example, facts or profile inputs can be derived from certain aspects of device usage, such as user online activity, user communications, calendar and scheduling capabilities. Algorithms for selecting emotion type codes can be rule-based or preconfigured using machine learning techniques. Emotion type codes can be combined with output statements to generate synthetic speech with emotional features for an improved user experience.</p><p>[0006] Other advantages will become apparent from the detailed description and drawings below.</p>

<figref num="1">[0007] It is a figure which shows the scenario which adopts the mobile communication device to which the technique of this disclosure is applied.</figref><figref num="2">[0008] FIG. 6 illustrates an exemplary embodiment of a process that can be performed by a device processor and other elements.</figref><figref num="3">[0009] FIG. 5 is a diagram illustrating an exemplary embodiment of processing performed by a dialog engine.</figref><figref num="4">[0010] FIG. 6 illustrates an exemplary embodiment of an emotion type classification block according to the present disclosure.</figref><figref num="5">[0011] It is a figure which shows the exemplary embodiment of the hybrid emotion type classification algorithm.</figref><figref num="6">[0012] FIG. 6 illustrates an exemplary embodiment of a rule-based algorithm.</figref><figref num="7">[0013] FIG. 6 illustrates an alternative exemplary embodiment of a rule-based algorithm.</figref><figref num="8">[0014] FIG. 6 illustrates an exemplary embodiment of a training scheme for deriving an algorithm trained to select an emotion type.</figref><figref num="9">[0015] FIG. 6 illustrates an exemplary embodiment of a method according to the present disclosure.</figref><figref num="10">[0016] FIG. 5 schematically illustrates a non-limiting computing system capable of performing one or more of the methods and processes described above.</figref><figref num="11">[0017] FIG. 6 illustrates an exemplary embodiment of an apparatus according to the present disclosure.</figref><figref num="12">[0018] FIG. 6 illustrates an exemplary embodiment in which the techniques of the present disclosure are incorporated into a dialog system in which emotional content is added to the display text instead of or in addition to the audible audio.</figref>

[0019] Various aspects of the techniques described herein are generally intended for techniques for selecting an emotion type code associated with an output statement within an electronically interactive dialog system. The detailed description given below with respect to the accompanying drawings is intended as an illustration of exemplary embodiments of the invention and is not intended to represent merely exemplary embodiments in which the invention is feasible. As used throughout this description, the term "exemplary" means "acting as an example, instance, or illustration" and is not necessarily construed as preferred or advantageous over other exemplary embodiments. Should not be. The detailed description includes specific details for a complete understanding of exemplary embodiments of the invention. Those skilled in the art will appreciate that exemplary embodiments of the invention can be practiced without these particular details. In some instances, well-known structures and devices are shown in the form of block diagrams to avoid obscuring the novelty of the exemplary embodiments presented herein.

[0020] FIG. 1 shows a scenario in which the mobile communication device 120 to which the technique of the present disclosure is applicable is adopted. It should be noted that FIG. 1 is provided for illustration purposes only and does not imply limiting the scope of this disclosure to the application of this disclosure to mobile communications devices. For example, the techniques described herein are readily applicable in other devices and systems, such as human interface systems for notebook and desktop computers, automotive navigation systems, and the like. Such alternative applications are intended to be within the scope of this disclosure.

[0021] In FIG. 1, the user 110 communicates with a mobile communication device 120, such as a handheld smartphone. It can be seen that smartphones include any mobile device with integrated communication capabilities, such as voice calls and Internet access with a relatively advanced microprocessor to implement a diverse array of computational tasks. User 110 can provide voice input 122 to microphone 124 on device 120. One or more processors 125 in device 120, and / or processors (not shown) available over the network (eg, implementing cloud computing schemes), see, for example, Figure 2 below. It is capable of processing audio signals received by the microphone 124, which performs functions as described further. Processor 125 does not have to have any particular shape, shape, or functional division as described herein for illustration purposes only, and such processors are generally known in various techniques in the art. Note that it can be implemented using.

[0022] Based on the processing performed by processor 125, device 120 can use audio speaker 128 to generate audio output 126 in response to audio input 122. In some scenarios, device 120 can also generate audio output 126 independent of audio input 122, for example device 120 can autonomously provide alerts or other users (not shown). The message from can be relayed to user 110 in the form of voice output 126. In an exemplary embodiment, the output in response to voice input 122 can also be displayed on display 129 of device 120, for example as text, graphics, animation, and the like.

[0023] FIG. 2 illustrates an exemplary embodiment of an interactive dialog system 200 that can be executed by the processor 125 of device 120 and other elements. It should be noted that the processing shown in FIG. 2 is for illustration purposes only and does not limit the scope of the present disclosure to any particular sequence or set of operations shown in FIG. For example, in an alternative exemplary embodiment, certain techniques disclosed herein for selecting emotion type codes are applicable regardless of the process shown in FIG. In addition, since the blocks shown in FIG. 2 can be combined or omitted depending on the particular functional division in the system, FIG. 2 shows any functional dependence of the blocks shown. It also does not imply gender or independence. These alternative exemplary embodiments are intended to be within the scope of this disclosure.

[0024] In FIG. 2, the audio input is received at block 210. The voice input 210 may correspond to a waveform representing an acoustic signal derived from, for example, the microphone 124 on the device 120. The output 210a of the audio input 210 may correspond to a digitized version of the acoustic waveform, including audio content.

[0025] At block 220, speech recognition is performed for output 210a. In an exemplary embodiment, speech recognition 220 converts speech as present in output 210a into text. The output 220a of the speech recognition 220 may correspondingly correspond to the textual representation of the speech present in the digitized acoustic waveform output 210a. For example, if the output 210a, such as picked up by the microphone 124, contains an audio waveform representation of human utterances such as "How is the weather tomorrow?", The voice recognition 220 will be based on its voice recognition capabilities to "weather tomorrow." You can output ASCII text (or other textual representation) that corresponds to the text "How is it?". Speech recognition, such as that performed by block 220, can be performed using acoustic and linguistic modeling techniques, including, for example, Hidden Markov Models (HMMs), neural networks, and so on.

[0026] At block 230, language comprehension is performed for output 220a of speech recognition 220, based on the predicted natural language knowledge of output 210a. In exemplary embodiments, natural language understanding techniques such as parsing and grammatical parsing can be performed using, for example, form and syntax knowledge to derive the intended meaning of the text in output 220a. The output 230a of the language comprehension 230 can include a formal representation of the semantic and / or emotional content of the speech present within the output 220a.

[0027] At block 240, the dialog engine produces a suitable response to speech as determined from output 230a. For example, if the language comprehension 230 determines that the user voice input corresponds to a query about the weather in a particular geography, the dialog engine 240 retrieves the required weather information from a source such as a weather forecast service or database. Can be assembled. For example, the retrieved weather information can correspond to a time / date code for the weather forecast, a weather type code for "sunny" weather, and a temperature field for an average temperature of 72 degrees Celsius (22.2 degrees Celsius). ..

[0028] In an exemplary embodiment, the dialog engine 240 can further "package" the retrieved information so that it can be presented for quick understanding by the user. Therefore, the semantic content output 240a of the dialog engine 240 can correspond to the expression of the semantic content such as "Today's weather is sunny and the temperature is 72 degrees Celsius".

[0029] In addition to the semantic content 240a, the dialog engine 240 can further generate the emotion type code 240b associated with the semantic content 240a. The emotion type code 240b can indicate a particular type of semantic content to be attached to the semantic content 240a when delivered to the user as output audio. For example, if a user is planning a picnic for a day, he or she can simultaneously deliver a sunny weather forecast with an emotionally cheerful voice. In this case, the emotion type code 240b can be described as an emotional content type corresponding to "moderate happiness". Techniques for generating emotion type code 240b based on the data, facts, and inputs available to the interactive dialog system 200 will be further described below with reference to, for example, FIG.

[0030] At block 250, language generation is performed for outputs 240a and 240b of the dialog engine 240. Language generation presents the output of the Dialog Engine 240 in natural language format, for example sentences in the target language according to vocabulary and grammatical rules, for quick understanding by human users. For example, based on semantic content 240a, language generation 250 can generate the statement "Today's weather will be sunny, 72 degrees."

[0031] In an exemplary embodiment, block 250 can further accept input 255a from system personality block 255. The system personality block 255 can specify the default parameter 255a for the dialog engine according to the "personality" preselected for the interactive dialog system. For example, if the system personality is selected as "male" or "female", or "cheerful" or "thoughtful", block 255 can specify the parameter corresponding to the system personality as the reference input 255a. In one exemplary embodiment, block 255 may be omitted or its functionality may be incorporated into another block, such as dialog engine 240 or language generation block 250, and such alternative exemplary embodiments are disclosed. Note that it is intended to be within the scope of.

[0032] In an exemplary embodiment, the language generation block 250 can combine the semantic content 240a, the emotion type code 240b, and the default emotion parameter 255a to synthesize the output statement 250a. For example, the emotion type code 240b for "moderate happiness" puts a sentence in natural language (eg English) on the block, such as "Great news, today's weather will be sunny, 72 degrees!" Can be generated. Subsequent text-speech block 260 is provided with output statement 250a of language generation block 250 to generate audio audio corresponding to output statement 250a.

[0033] Note that in some exemplary embodiments, some features of the language generation block 250 described above can be omitted. For example, language generation block 250 does not necessarily reflect emotion type code 240b when generating output statement 250a, but instead provides text-speech to provide complete emotional content in the synthesized speech output. You can rely on block 260 (which also has access to emotion type code 240b). In addition, the language generation block 250 can be effectively bypassed in some instances where the information retrieved by the dialog engine is already in natural language format. For example, the Internet weather service accessed by the dialog engine 240 can provide weather updates directly in a natural language such as English, so the language generation 250 does not necessarily have any substantial post-processing on the semantic content 240a. May not even need to be executed. These alternative exemplary embodiments are intended to be within the scope of this disclosure.

[0034] At block 260, a text-to-speech conversion is performed on the output 250a of language generation 250. In an exemplary embodiment, emotion type code 240b is also provided in TTS block 260 to synthesize speech with text content corresponding to 250a and emotion content corresponding to emotion type code 240b. The output of the text-to-speech conversion 260 can be an audio waveform.

[0035] At block 270, an acoustic output is generated from the output of the text-to-speech conversion 260. Speaker 128 on device 120 can provide audio output to listeners, such as user 110 in FIG.

[0036] As interactive dialog systems become more sophisticated, it is desirable to provide techniques for effectively selecting emotional type codes and other types of output that are suitable for the speech produced by these systems. .. For example, in one application, the audio output 270 is not only generated as an emotionally neutral textual representation, but also delivered to the listener, as proposed by providing emotional type code 240b with semantic content 240a. It is also desirable to incorporate pre-specified emotional content when it is done. Thus, the output statement 250a can be associated with the preferred emotion type code 240b so that the user 110 perceives that the appropriate emotional content is in the audio output 270.

[0037] For example, if the dialog engine 240 specifies that the semantic content 240a corresponds to the information that a baseball team has won the World Series, and that the user 110 is a fan of that baseball team, the user's Choosing emotion type code 240b to represent "excitement" (eg, as opposed to neutral or unhappy) to match emotional states results in a more satisfying interactive experience for user 110. there is a possibility.

[0038] Figure 3 illustrates an exemplary embodiment of the process performed by the dialog engine 240 to generate appropriate semantic content as well as associated emotion type codes. It should be noted that FIG. 3 is provided for illustration purposes only and does not limit the scope of this disclosure to any particular application of the techniques described herein.

[0039] In FIG. 3, the dialog engine 240.1 includes a semantic content generation block 310 and an emotion type classification block 320, also referred to herein as a "classification block." Both blocks 310 and 320 are provided with user dialog input 230a, which is the output of language comprehension 230 executed by user 110 on one or more statements or queries in the current or any previous dialog session. Can be included. In particular, the semantic content generation block 310 generates semantic content 240.1a corresponding to the information that will be delivered to the user, while the emotion type classification block 320 assigns to the semantic content 240.1a. Generate the appropriate emotion type represented by emotion type code 240.1b, which will be. Note that user dialog input 230a can be understood to contain any or all of user input from current or previous dialog sessions, such as stored in a history file in local device memory. ..

[0040] In addition to the user dialog input 230a, block 320 is further provided with a "fact or profile" input 301, which can contain parameters derived from the use of the device on which the dialog engine 240.1 is implemented. .. The emotion type classification block 320 is, for example, a combination of fact or profile input 301 and user dialog input 230a according to one or more algorithms, with parameters trained offline according to the machine learning techniques further disclosed below. Based on this, the appropriate emotion type code 240.1b can be generated. In an exemplary embodiment, the emotion type code 240.1b indicates that the emotion (eg, "happiness"), as well as the emotion (using, for example, 5 of the numbers 1-5 that indicates "very happy"). Degree indicators can be included to indicate the degree represented. In an exemplary embodiment, the emotion type code 240.1b is specified in emotion markup language (Emotion ML) to specify one of a plurality of predetermined emotion types that can be assigned to the output voice. It can be expressed in the following format.

[0041] For current consumer devices such as smartphones, the current trend is to play the role of an essential personal assistant that integrates a diverse set of features into a single mobile device that is often and often continuously carried by users. Keep in mind that it is becoming something to carry. To select emotion type code 240.1b by the interactive dialog system 200 by repeated use of these devices by a single user for a wide variety of purposes (eg voice communication, internet access, schedule planning, recreation, etc.) Allows potential access to vast amounts of related data. For example, if location services for smartphones are feasible, using data about a user's geographic location over a period of time, for example, a fan of a local sports team or going to a new restaurant in an area. You can infer the user's geographic preferences, such as the tendency to see. Other examples of using scenarios that generate relevant data include accessing the Internet using a smartphone to perform topic or keyword searches, scheduling calendar dates or appointments, and initializing the device. Includes, but is not limited to, setting up user profiles. Such data is collectively available by the dialog system to evaluate the emotion type code 240.1b appropriate for imparting to the semantic content 240.1a during an interactive dialog session with user 110. In view of these usage scenarios, it is particularly advantageous to derive at least one or more facts or profile inputs 301 from the use of mobile communication devices that implement an interactive dialog system.

[0042] Figure 4 shows an exemplary embodiment of the emotion type classification block according to the present disclosure. In FIG. 4, the exemplary fact or profile input 301.1 available by device 120 includes a plurality of facts or profile parameters 402-422 selected by the system designer to be relevant to the task of emotion type classification. Note that the exemplary facts or profile input 301.1 is given for illustration purposes only. In an alternative exemplary embodiment, it is possible to omit any of the individual parameters of fact or profile input 301.1 and / or add other parameters not shown in FIG. Parameters 402-422 do not necessarily describe disjoint classes of parameters, i.e., the single type of input used by emotion type classification block 320.1 is in two or more categories of inputs 402-422. May enter at the same time. These alternative exemplary embodiments are intended to be within the scope of this disclosure.

[0043] User configuration 402 includes information that is directly entered by user 110 into device 120 that is useful for emotion type classification. In an exemplary embodiment, during the setup of the device 120, or generally during the operation of the device 120, the user 110 may be required to answer a series of profile questions. For example, user 110 may be asked about age and gender, hobbies, interests, favorite movies, sports, personality traits, and so on. In some instances, information about a user's personality traits (eg, extrovert or introversion, dominant or obedience, etc.) can be inferred by asking questions from a personality profile questionnaire. Information from user configuration 402 can be stored for later use by emotion type classification block 320.1 to select emotion type code 240.1b.

[0044] User online activity 404 includes content of Internet usage statistics and / or data transmitted over device 120 to and from the Internet or other networks. In an exemplary embodiment, the online activity 404 can include a user search query, such as submitted to a web search engine via device 120, for example. The content of user search queries is noted and can be other statistics such as frequency and / or timing of similar queries. In an exemplary embodiment, the online activity 404 may further include identifying frequently visited websites, the content of email messages, and posting to social media websites.

[0045] User communication 406 includes text or voice communication performed using device 120. Such communications can include, for example, text messages sent via short messaging services (SMS), voice calls over wireless networks, and the like. User communication 406 can also include messaging on native or third party social media networks, such as internet websites accessed by user 110 using device 120, or instant messaging or chat applications. ..

[0046] User location 408 is a user who has access to device 120 via wireless communication, for example, with one or more cellular base stations, or Internet-based location services, if available. A record of the position can be included. User location 408 can further specify the user's location context, such as when the user is at home or at work, in a car, in a crowded environment, or in a conference room.

The calendar / scheduling function / local date and time 410 may include time information such as those related to emotion classification based on the user's schedule of activities. For example, such information can be premised on the use of device 120 by user 110 as a personal scheduling organizer. In an exemplary embodiment, whether a time segment on the user's calendar is available or unavailable may be related to the classification of emotion types. In addition, the nature of future appointments, such as scheduled vacations or significant opportunities, may also be relevant.

[0048] The calendar / scheduling feature / local date and time 410 can further incorporate information such as whether a time overlaps with the user's working hours or whether the current date corresponds to a weekend. ..

The user emotional state 412 contains data regarding the determination of the user's real-time emotional state. Such data can include the content of the user's utterances to the dialog system, as well as voice parameters, physiological signals, and the like. Emotion recognition techniques, such as those sensed by various sensors on device 120 (eg, physical sensor input 420), such as user conversations, facial expressions, and recent text messages communicated with device 120. It can be further utilized in estimating the user's emotions by sensing physiological signs including body temperature and heart rate.

[0050] The device usage statistic 414 can include information about how often the user 110 uses the device 120, how long and for what purpose the user used the device 120, and so on. In an exemplary embodiment, it is possible to record the time and frequency of user interactions with the device 120 throughout the day, as well as the applications used or the websites visited during those interactions.

[0051] The online information resource 416 may include news or events relating to the user's interests, such as those obtained from online sources. For example, based on the determination that user 110 is a fan of a sports team, the online information resource 416 can include news that the sports team has recently won a match. Alternatively, for example, if the user 110 decides to prefer a type of food, the online information resource 416 may include news that a new restaurant of that type has just opened near the user's home. it can.

[0052] Since the Digital Assistant (DA) personality 418 can specify a personality profile for the dialog system, the user's interaction with the dialog system will more closely mimic the interaction with the human assistant. The DA personality profile can specify, for example, whether the DA is extroverted or introverted, dominant or obedient, or the gender of the DA. For example, the DA personality 418 can specify a profile for a digital assistant that corresponds to a woman's cheerful personality. Note that this feature can be provided as an alternative to or in connection with system personality block 255 as described above with reference to FIG.

[0053] The physical sensor input 420 can include a signal derived from a sensor on the device 120 for sensing the physical parameters of the device 120. For example, the physical sensor input 420 can include sensor signals from an accelerometer and / or gyroscope in device 120, for example to determine if user 110 is currently walking or in a vehicle. Knowledge of the user's current mobility can provide information to emotion type classification block 320.1. The physical sensor input 420 can also include sensor signals from a microphone or other acoustic recording device on device 120, for example to infer environmental characteristics based on background noise.

[0054] The conversation history 422 can include any recording of current and past conversations between the user and the digital assistant.

[0055] Fact or profile input 301.1 and user dialog input 230a can be provided as inputs to the emotion type classification algorithm 450 of emotion type classification block 320.1. The emotion type classification algorithm 450 specifies a multidimensional vector specified by specific fact or profile input 301.1 and user dialog input 230a, eg, an appropriate emotion type and corresponding degree of emotion of emotion type code 240.1b. It can be mapped to a specific output decision.

[0056] FIG. 5 shows an exemplary embodiment of the hybrid emotion type classification algorithm 450.1. It should be noted that FIG. 5 is provided for illustration purposes only and does not limit the scope of this disclosure to any particular type of algorithm illustrated.

[0057] In FIG. 5, the emotion type classification algorithm 450.1 includes an algorithm selection block 510 for selecting at least one algorithm that will be used to select the emotion type. In an exemplary embodiment, the at least one algorithm includes a rule-based algorithm 512 and a trained algorithm 514. Rule-based algorithm 512 is capable of addressing algorithms specified by the designer of the dialog system and is generally designed to assign a given emotion type to a particular scenario, fact, profile, and / or user dialog input. It can be based on a basic principle that can be discerned by a person. On the other hand, the trained algorithm 514 can correspond to an algorithm whose parameters and functional mappings are derived, for example, offline from a large set of training data. The interrelationships between inputs and outputs within the trained algorithm 514 may be less transparent to the system designer than within the rule-based algorithm 512, and the trained algorithm 514 is generally determined from algorithm training. It will be understood that more complex interdependencies between such variables can be captured.

[0058] As seen in FIG. 5, both the rule-based algorithm 512 and the trained algorithm 514 can accept fact or profile input 301.1 and user dialog input 230a as inputs. The algorithm selection block 510 can select the appropriate one of algorithms 512 or 514 to use to select the emotion type code 240.1b in any instance. For example, in response to a fact or profile input 301.1 and / or user dialog input 230a corresponding to a given set of values, the selection block 510 may implement a particular rule-based algorithm 512 instead of the trained algorithm 514. You can choose or vice versa. In an exemplary embodiment, rule-based algorithm 512, in some cases from trained algorithm 514, where, for example, those designs based on basic principles result in a more accurate classification of emotion types. May also be preferable. The rule-based algorithm 512 may also be preferred, for example, in certain scenarios where sufficient training data is not available to design a type of trained algorithm 514. In an exemplary embodiment, rule-based algorithm 512 can be selected if it is relatively easy for the designer to derive the expected response based on a particular set of inputs.

[0059] FIG. 6 shows an exemplary embodiment 600 of a rule-based algorithm. FIG. 6 is shown for illustration purposes only and covers the scope of this disclosure to a rule-based algorithm, any particular implementation of a rule-based algorithm, or any particular fact or profile input 301.1 or emotion type 240b. Please note that it is not limited to format or content.

[0060] In FIG. 6, decision block 610 determines whether the user emotional state 412 is "happy." If not happy, the algorithm goes to block 612 and sets emotion type code 240b to "neutral". If happy, the algorithm proceeds to decision block 620.

[0061] In decision block 620, it is further determined whether the personality parameter 402.1 of user configuration 402 is "extroverted". If not extroverted, the algorithm proceeds to block 622 and sets the emotion type code 240b to "interested (1)", which indicates that the emotion type "interested" has a degree of 1. If extroverted, the algorithm goes to block 630 and sets emotion type code 240b to "happy (3)".

[0062] Rule-based algorithm 600 selectively based on the user's personality, assuming that extroverted users are more likely to be interested in a dialog system that represents a more cheerful or "happier" emotion type. It will be understood to set the emotion type code 240b. Further, the rule-based algorithm 600 assumes that a currently happy user responds more positively to a system that also has a happy emotion type, and sets the emotion type code 240b based on the current user emotional state. In an alternative embodiment, it may be facilitated to design other rule-based algorithms not expressly described herein to associate emotion type code 240b with other parameters and values of fact or profile input 301.1. it can.

[0063] As indicated by Algorithm 600, the determination of emotion type code 240b does not always utilize all available parameters in fact or profile input 301.1 and user dialog input 230a. In particular, algorithm 600 utilizes only the user emotional state 412 and the user configuration 402. Such exemplary embodiments of algorithms that utilize any subset of the available parameters, as well as alternative exemplary embodiments of algorithms that utilize parameters not expressly described herein, are within the scope of the present disclosure. It is intended to be in.

[0064] Figure 7 shows an alternative exemplary embodiment 700 of the rule-based algorithm. In FIG. 7, decision block 710 determines whether user dialog input 230a corresponds to a user query for updated news. If so, the algorithm proceeds to decision block 720.

[0065] In decision block 720, whether the user emotional state 412 is "happy" and whether the online information resource 416 indicates that the user's favorite sports team has just won the match. ,It is determined. In an exemplary embodiment, the user's favorite sports team derives itself from facts or other parameters of profile input 301.1, such as user configuration 402, user online activity 404, calendar / scheduling function 410, and so on. Can be done. If the output of decision block 720 is affirmative, the algorithm proceeds to block 730 and emotion type code 240b is set to "excitement (3)".

[0066] In addition to the rule-based algorithm for selecting emotion type code 240b, emotion type classification algorithm 450.1 can utilize a trained algorithm as an alternative or in connection with it. FIG. 8 shows an exemplary embodiment 800 of a training scheme for deriving a trained algorithm to select an emotion type. It should be noted that FIG. 8 is provided for illustration purposes only and does not limit the scope of this disclosure to any particular technique relating to training algorithms for selecting emotion types.

[0067] In Figure 8, during training stage 801 the algorithm training block 810 is filled with a series or multiple reference facts or profiles 301.1.^*, Pre-user input of corresponding set of criteria 230a^*, And the corresponding set of criteria emotion type code 240.1b^*Inputs are provided, including. Note that in the present specification, the parameter x {x} enclosed in parentheses indicates a plurality of or a series of objects x. In particular, each criterion fact or profile input 301.1^*Corresponds to a particular combination of facts or settings in profile input 301.1.

[0068] For example, one exemplary reference fact or profile input 301.1^*To "Seattle" with user location 408 as the city of residence, to include user configuration 402 to include the "extrovert" personality type, and to include multiple instances of online search for the phrase "Seahawks" with user online activity 404. It can be specified to correspond. This reference fact or profile input 301.1^*Corresponding to the reference user dialog input 230a^*Can include user queries for the latest sports news. In an alternate instance, enter this reference fact or profile 301.1^*Criteria corresponding to user dialog input 230a^*Can be a null string that does not indicate any previous user input. Criteria fact or profile input 301.1^*And the corresponding reference user dialog input 230a^*Based on this exemplary combination of, during training stage 801 to algorithm training block 810 with reference emotion type code 240.1b^*Can be specified.

[0069] In an exemplary embodiment, reference fact or profile input 301.1^*And user dialog input 230a^*Criteria for specific settings of emotion type code 240.1b^*Can be supplied by a human annotator or judge. During training stage 801 these human annotators can be presented with individual combinations of reference facts or profile inputs and reference user inputs, and in response to this situation, the emotions appropriate for each combination. You can annotate types. This process can be iterated using many human annotators and many combinations of reference facts or profile inputs and pre-user inputs, thus utilizing large amounts of training data for algorithm training block 810. it can. Based on the training data and the emotion type annotation of the reference, the optimal set of trained algorithm parameters 810a can be derived for the trained algorithm that most accurately maps a given combination of reference inputs to the reference output. ..

[0070] In an exemplary embodiment, a human annotator can have certain characteristics that are similar to or identical to the corresponding characteristics of the digital assistant's personality. For example, a human annotator can have the same gender or personality type as the preconfigured features of the digital assistant, for example, as specified by the system personality 255 and / or the digital assistant personality 418.

[0071] Algorithm training block 810 is a reference fact or profile input 301.1^*, User dialog input 230a^*, And the reference emotion type code 240.1b^*Configured to derive a set of algorithmic parameters, such as weights, structures, coefficients, etc., that optimally map each combination of inputs to the supplied reference emotion type in response to multiple supplied instances of. .. In exemplary embodiments, techniques from machine learning, such as management learning, can be utilized that optimally derive general rules for mapping inputs to outputs. Correspondingly, the algorithm training block 810 produces an optimal set of trained algorithm parameters 810a provided in exemplary embodiment 514.1 of the trained algorithm block 514 as shown in FIG. In particular, block 514.1 selects emotion type 240.1b during real-time operation 802 according to the trained algorithm parameter 810a.

[0072] Further, exemplary applications of the techniques of the present disclosure will be described below. It should be noted that this example is provided for illustration purposes only and does not limit the scope of this disclosure to any particular set or type of fact or profile input, system response, or scenario. ..

[0073] Mark is a football fan. He's always paying attention to the news about the National Football League (NFL). His favorite team in Seattle is the Seattle Seahawks. Every Sunday, Mark watches football games online on his smartphone and discusses players and teams with his friends through an online chat application. He also shares his activities and interests on social media applications. A few months ago, when the Seahawks defeated the 49ers in overtime, he was very excited to discuss the victory extensively on his social media profile page.

[0074] One Monday, the Seahawks were playing against the 49ers at the San Francisco Monday Night Football. Unfortunately, Mark was having dinner with his client and overlooked the match. The supper was an important deal for Mark, and he was likely to close the deal. As a result, the deal went very well and Mark's proposal was approved by the client. After dinner, Mark activated an interactive dialog system (or Digital Assistant DA) on his smartphone for real-time voice conversations in the car returning home.

[0075] Mark: "Hey, it was a great supper! I signed a contract. Dan (Mark's boss) will surely be pleased."

[0076] Digital Assistant (in a cheerful voice): "Yeah! Good!" (DA response 1)

[0077] Mark: "By the way, how was the Seahawks, tell me!"

[0078] Digital Assistant (in an excited voice): "Guess. Your Seahawks won! You beat the 49ers 30-25. Russell Wilson made two touchdowns in the fourth quarter. I've decided on a path. "(DA response 2)

[0079] Mark: "Wow, that's amazing. I'm sorry I couldn't see the match. I'm going to the playoffs again this year!"

[0080] Digital Assistant (still excited, a little restrained): "Yeah! I have to block your calendar during the playoffs. You don't want to miss it!" (DA Response 3)

[0081] The above example illustrates some aspects of the technique of the present disclosure. In particular, the interactive dialog system also knows that Mark is a football fan and a Seahawks fan. The system obtains this information from an explicit setting that Mark has configured on his digital assistant, for example, indicating that Mark wants to track football news, and that his favorite team is the Seahawks. get. The DA also knows from online sources that the Seahawks played against rival San Francisco 49ers that night, and that the Seahawks had a come-from-behind victory. This allows the DA to select the emotion type that corresponds to the excited tone voice (DA Response 2) when reporting the Seahawks victory news to Mark. In addition, DA selects an excited tone voice (DA Response 3) when proposing time to block Mark's calendar, based on his knowledge of Mark's preferences and his pre-input.

[0082] In addition, the dialog system may include, for example, Mark's smartphone usage patterns (eg, frequency of use, usage time, etc.), personal interests and hobbies indicated by Mark during the setup of his smartphone, and his social media. It has information about the personality of the mark, which is derived from the status update for the network. In this example, the dialog system is based on a machine learning algorithm designed to handle the large amount of statistics generated by his phone usage patterns to infer Mark's personality, and Mark is extroverted sincere. You can decide to be human.

[0083] Further information is derived from the fact that Mark started the DA system two months ago and that he has been using DA regularly and more and more frequently since then. Last week, Mark interacted with DA an average of five times a day. In an exemplary embodiment, one emotion type classification algorithm can infer that the frequency of these dialogues increases the intimacy between Mark and DA.

[0084] In addition, DA determines from his voice that Mark's current emotional state is happy. By using the calendar / scheduling features on his device, DA knows that he is after working hours and that Mark has just closed a deal with a client. During the dialogue, the DA will say, for example, the establishment of a wireless Bluetooth® connection with the car's electronics, the rest period following the accelerometer-determined walking period, the low level of background noise in the car, the measured speed of movement, etc. To identify that the mark is in the car. Furthermore, from past data such as position data history that matches the time statistics, it can be inferred that Mark has returned home by car after dinner. Therefore, by a classification algorithm as described with reference to block 450.1 in FIG. 4, DA selects the emotion type corresponding to the cheerful tone voice (DA response 1).

[0085] FIG. 9 shows an exemplary embodiment of Method 900 according to the present disclosure. It should be noted that FIG. 9 is provided for illustration purposes only and does not limit the scope of the present disclosure to any particular method illustrated.

[0086] In Figure 9, at block 910, the method involves selecting an emotion type code associated with an output statement based on at least one fact or profile input, where the emotion type code is a plurality of predetermined emotions. Specify one of the types.

[0087] At block 920, the method comprises generating a voice corresponding to an output statement, the generated voice having a predetermined emotion specified by an emotion type code. In an exemplary embodiment, at least one fact or profile input is derived from the use of a mobile communication device that implements an interactive dialog system.

[0088] FIG. 10 schematically illustrates a non-limiting computing system 1000 capable of performing one or more of the methods and processes described above. The computing system 1000 is shown in abbreviated form. It will be appreciated that virtually any computer architecture can be used without departing from the scope of this disclosure. In different embodiments, the computing system 1000 is a mainframe computer, server computer, cloud computing system, desktop computer, laptop computer, tablet computer, home entertainment computer, network computing device, mobile computing device, mobile communication device. , Smartphones, gaming devices, etc.

[0089] The computing system 1000 includes a processor 1010 and a memory 1020. The computing system 1000 can optionally include a display subsystem, a communication subsystem, a sensor subsystem, a camera subsystem, and / or other components not shown in FIG. The computing system 1000 may optionally include user input devices such as keyboards, mice, game controllers, cameras, microphones, and / or touch screens.

[0090] Processor 1010 may include one or more physical devices configured to execute one or more instructions. For example, a processor can be configured to execute one or more instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical configurations. Is. These instructions can be implemented to perform a task, implement a data type, transform the state of one or more devices, or, in some cases, achieve the desired result.

[0091] Processors may include one or more processors configured to execute software instructions. As an addition or alternative, the processor can include one or more hardware or firmware logical machines that are configured to execute hardware or firmware instructions. The processor of the processor may be single-core or multi-core, and the programs executed on it can be configured for parallel or distributed processing. The processor can optionally include individual components that are remotely located and / or that can be configured for adjustment processing and are distributed across two or more devices. One or more aspects of the processor can be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration.

[0092] Memory 1020 includes one or more physical devices configured to hold data and / or instructions that can be executed by a processor to implement the methods and processes described herein. be able to. When these methods and processes are implemented, the state of memory 1020 can be transformed (eg to hold different data).

[0093] Memory 1020 can include removable media and / or embedded devices. The memory 1020 is, among other things, optical memory devices (eg CDs, DVDs, HD-DVDs, Blu-Ray disks, etc.), semiconductor memory devices (eg RAMs, EPROMs, EEPROMs, etc.), and / or magnetic memory devices (eg, RAMs, EPROMs, EEPROMs, etc.), and / or magnetic memory devices (eg, RAMs, EPROMs, EEPROMs, etc.). It can include hard disk drives, floppy disk drives, tape drives, MRAM, etc.). Memory 1020 is one of the features of volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and content addressable. It can include devices with one or more. In some embodiments, the processor 1010 and memory 1020 can be integrated into one or more common devices, such as application-specific integrated circuits or system-on-chip.

[0094] Memory 1020 executable data you to implement the methods and processes described herein can be used to store and / or forward and / or instructions, in the form of a removable computer-readable storage medium You can also take it. Memory 1020 can take the form of CD, DVD, HD-DVD, Blu-Ray disc, EEPROM, and / or floppy disc, among other things.

[0095] It will be appreciated that memory 1020 contains one or more physical devices that store information. The terms "module," "program," and "engine" can be used to describe aspects of a computing system 1000 implemented to perform one or more specific functions. In some cases, these modules, programs, or engines can be instantiated through processor 1010, which executes instructions held by memory 1020. It will be appreciated that different modules, programs, and / or engines can be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same modules, programs, and / or engines can be instantiated from different applications, services, code blocks, objects, libraries, routines, APIs, functions, and so on. The terms "module," "program," and "engine" are meant to include individual or groups of executable files, data files, libraries, drivers, scripts, database records, and so on.

[0096] In aspects, the computing system 1000 holds a memory 1020 that holds instructions that can be executed by processor 1010 to select the emotion type code associated with the output statement based on at least one fact or profile input. It is possible to support computing devices, including, and the emotion type code specifies one of a plurality of predetermined emotion types. The instruction can be further executed by processor 1010 to generate a voice corresponding to the output statement, and the generated voice has a predetermined emotion type specified by the emotion type code. In an exemplary embodiment, at least one fact or profile input is derived from the use of a mobile communication device that implements an interactive dialog system. It should be understood that these computing devices correspond to processes, machines, manufacturing, or compositions.

[0097] FIG. 11 shows an exemplary embodiment of apparatus 1100 according to the present disclosure. It should be noted that device 1100 is provided for illustration purposes only and does not limit the scope of this disclosure to any particular device illustrated.

[0098] In FIG. 11, classification block 1120 is configured to select the emotion type code 1120a associated with output statement 1110a based on at least one fact or profile input 1120b. The emotion type code 1120a specifies one of a plurality of predetermined emotion types. Text-speech block 1130 is configured to generate speech 1130a corresponding to output statement 1110a and a given emotion type specified by emotion type code 1120a. In an exemplary embodiment, at least one fact or profile input 1120b is derived from the use of a mobile communication device that implements an interactive dialog system.

It should be noted that the techniques of the present disclosure are not necessarily limited to embodiments incorporating mobile communication devices. In an alternative exemplary embodiment, the technique can also be incorporated into non-mobile devices, such as desktop computers, home gaming systems, and the like. Furthermore, mobile communication devices incorporating this technique are not necessarily limited to smartphones, but may include wearable devices such as computerized wristwatches and eyeglasses. These alternative exemplary embodiments are intended to be within the scope of this disclosure.

[00100] FIG. 12 illustrates an exemplary embodiment 1200 in which the techniques of the present disclosure are incorporated into a dialog system with emotional content added to text displayed in addition to or in addition to audible audio. Note that the blocks shown in FIG. 12 correspond to the blocks similarly labeled in FIG. 2, and some blocks shown in FIG. 2 have been omitted from FIG. 12 for ease of explanation.

[00101] In FIG. 12, the output 250a of the language generation block 250 is combined with the emotion type code 240b generated by the dialog engine 240 and input to the text-speech and / or display text block 1260. In the text-speech aspect, block 1260 produces speech with semantic content 240a and emotion type code 240b. In the display text aspect, block 1260 generates display text as an alternative or additionally with semantic content 240a and emotion type code 240b. Emotion type code 240b uses techniques such as adjusting the size or font of displayed text characters, providing emoticons (eg, smiles or other pictures) that correspond to emotion type code 240b. It will be understood that it can be used to add emotion to the displayed text. In an exemplary embodiment, block 1260 produces emotion-based animations or graphical modifications for one or more avatars representing DAs or users on the display as an alternative or in addition. For example, if emotion type code 240b corresponds to "sadness," an avatar representing a preselected DA would be generated with a preconfigured "sad" facial expression, or in some cases, for example, for example. It can be animated to express sadness through movements such as "crying movements". These alternative exemplary embodiments are intended to be within the scope of this disclosure.

[00102] In the specification and claims, if an element is described as being "connected" or "combined" to another element, is it directly connected or connectable to the other element? , Or it will be understood that intervening elements can exist. On the other hand, if an element is described as being "directly connected" or "directly connected" to another element, then there are no intervening elements. Furthermore, when an element is described as being "electrically coupled" to another element, this indicates that there is a low resistance path between these elements, and the element is simply "bonded" to another element. There may or may not be a low resistance path between these elements.

[00103] The functions described herein can be performed, at least in part, by one or more hardware and / or software logic components. For example, the example types of hardware logic components that can be used without limitation are field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific integrated circuits (ASSPs), and system-on-chip systems (SOCs). ), Combined programmable logic devices (CPLDs), etc.

[00104] Although various modifications and alternative configurations are possible in the present invention, some exemplary embodiments are shown in the drawings, which are described in detail above. However, it is understood that the invention is not intended to be limited to the specified form disclosed, but rather to cover all modifications, alternative configurations, and equivalents that fall within the spirit and scope of the invention. I want to be.