Dynamic and adaptive selection of vocabulary and acoustic models based on a call context for speech recognition

BACKGROUND

[0001] Aspects of the present invention relate to automated-speech processing. Other aspects of the present invention relate to adaptive automatic speech recognition.

[0002] In a society that is becoming increasingly service oriented, choices of products are often determined according to accompanying customer services offered with the products. Companies invest much capital in providing such services in order to attract customers. For example, a customer who purchases a computer from a manufacturer may be provided with a toll free telephone number so that the customer can call for any technical support or service questions. To facilitate offered customer services, a manufacturer may establish a call center equipped with call routing capabilities (e.g., route a call to an available agent), backend database systems managing relevant information such as customer profiles, and personal who are capable of handling different types of questions. There may be other possible system configurations besides call centers that are deployed to facilitate customer services.

[0003] Maintaining a call center is costly. To effectively compete in the market place, the cost associated with customer services has to be kept low. Various strategies of saving cost have been developed. One such strategy is to introduce automatic call routing capability so that there is no need to hire operators whose job is merely direct calls to appropriate agents. Such automatic call routing facilities automatically interpret the needs related to a calling customer (e.g., a customer may have a billing question) and then automatically route the customer's call to an agent who is specialized in the particular domain (e.g., an agent who is responsible for handling billing related questions).

[0004] There are mainly two categories of techniques to realize automatic call routing. One is to prompt a calling customer to enter coded choices. For example, a call center may prompt a customer to “enter 1 for placing an order; enter 2 for billing questions; enter 3 for promotions.”. With this implementation, a customer may enter the code corresponding to desired service using a device with keys such as a telephone. Since this type of solution requires a calling customer's effort, it may annoy some customers, especially when the number of choices is large enough so that a customer may have trouble to remember the code for each service choice after hearing the prompt.

[0005] Another category of techniques is to automate call routing via voice. In this case, a call center may prompt a calling customer to say what category of service is being requested. Since a customer, in this case, does not have to remember the code for each choice, it is often more convenient. To realize this type of solutions, a call center usually deploys an automatic speech recognition system that recognizes the spoken words from the speech of a calling customer. The recognized spoken words are then used to route the call. Due to the fact that a call center usually handles calls potentially from many different customers, it usually deploys an automatic speech recognition system that is speaker independent (as opposed to speaker dependent system). Independent speaker voice recognition, although more flexible than speaker dependent voice recognition, is less accurate.

[0006] To minimize the recognition inaccuracy of using a speaker independent system, a smaller than normal vocabulary may be used. With this technique, if a call center prompts a calling customer at a particular stage of the call to state one of the three given service choices, to recognize what the customer will say, a vocabulary of only three words may be selected for recognition. For example, if a customer is given choices of “information”, “operator”, and “billing”, to recognize what the customer will choose, a vocabulary consisting of only these three words is selected (as opposed to a generic vocabulary containing thousands of words) for recognition purposes. Using a smaller vocabulary helps to narrow the scope of the recognition and, hence, improve recognition accuracy. With this technique, at different stages of a call, different vocabularies are selected based on the requirements of an underlying application.

[0007] In many real systems, even with flexible selection of vocabularies at different stages of a call, recognition accuracy is often not good enough. This is especially true when the underlying vocabulary is not small enough. It is a difficult task to perform automated speech recognition that is speaker independent. Even with a relatively small vocabulary, different customers may state a same choice with very different speech features. For example, word “operator” may be pronounced quite differently by an American native speaker and a Japanese national.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present invention is further described in terms of exemplary embodiments, which will be described in detail with reference to the drawings. These embodiments are non-limiting exemplary embodiments, in which like reference numerals represent similar parts throughout the several views of the drawings, and wherein:

[0009]FIG. 1 depicts a framework in which a caller's speech is recognized using vocabulary and acoustic models adaptively selected based on a call context, according to embodiments of the present invention;

[0010]FIG. 2 depicts the internal high level functional block diagram of a speech recognition mechanism that is capable of adapting its vocabulary and acoustic models to a call context, according to embodiments of the present invention;

[0011]FIG. 3 illustrates exemplary relevant information of a call context that may affect the adaptive selection of a vocabulary and associated acoustic models, according to embodiments of the present invention;

[0012]FIG. 4 describes an exemplary relationship between vocabularies and acoustic models, according to an embodiment of the present invention;

[0013]FIG. 5 is an exemplary flowchart of a process, in which a caller's speech is recognized using vocabulary and acoustic models adaptively selected based on a call context, according to an embodiment of the present invention;

[0014]FIG. 6 is an exemplary flowchart of a process, in which a vocabulary adaptation mechanism dynamically selects an appropriate vocabulary according to a call context, according to an embodiment of the present invention;

[0015]FIG. 7 is an exemplary flowchart of a process, in which an acoustic model adaptation mechanism dynamically selects an appropriate acoustic models with respect to a vocabulary based on a call context, according to an embodiment of the present invention; and

[0016]FIG. 8 is an exemplary flowchart of a process, in which acoustic models used for speech recognition are adaptively adjusted based on speech recognition performances, according to an embodiment of the present invention.

DETAILED DESCRIPTION

[0017] The processing described below may be performed by a properly programmed general-purpose computer alone or in connection with a special purpose computer. Such processing may be performed by a single platform or by a distributed processing platform. In addition, such processing and functionality can be implemented in the form of special purpose hardware or in the form of software being run by a general-purpose computer. Any data handled in such processing or created as a result of such processing can be stored in any memory as is conventional in the art. By way of example, such data may be stored in a temporary memory, such as in the RAM of a given computer system or subsystem. In addition, or in the alternative, such data may be stored in longer-term storage devices, for example, magnetic disks, rewritable optical disks, and so on. For purposes of the disclosure herein, a computer-readable media may comprise any form of data storage mechanism, including such existing memory technologies as well as hardware or circuit representations of such structures and of such data.

[0018]FIG. 1 depicts a framework <b>100</b> in which a caller's speech is recognized using vocabulary and acoustic models adaptively chosen based on a call context, according to embodiments of the present invention. Framework <b>100</b> comprises a plurality of callers (caller <b>1</b><b>110</b><i>a</i>, caller <b>2</b><b>110</b><i>b</i>, . . . , caller n <b>110</b><i>c</i>), a voice response system <b>130</b>, and a speech recognition mechanism <b>140</b>. A caller communicates with the voice response system <b>130</b> via a network <b>120</b>. Upon receiving a call from a caller via the network <b>120</b>, the voice response system <b>140</b> identifies and forwards information relevant to the call to the speech recognition mechanism <b>140</b>. Based on such information, the speech recognition mechanism <b>140</b> adaptively selects one or more vocabularies and acoustic models, appropriate with respect to the call information and the caller, that are then used to recognize spoken words uttered by the caller during the call.

[0019] A caller may place a call via either a wired or a wireless device, which can be a telephone, a cellular phone, or any communication device, such as a personal data assistant (PDA) or a personal computer, that is capable of transmitting either speech (voice) data or features transformed from speech data. The network <b>120</b> represents a generic network, which may correspond to, but not limited to, a local area network (LAN), a wide area network (WAN), the Internet, a wireless network, or a proprietary network. The network <b>120</b> is capable of not only transmitting data but also relaying useful information related to the transmission, together with the transmitted data, to the voice response system <b>130</b>. For example, the network <b>120</b> may include switches, routers, and PBXes that are capable of extracting information related to the caller and attaching such information with the transmitted data.

[0020] The voice response system <b>130</b> represents a generic voice enabled system that responds to the speech from a caller by taking appropriate actions based on what a caller says during a call. For example, the voice response system <b>130</b> may correspond to an interactive voice response (IVR) system deployed at a call center. When a caller places a call to the call center, the IVR system may automatically direct the call to an appropriate agent at the call center based on what is said by the caller. For instance, if a caller calls for a billing question, the IVR system should direct the call to an agent who is trained to answer billing questions. If the caller asks for directory assistance, the IVR system should direct the call to an agent who is on duty to help callers to find desired phone numbers.

[0021] To appropriately act based on a caller's voice request, the voice response system <b>130</b> relies on the speech recognition mechanism <b>140</b> to recognize what is being said in the caller's speech. To improve recognition accuracy, the voice response system <b>130</b> may actively prompt a caller to answer certain questions. For example, upon intercepting a call, the voice response system <b>130</b> may ask the caller to state one of several given types of assistance that he/she is seeking (e.g., “place an order”, “directory assistance”, and “billing”).

[0022] The answer from the caller may be not only utilized to guide the voice response system <b>130</b> to react but also useful in terms of selecting an appropriate vocabulary for speech recognition purposes. For instance, knowing that a caller's request is for billing service, the voice response system <b>130</b> may further prompt the caller to provide an account number. Given this context, the speech recognition mechanism <b>140</b> may utilize a digits vocabulary (a vocabulary consisting of only digits, if an account number is known to consist of only digits) to recognize what will be said in the caller's response. The choice of a particular vocabulary may depend on an underlying application. For example, if an account is known to be composed of a combination of digits and letters, the speech recognition mechanism <b>140</b> may utilize both a digit vocabulary and a letter vocabulary (consisting of only letters) to form a combined vocabulary. A choice of a vocabulary may also be language dependent. For instance, if a caller speaks only Spanish, a Spanish vocabulary has to be used.

[0023] Using a particular vocabulary in speech recognition may narrow down the scope of what needs to be recognized, which improves both the efficiency and accuracy of the speech recognition mechanism <b>140</b>. Another dimension affecting the performance of a speech recognizer involves whether a caller's speech characteristics are known. For example, a French person may speak English with a French accent. In this case, even with an appropriately selected vocabulary, recognition accuracy on, for example, recognizing English digits spoken by a French person using an English digit vocabulary may result in poor recognition accuracy. In speech recognition, acoustic models capture the acoustic realization of phonemes in context corresponding to spoken words. A vocabulary realized in different languages may correspond to vastly different acoustic models. Similarly, a vocabulary in a particular language yet realized with different accents (e.g., speak digits in English with a French accent) may also yield distinct acoustic models.

[0024] The speech recognition mechanism <b>140</b> adaptively selects both vocabulary and associated acoustic models appropriate for recognition purposes. It comprises a vocabulary adaptation mechanism <b>150</b>, an acoustic model adaptation mechanism <b>170</b>, and an automatic speech recognizer <b>160</b>. The vocabulary adaptation mechanism <b>150</b> determines appropriate vocabularies based on information related to a particular call as well as the underlying application. For example, it may select an English digit vocabulary based on the fact that the caller is known to be an English speaker (e.g., based on either prior knowledge about the customer or automated recognition results) and the caller requests service related to billing questions. In this case, the English digit vocabulary is chosen for upcoming recognition of what will be said by the caller in answering the question, for instance, about his/her account number. Therefore, an appropriate vocabulary may selected based on both application needs (e.g., to answer a billing question, an account number is required) and the information about a particular caller (e.g., English speaking with a French accent).

[0025] The acoustic model adaptation mechanism <b>170</b> adaptively selects acoustic models based on a selected vocabulary (by the vocabulary adaptation mechanism <b>150</b>) and the information related to the underlying call. For example, assume an incoming call is for a billing related question and the caller is known (e.g., the customer profile associated with the caller ID may reveal so) to be an English speaker with a French accent. In this case, the vocabulary adaptation mechanism <b>150</b> selects an English digit vocabulary. Based on the vocabulary selection and the known context of the call (e.g., information about the caller), the acoustic model adaptation mechanism <b>170</b> may select the acoustic models that characterize speech properties of spoken English digits with French accent.

[0026] If a caller's speech characteristics are not known a priori (e.g., accent), the acoustic model adaptation mechanism <b>170</b> may determine, on-the-fly, the best acoustic models suitable for a particular caller. For example, the acoustic model adaptation mechanism <b>170</b> may dynamically, during the course of speech recognition, adapts to appropriate acoustic models based on the recognition performance of the automatic speech recognizer <b>160</b>. It may continuously monitor the speech recognition performance and accordingly adjust the acoustic models to be used. The updated information is then stored and associated with the call-information for future use.

[0027] When a vocabulary and corresponding acoustic models are both appropriately chosen, the automatic speech recognizer <b>160</b> performs speech recognition on incoming speech (from the caller) using the selected vocabulary and acoustic models. The recognition result is then sent to the voice response system <b>130</b> so that it can properly react to the caller's voice request. For example, if a caller's account number is recognized, the voice response system <b>130</b> may pull up the account information and prompt the caller to indicate what type of billing information the caller is requesting.

[0028] The reaction of the voice response system <b>130</b> may further trigger the speech recognition mechanism <b>140</b> to adapt to select different vocabulary and acoustic models for upcoming recognition. For example, to facilitate the automatic speech recognizer <b>160</b> to recognize the upcoming answer about the type of billing question (from the caller), the vocabulary adaptation mechanism <b>150</b> may select a vocabulary consisting of three words corresponding to three types of billing questions (e.g., “balance”, “credit”, and “last payment”). The acoustic model adaptation mechanism <b>170</b> may then accordingly select the acoustic models of the three-word vocabulary that correspond to, for example, French accent. Therefore, both the vocabulary adaptation mechanism <b>150</b> and the acoustic adaptation mechanism <b>170</b> adapt to the changing context of a call and dynamically select the vocabularies and acoustic models that are most appropriate given the call context.

[0029]FIG. 2 depicts the internal high level functional block diagram of the speech recognition mechanism <b>140</b>, according to embodiments of the present invention. The vocabulary adaptation mechanism <b>150</b> comprises an application controller <b>210</b>, a call context detection mechanism <b>240</b>, a vocabulary selection mechanism <b>220</b>, and a plurality of available vocabularies <b>230</b>. The vocabulary selection mechanism <b>220</b> chooses appropriate vocabularies based on a call context, detected by the call context detection mechanism <b>240</b>, and the application requirement, determined by the application controller <b>210</b>.

[0030] The application controller <b>210</b> may dictate the choice of type of vocabulary from the standpoint of what an application requires. For example, if an account number in a particular application consists of only digits (determined by the application controller <b>210</b>), a digit vocabulary is needed to recognize a spoken account number. If an account number in a different application consists of digits and letters, both a digit vocabulary and a letter vocabulary are required to recognize a spoken account number.

[0031] A call context associated with a call (may also associate with different time instances during the call) may dictate the choice of a vocabulary from the standpoint of linguistic requirement. For example, if a digit vocabulary is required by an application, there are choices in terms of which digit vocabulary of a particular language is required. This may be determined according to the call context. For example, if the caller is a French speaking person, a French digit vocabulary is needed.

[0032] The call context detection mechanism <b>240</b> receives information either forwarded from the voice response system <b>130</b> or retrieved from a customer profile associated with the caller or from the network. For example, the voice response system <b>130</b> may forward call related information such as a caller identification number (caller ID) or an area code representing an area from where the call is initiated. A caller ID may be used to retrieve a corresponding customer profile that may provide further information such as the language preference of the caller. Using such information, the call context detection mechanism <b>240</b> constructs the underlying call context, which may be relevant to the selection of appropriate vocabularies or acoustic models.

[0033]FIG. 3 illustrates exemplary relevant types of information within a call context that may affect the selection of a vocabulary and associated acoustic models, according to embodiments of the present invention. The information forwarded from the voice response system <b>130</b> may correspond to geographical information <b>310</b>, including, for example, an area code <b>320</b>, an exchange number <b>330</b>, or a caller ID <b>340</b>. Such information may be associated with a physical location where the call is initiated, which may be identified from the area code <b>320</b>, the exchange number <b>330</b>, or, probably most precisely, from the caller ID <b>340</b>. Geographical information may be initially gathered at a local carrier when the call is initiated and then routed (with the call) via the network <b>120</b> to the voice response system <b>130</b>.

[0034] The customer information retrieved from a customer profile may include, for example, one or more corresponding caller IDs <b>340</b>, an account number <b>360</b>, . . . , and language preference <b>370</b>. Using a received caller ID (from the voice response mechanism <b>130</b>), information contained in the associated customer profile may be retrieved. For example, given a caller ID, the language preference <b>370</b> may be retrieved from an associated customer profile. The language preference <b>370</b> may be indicated via different means. For instance, it may be entered when the underlying account is set up or it may be established during the course of dealing with the customer.

[0035] Different callers may use a same caller ID. A customer profile may record each of such individual potential callers and their language preferences (not shown in FIG. 3). Alternatively, a customer profile may distinguish female callers <b>380</b> from male callers <b>390</b> (e.g., in a household) and their corresponding language preferences due to the fact that female and male speakers usually present substantially different speech characteristics so that distinct acoustic models may be used to recognize their speech.

[0036] Geographical information related to a call can be used to obtain more information relevant to the selection of vocabularies and acoustic models. For example, a caller ID forwarded from the voice response system <b>130</b> can be used to retrieve a corresponding customer profile that provides further relevant information such as language preference. Using the retrieved language preference <b>370</b> (combined with a required type of vocabulary according to application needs), an appropriate vocabulary (e.g., English digit vocabulary) and acoustic models (e.g., acoustic models for English digits in French accent) may be determined.

[0037] When a caller ID is not available, direct access to a customer profile may not be possible. Consequently, a preferred language may not be known. In this case, the area code <b>320</b> or the exchange number <b>330</b> may be used to infer a language preference. For instance, if the area code <b>320</b> corresponds to a geographical area in Texas, it may be inferred that acoustic models corresponding to a Texan accent may be appropriate. As another example, if the exchange number <b>330</b> corresponds to a region (e.g., Chinatown in New York City), in which majority people speak English with a particular accent (i.e., Chinese living in Chinatown of New York City speak English with Chinese accent), a particular set of acoustic models corresponding to the inferred accent may be considered as appropriate.

[0038] As discussed earlier, selection of acoustic models depends on not only the speech characteristics of a caller but also the choice of vocabularies. FIG. 4 illustrates an exemplary relationship between vocabularies and acoustic models, according to an embodiment of the present invention. The vocabularies <b>230</b> includes a plurality of vocabularies (vocabulary <b>1</b><b>410</b>, vocabulary <b>2</b><b>420</b>, . . . , vocabulary n <b>430</b>). Each vocabulary may have realizations in different languages. For example, digit vocabulary <b>420</b> may include Spanish digit vocabulary <b>440</b>, English digit vocabulary <b>450</b>, . . . , and Japanese digit vocabulary <b>460</b>. In addition, with respect to each vocabulary in a given language, a plurality of acoustic models corresponding to different accents may be available. For instance, for the English digit vocabulary <b>450</b>, acoustic models corresponding to Spanish accent (<b>470</b>), English accent <b>480</b>, and French accent <b>490</b> may be selected consistent with the speech characteristics of a caller.

[0039] To select appropriate acoustic models, the acoustic model adaptation mechanism <b>170</b> may make the selection based on either the given information, such as the selection of a vocabulary (made by the vocabulary adaptation mechanism <b>150</b>) and the information contained in a call context, or information gathered on-the-fly, such as the speech characteristics detected from a caller's speech. Referring to FIG. 2, the acoustic model adaptation mechanism <b>170</b> comprises an acoustic model selection mechanism <b>260</b>, an adaptation mechanism <b>280</b>, and a collection of available acoustic models <b>270</b>. The acoustic selection mechanism <b>260</b> receives a call context from the call context detection mechanism <b>240</b>. Information contained in the call context may be used to determine a selection of appropriate acoustic models (see FIG. 3).

[0040] When the received call context does not provide needed information to make a selection, the adaptation mechanism <b>280</b> may detect, during the call, speech characteristics from the caller's speech (e.g., whether the caller is a female or a male speaker) that may be relevant to the selection. The detected speech characteristics may also be used to identify information in the associated customer profile that are useful to the selection. For example, if a female voice is detected, the acoustic model selection mechanism <b>260</b> may use that information to see whether there is a language preference associated with a female speaker in the customer profile (accessed using, for example, a caller ID in the call context). In this case, the selection is dynamically determined, on-the-fly, according to the speech characteristics of the caller.

[0041] A different exemplary alternative to achieve adaptation on-the-fly when there is no information available to assist the selection of acoustic models is to initially select a set of acoustic models according to some criteria and then refine the selection based on the on-line performance of speech recognition. For example, given an English digit vocabulary, the acoustic model selection mechanism <b>260</b> may initially choose acoustic models corresponding to English accent, Spanish accent, and French accent. All such initially selected acoustic models are then fed to the automatic speech recognizer <b>160</b> for speech recognition (e.g., parallel speech recognition against different accents). The performance measures (e.g., scores of the recognition) are produced during the recognition and sent to the adaptation mechanism <b>280</b> to evaluate the appropriateness of the initially selected acoustic models. The acoustic models resulting in poorer recognition performance may not be considered for further recognition in the context of this call. Such on-line adaptation may continue until the most appropriate acoustic models are identified.

[0042] The final on-line adaptation results (choices of acoustic models adjusted to achieve optimal speech recognition performance) may be used to update the underlying customer profile. For example, an underlying customer profile that originally has no indication of any language preference and accent may be updated with the on-line adaptation results, together with associated speech characteristics. For instance, a female speaker (speech characteristics) of a household (corresponding to a caller ID) has a French accent. Such updated information in the customer profile may be used in the future as a default selection with respect to a particular kind of speaker.

[0043]FIG. 5 is an exemplary flowchart of a process, in which a caller's speech is recognized using vocabulary and acoustic models that are adaptively selected based on a call context, according to an embodiment of the present invention. A call is first received at act <b>510</b>. Information relevant to the call is then forwarded, at act <b>520</b>, from the voice response system <b>130</b> to the speech recognition mechanism <b>140</b>. A call context is detected at act <b>530</b> and is used to select, at act <b>540</b>, an appropriate vocabulary. Based on the selected vocabulary and the detected call context, proper acoustic models are identified at act <b>550</b>. Using such selected vocabulary and the acoustic models, the automatic speech recognizer <b>160</b> performs speech recognition, at act <b>560</b>, on the caller's speech.

[0044]FIG. 6 is an exemplary flowchart of a process, in which the vocabulary adaptation mechanism <b>160</b> dynamically selects an appropriate vocabulary according to a call context, according to an embodiment of the present invention. Information relevant to a call is received at act <b>610</b>. Based on the call information, a customer profile may be retrieved at act <b>620</b>. From the call information and the customer profile, a call context is detected, at act <b>630</b>, and an appropriate vocabulary is selected accordingly at act <b>640</b>. The selected vocabulary, together with the call context, is then sent, at act <b>640</b>, to the acoustic model adaptation mechanism <b>170</b>.

[0045]FIG. 7 is an exemplary flowchart of a process, in which the acoustic model adaptation mechanism <b>170</b> dynamically selects appropriate acoustic models with respect to a vocabulary based on a call context, according to an embodiment of the present invention. A call context and a selected vocabulary is first received at act <b>710</b>. Using the call context, relevant customer information is analyzed at act <b>720</b>. When necessary, speech characteristics of the caller are determined at act <b>730</b>. Acoustic models that are appropriate with respect to the given vocabulary and the call context (including the speech characteristics detected on-the-fly) are selected at act <b>740</b>.

[0046]FIG. 8 is an exemplary flowchart of a process, in which vocabularies and acoustic models used for speech recognition are adaptively adjusted on-the-fly based on speech recognition performances, according to an embodiment of the present invention. Adaptively selected vocabulary and acoustic models are first retrieved, at act <b>810</b>, and then used to recognize, at act <b>820</b>, the speech from a caller. Performance measures are generated during the course of recognition and are used to assess, at act <b>830</b>, the recognition performance. If the assessment indicates that a high confidence is achieved during the recognition, determined at act <b>840</b>, current vocabulary and acoustic models are continuously used for on-going speech. Otherwise, vocabulary and acoustic models that may lead to improved recognition performance are re-selected at act <b>850</b>. Information related to the re-selection (e.g., the newly selected vocabulary and acoustic models) is used to update the underlying customer profile. This model adaptation process may continue until the end of the call.

[0047] While the invention has been described with reference to the certain illustrated embodiments, the words that have been used herein are words of description, rather than words of limitation. Changes may be made, within the purview of the appended claims, without departing from the scope and spirit of the invention in its aspects. Although the invention has been described herein with reference to particular structures, acts, and materials, the invention is not to be limited to the particulars disclosed, but rather can be embodied in a wide variety of forms, some of which may be quite different from those of the disclosed embodiments and extends to all equivalent structures, acts, and, materials, such as are within the scope of the appended claims.