Training a device specific acoustic model

0001This application is a continuation of pending U.S. patent application Ser. No. 17/237,003 filed Apr. 21, 2021, which is a continuation of U.S. patent application Ser. No. 15/996,393 filed Jun. 1, 2018 and granted as U.S. Pat. No. 11,011,162.

BACKGROUND

0002The technology disclosed relates to automatic speech recognition (ASR). In particular, the technology disclosed relates to creation, identification, selection and implementation of custom acoustic models in intelligent speech recognition systems.

0003Speech recognition systems have become more prevalent in today's society. More and more everyday devices, such as appliances, vehicles, mobile devices, etc., are being equipped with speech recognition capabilities. The problem is that these everyday devices are not able to provide meaningful responses based on received speech audio from the user. One of the root causes of this problem is that the everyday devices and/or local or remote services connected thereto are not able to accurately convert the received speech audio to appropriate transcriptions. Typically, the received speech audio is converted to phonemes using an acoustic model. However, these everyday devices and/or local or remote services connected thereto are using acoustic models that are not tailored to their (i) environment, (ii) expected use conditions and/or (iii) expected use case results. Therefore, these everyday devices that are enabled with speech recognition are not able to accurately recognize the received speech audio into a reliable transcription, from which helpful results can be communicated back to the user, and according to which the user or the user's device can appropriately respond.

0004An example of this problem is provided below. Suppose a coffee shop decides to upgrade their espresso machine to a brand new high-tech machine that is voice activated (i.e., that incorporates a speech recognition system). The acoustic model that is implemented by the espresso machine is generic and it has not been customized to the environment to which it is being used. This particular coffee shop has a minimalistic industrial decor, causing sounds to echo and reverberate much more than what is typical in other environments. Further, the espresso machine is located, such that the area for taking a customer's order is on one side of the espresso machine, there is a sink in front of the espresso machine and the bean grinder is on the other side of the espresso machine. Needless to say, there is a lot of background noise that is received by the speech recognition system of the espresso machine. Additionally, the temperature of the coffee shop tends to be on the warm side and the components of the espresso machine become extremely hot due to constant use. These temperatures cause the characteristics of the one or more microphones and the related electrical components to behave outside of what is considered normal. All of these factors coupled with the generic acoustic model cause the espresso machine to have terribly inaccurate transcriptions and responses to the point that it is impractical to use the speech recognition features of the espresso machine.

0005These problems provide an opportunity to develop a technology that is capable of implementing acoustic models can be tailored to specific devices and can be tailored based on various environmental and operating conditions, such as those mentioned above. The technology disclosed solves these problems and is able to provide a more accurate speech recognition system and meaningful results.

SUMMARY

0006Generally, the technology disclosed relates to automatic speech recognition (ASR) for analyzing utterances. In particular, the technology disclosed relates to identifying, selecting and implementing acoustic models in a speech recognition system, so that meaningful results can be provided to the end user. The technology disclosed is able to determine which acoustic model should be implemented when speech audio is received along with other data (i.e., metadata) that indicates a type of device and/or one or more conditions of the device (e.g., an end user device, such as an espresso machine or a washing machine or a vehicle). The technology disclosed also provides a speech recognition system that has an interface that allows a product manufacturer or developer to select which types of acoustic models should be implemented or should most likely be implemented to be able to provide meaningful results.

0007Particular aspects of the technology disclosed are described in the claims, specification and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

0008<figref idref="DRAWINGS">FIG. <b>1</b></figref> is a block diagram that illustrates a general framework implemented by a speech recognition and natural language understanding system.

0009<figref idref="DRAWINGS">FIG. <b>2</b></figref> is a block diagram that illustrates an example embodiment of the interaction between acoustic models and language models for natural language understanding (NLU).

0010<figref idref="DRAWINGS">FIG. <b>3</b></figref> illustrates a diagram of an example environment in which various acoustic models can be implemented.

0011<figref idref="DRAWINGS">FIG. <b>4</b></figref> illustrates an example implementation of the environment of <figref idref="DRAWINGS">FIG. <b>3</b></figref>, in which various acoustic models can be implemented.

0012<figref idref="DRAWINGS">FIG. <b>5</b></figref> illustrates an example implementation of the environment of <figref idref="DRAWINGS">FIG. <b>3</b></figref>, in which various acoustic models can be implemented.

0013<figref idref="DRAWINGS">FIG. <b>6</b></figref> illustrates an example implementation of the environment of <figref idref="DRAWINGS">FIG. <b>3</b></figref>, in which various acoustic models can be implemented.

0014<figref idref="DRAWINGS">FIG. <b>7</b></figref> illustrates an example implementation of the environment of <figref idref="DRAWINGS">FIG. <b>3</b></figref>, in which various acoustic models can be implemented.

0015<figref idref="DRAWINGS">FIG. <b>8</b></figref> illustrates an example implementation of the environment of <figref idref="DRAWINGS">FIG. <b>3</b></figref>, in which customized acoustic models can be trained.

0016<figref idref="DRAWINGS">FIG. <b>9</b></figref> is a block diagram of an example computer system that can implement various components of the environment of <figref idref="DRAWINGS">FIG. <b>3</b></figref>.

DETAILED DESCRIPTION

0017The following detailed description is made with reference to the figures. Example implementations are described to illustrate the technology disclosed, not to limit its scope, which is defined by the claims. Those of ordinary skill in the art will recognize a variety of equivalent variations on the description that follows.

0018<figref idref="DRAWINGS">FIG. <b>1</b></figref> is a block diagram that illustrates a general framework implemented by a speech recognition and natural language understanding system (e.g., a natural language understanding platform/server). In state of the art implementations of speech recognition and natural language understanding systems, speech recognition is typically applied first to produce a sequence of words or a set of word sequence hypotheses. Sometimes, this type of system is referred to as a combination of acoustic recognition and language, or linguistic, recognition. Speech recognition output is sent to the NLU system to extract the meaning.

0019Referring to <figref idref="DRAWINGS">FIG. <b>1</b></figref>, the general framework <b>100</b> includes receiving speech audio that includes natural language utterances. An example of speech audio would be a recording of a person speaking the phrase “ice cream cone.” The speech audio can be received from any source (e.g., a mobile phone, a washing machine, a vehicle, etc.).

0020The speech audio is then analyzed by an acoustic front end <b>102</b>, using an acoustic model <b>103</b> to extract phonemes from the speech audio. This is often times referred to as acoustic recognition. An example of this operation would be generating the phonemes “AY S<sil>K R IY M<sil>K OW N” (represented by the Carnegie Mellon University (CMU) Phoneme Set) based on the received speech audio.

0021Next, the phonemes generated by the acoustic front end <b>102</b> are received by a language model <b>104</b>, which can be implemented to transcribe the detected phonemes (e.g., “AY S<sil>K R IY M<sil>K OW N”) into an actual sentence, such as “ice cream cone.” Transcribing the phonemes into a transcription is not a simple process and various factors come into play.

0022Once one or more transcription is determined, natural language understanding <b>106</b> is performed by an NLU system to extract meaning from the transcription “ice cream cone.” Oftentimes meaning is associated with the transcription based on the domain or vertical or based on surrounding context. For example, if the vertical is related to searching for food, or more specifically, searching for places that serve food, then the meaning applied to the transcription “ice cream cone” could implement a search for local (nearest) places that serve ice cream for immediate consumption. In contrast, if the vertical is associated with places that sell food for later consumption (e.g., a grocery store) then the meaning would result in a search for grocery stores or markets that sell ice cream cones.

0023The technology disclosed is focused on creating, selecting and implementing the best acoustic model to create phonemes from received speech. Much effort has previously been put into determining the best transcriptions and the best meanings, but not much effort has previously been put into determining the best or most appropriate acoustic model to implement at any given time.

0024<figref idref="DRAWINGS">FIG. <b>2</b></figref> is a block diagram that illustrates an example embodiment of the interaction between acoustic models and language models for natural language understanding. <figref idref="DRAWINGS">FIG. <b>2</b></figref> is simply a more detailed version of <figref idref="DRAWINGS">FIG. <b>1</b></figref> and illustrates an example process or flow from receiving speech audio from a user to determining a meaningful response to the user. While the technology disclosed focuses on the creation, selection and implementation of custom acoustic models, <figref idref="DRAWINGS">FIG. <b>2</b></figref> provides a nice example framework of the various steps and processing required to perform natural language understanding (NLU). All of the operations described with reference to <figref idref="DRAWINGS">FIG. <b>2</b></figref> are not necessary to implement the technology disclosed. The technology disclosed is capable of performing the creation, selection and implementation of custom acoustic models in many different ways, some of which coincide with the description of <figref idref="DRAWINGS">FIG. <b>2</b></figref> and some of which do not coincide with the description of <figref idref="DRAWINGS">FIG. <b>2</b></figref>.

0025Example embodiment <b>200</b>, includes an automatic speech recognition (ASR) system <b>203</b>, which includes an acoustic front end <b>204</b>, acoustic models <b>206</b>, a word sequence recognizer <b>208</b> and language models <b>210</b>. The example embodiment <b>200</b> also includes natural language understanding <b>212</b>. When a person speaks, speech audio <b>202</b> including natural language utterances are input into the acoustic front end <b>204</b>. The acoustic front end <b>204</b> processes acoustic features of the speech audio <b>202</b> using one or more acoustic models <b>206</b>. The ASR system <b>203</b> can also receive selection criteria (e.g., metadata) that is used to assist, for example, the acoustic front end <b>204</b> in making a selection of an acoustic model from the one or more acoustic models <b>206</b>. Further, the acoustic front end generates one or more phoneme sequences.

0026The word sequence recognizer <b>208</b> receives the one or more phoneme sequences from the acoustic front end <b>204</b> and implements one or more language models from the language models <b>210</b> to transcribe the phonemes. The ASR system <b>203</b> can implement various types of scoring systems to determine the best phonemes and/or transcriptions. For example, each possible phoneme sequence can be associated with a score indicating the probability that the sequence is the most likely intended sequence. For example, a speaker may say, “I read a good book.” Table 1 (below) shows example alternate phoneme sequences with scores that might be generated by the acoustic front end <b>204</b>. The phoneme sequences can be represented using a phoneme set such as Carnegie Mellon University (CMU) Phoneme Set, or any other phonetic representation.

0027<tables id="TABLE-US-00001" num="00001"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="1"><colspec colname="1" colwidth="217pt" align="center" /><thead><row><entry namest="1" nameend="1" rowsep="1">TABLE 1</entry></row></thead><tbody valign="top"><row><entry namest="1" nameend="1" align="center" rowsep="1" /></row><row><entry>Phoneme sequences generated by Speech Engine Front End</entry></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="3"><colspec colname="offset" colwidth="14pt" align="left" /><colspec colname="1" colwidth="133pt" align="left" /><colspec colname="2" colwidth="70pt" align="center" /><tbody valign="top"><row><entry /><entry>Phoneme Sequence</entry><entry>Score</entry></row><row><entry /><entry namest="offset" nameend="2" align="center" rowsep="1" /></row><row><entry /><entry>AY + R EH D + AH + G <b>EH </b>D + B UH K</entry><entry>0.000073034</entry></row><row><entry /><entry><b>AY </b>+ R EH D + AH + G <b>UH </b>D + B UH K</entry><entry>0.000083907</entry></row><row><entry /><entry><b>AH </b>+ R EH D + AH + G UH D + B UH K</entry><entry>0.000088087</entry></row><row><entry /><entry namest="offset" nameend="2" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>

0028As seen in Table 1, certain phonemes are bolded to call attention to the differences between these three very similar phonetic sequences. Thought it is an incorrect transcription, the third alternative phoneme sequence has the highest acoustic score. This type of error occurs in cases of noise, accents, or imperfections of various speech recognition systems. This is an example of why it is beneficial to implement a proper acoustic model.

0029Next, the natural language understanding (NLU) <b>212</b> is performed on the transcription to eventually come up with a meaningful representation (or a data structure) of the speech audio <b>202</b>.

0030The technology disclosed relates to selecting customer acoustic models (e.g., acoustic models <b>206</b>) to be implemented by, for example, the acoustic front end <b>204</b>). The selected custom acoustic model can be implemented in an environment such as example embodiment <b>200</b>, or in any other framework that would be apparent to a person of skill in the art.

0031<figref idref="DRAWINGS">FIG. <b>3</b></figref> illustrates a diagram of an example environment <b>300</b> in which various acoustic models can be implemented. The environment <b>300</b> includes at least one user device <b>302</b>, <b>304</b>, <b>306</b>. The user device <b>302</b> could be a mobile phone, tablet, workstation, desktop computer, laptop or any other type of user device running an application <b>303</b>. The user device <b>304</b> could be an automobile and the user device <b>306</b> could be a washing machine, each of which is running an application <b>303</b>. Various example implementations of these user devices are discussed in more detail below. The user devices <b>302</b>, <b>304</b>, <b>306</b> are connected to one or more communication networks <b>308</b> that allow for communication between various components of the environment <b>300</b>. In one implementation, the communication networks <b>308</b> include the Internet. The communication networks <b>308</b> also can utilize dedicated or private communication links that are not necessarily part of the Internet. In one implementation the communication networks <b>308</b> uses standard communication technologies, protocols, and/or inter-process communication technologies.

0032The environment <b>300</b> also includes applications <b>310</b> that can be preinstalled on the user devices <b>302</b>, <b>304</b>, <b>306</b> or updated/installed on the user devices <b>302</b>, <b>304</b>, <b>306</b> over the communications networks <b>308</b>. The environment <b>300</b> also includes a speech recognition platform/server <b>311</b>, which is part of the speech recognition system. The speech recognition platform/server <b>311</b> can be a single computing device (e.g., a server), a cloud computing device, or it can be any combination of computing device, cloud computing devices, etc., that are capable of communicating with each other to perform the various tasks required to perform meaningful speech recognition. The speech recognition platform/server <b>311</b> includes a phrase interpreter <b>312</b> that performs, for example, the functions of the general framework <b>100</b> discussed above with reference to <figref idref="DRAWINGS">FIG. <b>1</b></figref>. Since the speech recognition platform/server <b>311</b> can be spread over multiple servers and/or cloud computing device, the operations of the phrase interpreter <b>312</b> can also be spread over multiple servers and/or cloud computing device. The applications <b>310</b> can be used by and/or in conjunction with the phrase interpreter <b>312</b> to understand spoken input. The various components of the environment <b>300</b> can communicate (exchange data) with each other using customized Application Program Interfaces (API) for security and efficiency.

0033The user devices <b>302</b>, <b>304</b>, <b>306</b>, and the phrase interpreter <b>312</b> each include memory for storage of data and software applications, a processor for accessing data in executing applications, and components that facilitate communication over the network <b>308</b>. The user devices <b>302</b>, <b>304</b>, <b>306</b> execute applications <b>303</b>, such as web browsers (e.g., a web browser application <b>303</b> executing on the user device <b>302</b>), to allow developers to prepare and submit applications <b>310</b> and allow users to submit speech audio including natural language utterances to be interpreted by the phrase interpreter <b>312</b>. The phrase interpreter <b>312</b> essentially performs the functions of the general framework <b>100</b> discussed above with reference to <figref idref="DRAWINGS">FIG. <b>1</b></figref>.

0034The environment <b>300</b> also includes an acoustic model selection interface <b>320</b> that allows developers and/or users to select one or more appropriate acoustic models from a repository of acoustic models <b>322</b>. The repository of acoustic models <b>322</b> are not necessarily stored at the same location and can be a collection of acoustic models from various sources and the acoustic models can be customized by the developer and/or end user, depending upon the particular implementation. The acoustic model selection interface <b>320</b> can be any type of interface that allows acoustic models to be chosen for implementation by the phrase interpreter <b>312</b> such as a browser or command line interface. Further, multiple acoustic models can be selected for implementation by the phrase interpreter <b>312</b> and the phrase interpreter <b>312</b> can intelligently select the best acoustic model to be implemented at any given point. Further details about the selection and implementation of the acoustic models are provided below with reference to other figures.

0035The phrase interpreter <b>312</b> implements one or more acoustic models <b>314</b>, language models <b>316</b> and natural language domain <b>318</b>. The acoustic models <b>314</b>, as discussed above with reference to <figref idref="DRAWINGS">FIG. <b>1</b></figref>, can output phonemes and/or sound tokens. The language models <b>316</b> of the phrase interpreter <b>312</b> to create a transcription of the received speech audio. The language models <b>316</b> can be single stage or multiple stage models that add an application of separate linguistic analysis. For example, the acoustic models <b>314</b> can process received speech audio to produce phonemes. These phonemes can be passed to the language models <b>316</b> that consider and scores sequences of phonemes. The language models <b>316</b> can sometimes use diphone or triphone analysis to recognize likely sequences of phonemes. The language models <b>316</b> can use statistical language models to recognize statistically likely sequences of words.

0036The natural language domain <b>318</b> implemented by the phrase interpreter <b>312</b> is what adds real meaning to the transcription of the received speech audio. As mentioned above with reference to <figref idref="DRAWINGS">FIG. <b>1</b></figref>, the natural language domain <b>318</b> is able to put context and meaning to the transcription. As a brief example that is further expounded upon later in this document, let's say that the washing machine <b>306</b> transmits speech audio that says “please wash soccer jersey from today's game.” Once the phrase interpreter <b>312</b> correctly generates the phonemes and transcription, the natural language domain <b>318</b> is able to apply meaning to the transcribed phrase by providing the washing machine with instructions to use cold water with extra-long soak and rinse cycles. Alternately, the natural language domain <b>318</b> can just send instructions to the washing machine <b>306</b> “wash soccer jersey” and then the washing machine <b>306</b> can intelligently decide which wash settings to implement.

0037The phrase interpreter <b>312</b> is implemented using at least one hardware component and can also include firmware, or software running on hardware. Software that is combined with hardware to carry out the actions of a phrase interpreter <b>312</b> can be stored on computer readable media such as rotating or non-rotating memory. The non-rotating memory can be volatile or non-volatile. In this application, computer readable media does not include a transitory electromagnetic signal that is not stored in a memory; computer readable media stores program instructions for execution. The phrase interpreter <b>312</b>, as well as the acoustic model selection interface <b>320</b>, the applications <b>310</b> and the repository of acoustic models <b>322</b> can be wholly or partially hosted and/or executed in the cloud or by other entities connected through the communications network <b>308</b>.

0038<figref idref="DRAWINGS">FIG. <b>4</b></figref> illustrates an example implementation of the environment of <figref idref="DRAWINGS">FIG. <b>3</b></figref>, in which various acoustic models can be implemented. In particular, <figref idref="DRAWINGS">FIG. <b>4</b></figref> illustrates the environment <b>300</b> of <figref idref="DRAWINGS">FIG. <b>3</b></figref> and additionally illustrates an example implementation in which a washing machine <b>306</b> is used as a client device for speech recognition. While <figref idref="DRAWINGS">FIG. <b>4</b></figref> provides an example implementation of the washing machine <b>306</b> as the user device, any other user device can replace the washing machine <b>306</b>. In other words, this example implementation is not limited to just a washing machine <b>306</b> as the user device.

0039Specifically, <figref idref="DRAWINGS">FIG. <b>4</b></figref> illustrates that a user <b>402</b> communicates directly to the washing machine <b>306</b> using a microphone/speaker interface (not illustrated) and that the user <b>402</b> can communicate to the washing machine <b>306</b> using another electronic device, such as a mobile phone <b>404</b>. As an example, the user <b>402</b> may communicate speech audio to the washing machine <b>306</b> as “please wash soccer jersey from today's game.” Again, this speech audio can be directly communicated to the washing machine <b>306</b> or it can be communicated to the washing machine <b>306</b> via the mobile phone <b>404</b>. The washing machine <b>306</b> then, via the communication networks <b>308</b>, provides the recorded speech audio to the speech recognition platform/server <b>311</b> that performs speech recognition and natural learning understanding. Alternatively, the mobile phone <b>404</b> can also communicate the recorded speech audio to the speech recognition platform/server <b>311</b> via the communication networks <b>308</b>. The speech recognition platform/server <b>311</b> then implements the phrase interpreter <b>312</b>. Along with the speech audio, the washing machine <b>306</b> also transmits metadata. Note that the metadata can be transmitted from the washing machine <b>306</b> to the communication networks <b>308</b> and/or from the washing machine <b>306</b> to the mobile phone <b>404</b> and then from the mobile phone <b>404</b> to the communication networks <b>308</b>. Other combinations of communications between the washing machine <b>306</b>, the mobile phone <b>404</b> and the communications networks <b>308</b>, for the purpose of getting the speech audio and the metadata communicated to the speech recognition platform/server <b>311</b> will be apparent to a person skilled in the art.

0040The phrase interpreter <b>312</b> then uses the metadata for selection of an appropriate acoustic model <b>314</b>. The metadata can include any meaningful information that would assist in the selection of the appropriate acoustic model <b>314</b>. For example, the metadata can include either or both of a device type and a specific device condition. Specifically, the metadata can include (i) a unique identification of the washing machine <b>306</b> (e.g., device type, model number, serial number, etc.), (ii) usage conditions, such as temperature and/or environmental conditions in the laundry room, (iii) other environmental conditions, such as outdoor weather, (iv) information that could affect the surrounding acoustics, (v) information related to other types of noises that could interfere with the accuracy of the acoustic model, (vi) current operating conditions of the washing machine <b>306</b> as well as operating conditions of other devices located nearby, such as a dryer or laundry tub, and (vii) information regarding one or more hardware and software components of the washing machine <b>306</b> or other components involved in the receiving of the speech audio and/or for providing audio feedback to the user. Generally, the ability of a system to optimize the choosing or adapting of an acoustic model is improved by having more metadata information with utterances.

0041Once the phrase interpreter <b>312</b> receives the speech audio and the metadata, the phrase interpreter <b>312</b> (or some other component of the overall system or platform that performs the speech recognition) can decide which acoustic model would be the best for extracting phonemes. Some embodiments use only the model number or device type of the washing machine <b>306</b>, and the phrase interpreter <b>312</b> is able to select an acoustic model that has been created or tuned for that specific device type. The same goes for the other possibilities of metadata, as described above. Furthermore, if the user of the washing machine <b>306</b> can be identified, then an acoustic model that is tuned for that specific user's voice can be implemented. Note that different features of different acoustic models can be combined. For example, features that tune an acoustic model to a particular user's voice can be combined with features of an acoustic model that is tuned for dryer noise. This is a mix-and-match type acoustic model that is intelligently created and implemented in dependence upon many pieces of information included in the metadata and various different acoustic models that are at the disposal of the speech recognition platform/server <b>311</b>.

0042A developer or subscriber to a speech recognition service has the ability to pre-select which acoustic models are available to implement for certain devices. The phrase interpreter <b>312</b> is able to store those pre-selected acoustic models and/or is able to obtain those pre-selected acoustic models from the repository of acoustic models <b>322</b>. The phrase interpreter <b>312</b> can also obtain other conditions <b>406</b> that might be helpful in the selection of the best acoustic model <b>314</b>. Once the phrase interpreter <b>312</b> receives the necessary information it is able to select the best acoustic model <b>314</b> for the job and then proceed to use the selected acoustic model <b>314</b> to generate the phonemes, then implement the language model <b>316</b> to transcribe the phonemes and then apply natural language domain <b>318</b> to be able to provide meaningful instructions.

0043The metadata can also include information that would assist in the natural language domain <b>318</b>. For example, if the metadata included information indicating a certain type of weather (e.g., raining), then the phrase interpreter <b>312</b> could intelligently determine that the soccer jersey was most likely to be very muddy due to the weather conditions. The meaningful instructions provided by the phrase interpreter <b>312</b> could be instructions for the washing machine <b>306</b> to dispense a certain type of soap, to run extra cycles of washing and rinsing, to use certain temperatures of water, etc. Alternatively, the meaningful instructions provided by the phrase interpreter <b>312</b> could simply be “dirty sports uniform” and then the washing machine <b>306</b> would have to intelligently determine which wash cycles and options to implement. Either way, the most efficient and accurate way to be able to provide the meaningful instructions is to be able to select the appropriate acoustic model. If the acoustic model is not “tuned” or “trained” for the conditions in which the speech audio is received, then the likelihood of the language model <b>316</b> and the natural language domain <b>318</b> being successful is greatly reduced.

0044The meaningful information returned to the washing machine <b>306</b> and/or the user <b>402</b> can be a request for further clarification, etc. The user <b>402</b> and/or the washing machine <b>306</b> can then provide further information back to the phrase interpreter <b>312</b>. Additionally, the other conditions <b>406</b> could be information that could be provided in the metadata by the washing machine <b>306</b>, but could be learned from other sources (e.g., weather, calendar information of the user, etc.). For example, if the phrase interpreter <b>312</b> is able to know the approximate date/time of the soccer game and the location, it could be possible to more accurately know the weather, how long the stains have had to settle in and what type of ground (e.g., artificial grass, red dirt, etc.) might be on the soccer field.

0045<figref idref="DRAWINGS">FIG. <b>5</b></figref> illustrates an example implementation of the environment of <figref idref="DRAWINGS">FIG. <b>3</b></figref>, in which various acoustic models can be implemented. In particular, <figref idref="DRAWINGS">FIG. <b>5</b></figref> illustrates the environment <b>300</b> of <figref idref="DRAWINGS">FIG. <b>3</b></figref> and further illustrates an example implementation in which a vehicle <b>304</b> is used as a client device for natural language recognition. While <figref idref="DRAWINGS">FIG. <b>5</b></figref> provides an example implementation of the vehicle <b>304</b> as the user device, any other user device can replace the vehicle <b>304</b>. In other words, this example implementation is not limited to just a vehicle <b>304</b> as the user device.

0046Specifically, <figref idref="DRAWINGS">FIG. <b>5</b></figref> illustrates that a vehicle <b>304</b> is the user device and that the vehicle <b>304</b> transmits the speech audio and the metadata to the speech recognition platform/server <b>311</b> that performs the natural language understanding. Similar to the discussion above regarding <figref idref="DRAWINGS">FIG. <b>4</b></figref>, the speech audio and the metadata can be transmitted/received using a combination of communication devices such as the vehicle <b>304</b> itself as well as one or more mobile devices. This example with the vehicle <b>304</b> follows the same process as described above with respect to the washing machine <b>306</b>, except that the conditions and meaningful instructions will be quite different. The vehicle <b>304</b> might have multiple microphones and speakers and different configurations of drivers and passengers, making it beneficial to identify the locations of the passenger or passengers that are speaking. Furthermore, the vehicle <b>304</b> is likely to encounter many different types of noise environments depending on its location and type of operation. There might be a traffic jam in downtown New York City, there could be a hail storm, there could be a crying infant, the vehicle could have its windows down, the radio could be on, it could be running at high rotations per minute (RPMs) or low RPMs, or the vehicle <b>304</b> could be in a tunnel. The vehicle <b>304</b> can be constantly monitoring all of these situations and storing the appropriate metadata that can be used when the user invokes speech recognition. Additionally, metadata can be gathered from a mobile device of the user, which can then be stored and/or transmitted by the vehicle <b>304</b>.

0047As an example, a passenger in the back seat of the vehicle <b>304</b> may say “call Grandma Moses on her cell phone.” Aside from the metadata discussed above regarding the washing machine <b>306</b>, the metadata can include information such as which microphone and/or microphones were used to record the passenger's voice, whether or not the windows of the vehicle <b>304</b> were open or closed, whether the heating ventilation and air conditioning (HVAC) of the vehicle <b>304</b> was running at full blast, as well as any other information that could be collected that could affect the selection of the best acoustic model.

0048<figref idref="DRAWINGS">FIG. <b>6</b></figref> illustrates an example implementation of the environment of <figref idref="DRAWINGS">FIG. <b>3</b></figref>, in which various acoustic models can be implemented. In particular, <figref idref="DRAWINGS">FIG. <b>6</b></figref> illustrates the environment <b>300</b> of <figref idref="DRAWINGS">FIG. <b>3</b></figref> and further illustrates an example implementation in which a vehicle <b>304</b> is used as a client device for natural language recognition. While <figref idref="DRAWINGS">FIG. <b>6</b></figref> provides an example implementation of the vehicle <b>304</b> as the user device, any other user device can replace the vehicle <b>304</b>. In other words, this example implementation is not limited to just a vehicle <b>304</b> as the user device.

0049<figref idref="DRAWINGS">FIG. <b>6</b></figref> is very similar to <figref idref="DRAWINGS">FIG. <b>5</b></figref>, except that different embodiments are illustrated.

0050One of the embodiments involves the vehicle <b>304</b> selecting an appropriate acoustic model from a set of locally stored acoustic models <b>322</b> and then coming up with meaningful instructions. This embodiment offloads the selection of the acoustic model from the speech recognition platform/server <b>311</b> and allows the vehicle <b>304</b> to select the best acoustic model. A developer and/or user can preconfigure the vehicle <b>304</b> with acoustic models that are customized for that vehicle <b>304</b> and then the vehicle can choose the acoustic model itself. As illustrated in <figref idref="DRAWINGS">FIG. <b>6</b></figref>, the vehicle <b>304</b> can store acoustic models <b>406</b> for implementation. The vehicle <b>304</b> can implement the acoustic model itself or it can transmit the acoustic model over the communication network <b>308</b>.

0051In another embodiment, the vehicle <b>304</b> can (i) select and implement the acoustic model itself to obtain meaningful instructions and (ii) transmit the speech audio, the metadata and meaningful instructions to the speech recognition platform/server <b>311</b>. Then the phrase interpreter <b>312</b> can consider the speech audio and metadata to make its own selection of an acoustic model <b>314</b> to develop meaningful results. The phrase interpreter <b>312</b> then can compare its own meaningful instructions with the meaningful instructions received from the vehicle <b>304</b> to determine the best meaningful instructions and then transmit the best meaningful instructions to the vehicle <b>304</b>. This implementation would be beneficial in a situation where perhaps the speech recognition platform/server <b>311</b> has been updated with more accurate acoustic models or visa-versa.

0052The discussions regarding <figref idref="DRAWINGS">FIGS. <b>4</b>-<b>6</b></figref> are merely examples, as the user devices implementing speech recognition can greatly vary and the pool of which is ever increasing.

0053<figref idref="DRAWINGS">FIG. <b>7</b></figref> illustrates an example implementation of the environment of <figref idref="DRAWINGS">FIG. <b>3</b></figref>, in which various acoustic models can be implemented. In particular, <figref idref="DRAWINGS">FIG. <b>7</b></figref> illustrates the environment <b>300</b> of <figref idref="DRAWINGS">FIG. <b>3</b></figref> and further illustrates an interface <b>702</b> that can be used by a developer to select custom acoustic models for implementation and/or training, etc.

0054Specifically, <figref idref="DRAWINGS">FIG. <b>7</b></figref> illustrates that the acoustic model selection interface <b>320</b> provides an interface <b>702</b> to a developer. The interface <b>702</b> can be a graphical user interface provided through a customized application or program, or it can be viewed through a web browser. A person of skill in the art will recognize the various types of interfaces encompassed by the interface <b>702</b>. For example, the interface <b>702</b> could be a command line interface that responds to text instructions. Further, the interface <b>702</b> can allow the developer to select different models to implement for different types of conditions, device types, etc. In this example, the developer is able to select whether Acoustic Model A or Acoustic Model B should be implemented for a first condition. The first condition would be whether or not the user device (e.g., an espresso machine) is being used in a home environment or a business/commercial environment. The user device can also select either Acoustic Model C or Acoustic Model D for a second condition, which could be related to whether or not there are nearby appliances that make noise. For example, Acoustic Model C could be selected by the developer when a bean grinder is known or is expected to be nearby. In this example, let's say that the developer has selected Acoustic Model B (home use) and Acoustic Model C (bean grinder in close proximity).

0055Accordingly, Acoustic Models B and C can be stored in relation to a device ID, or any other type of identification discussed in this document. In an implementation where the phrase interpreter <b>312</b> selects the best acoustic model, then the Acoustic Models B and C can be transmitted and/or stored by the phrase interpreter <b>312</b>, or the phrase interpreter <b>312</b> can be made aware of the locations of Acoustic Models B and C and be made aware that Acoustic Models B and C are the options for the espresso machine what that certain identifier (ID). Now the phrase interpreter <b>312</b> can select either Acoustic Model B or Acoustic Model C based on the metadata received along with the speech audio. Additionally, as mentioned above, the acoustic models can be mixed-and-matched (e.g., partially combined) to provide the best results.

0056Alternatively, Acoustic Model B and Acoustic Model C can be transmitted to the user device <b>302</b> (e.g., the espresso machine) so that the user device <b>302</b> can make the selection of the appropriate acoustic model. A developer of an application running on the user devices <b>302</b>, <b>304</b>, <b>306</b> may select the acoustic models from the interface <b>702</b>.

0057The acoustic model selection interface <b>320</b> can also be capable of providing speech audio along with metadata to the speech recognition platform/server <b>311</b> as opposed to the speech audio and metadata going from the user devices <b>302</b>, <b>304</b>, <b>306</b> to the speech recognition platform/server <b>311</b>.

0058For example, a developer could select, through the interface <b>702</b> that is running on a computer system, a set of at least two acoustic models (or just a single acoustic model) appropriate for a specific type of user device (e.g., the washing machine <b>306</b>). Then, at a later point, speech audio along with metadata that has been received by the acoustic model selection interface <b>320</b> can be transmitted to the speech recognition platform/server <b>311</b> via a computer system running the acoustic model selection interface <b>320</b> and/or from the user device. The speech recognition platform/server <b>311</b> then (using the phrase interpreter <b>312</b>) provides the computer system running the acoustic model selection interface <b>320</b> meaningful instructions in dependence upon a selection of one of the acoustic models from the set. The phrase interpreter <b>312</b> is able to intelligently select one of the models from the set based on the metadata and then proceed to determine the meaningful instructions.

0059Alternatively, the computer system running the acoustic model selection interface <b>320</b> may select one of the acoustic models from the set of acoustic models in dependence upon the metadata or other information and then instruct the phrase interpreter <b>312</b> to implement the selected acoustic model.

0060Furthermore, the acoustic models presented to the developer on the interface <b>702</b> can be preselected so that they are acoustic models that are appropriate for the specific type of the user device. These acoustic models that are appropriate for the specific type of computing the can be preselected in dependence upon metadata received from or related to the specific type of user device in an active session with the user device. In other words, metadata received from a user device can be analyzed and the appropriate acoustic models can be preselected in dependence on the received metadata and then presented to the developer. The purpose of such an implementation is to not overwhelm the developer with acoustic models that are not relevant or to prevent the developer from selecting acoustic models that would be more detrimental than beneficial. Also, outside of an active session with a user device, the developer still may make selections of acoustic models appropriate for different types of user devices. In a similar manner as discussed above, the computing system running the acoustic model selection interface <b>320</b> can preselect acoustic models that are appropriate for each type of user device that the developer is configuring.

0061Moreover, the developers and/or manufacturers of the user devices <b>302</b>, <b>304</b>, <b>306</b> may have developed customized acoustic models or trained customized acoustic models that are tailored specifically for the types of user devices <b>302</b>, <b>304</b>, <b>306</b> and/or the environments or conditions in which the user devices <b>302</b>, <b>304</b>, <b>306</b> may or can be implemented. These customized models can be presented to the developer via the interface <b>702</b> for selection. An environment for training acoustic models is discussed below with reference to <figref idref="DRAWINGS">FIG. <b>8</b></figref>.

0062<figref idref="DRAWINGS">FIG. <b>8</b></figref> illustrates an example implementation of the environment of <figref idref="DRAWINGS">FIG. <b>3</b></figref>, in which customized acoustic models can be trained. In particular, <figref idref="DRAWINGS">FIG. <b>8</b></figref> illustrates the environment <b>300</b> of <figref idref="DRAWINGS">FIG. <b>3</b></figref> and also illustrates how acoustic models <b>322</b> can be trained prior to implementation.

0063Specifically, <figref idref="DRAWINGS">FIG. <b>8</b></figref> illustrates that model training <b>808</b> can be performed by inputting training data such as clean speech <b>802</b> and noise <b>804</b> into an acoustic model. The acoustic models that are trained can be provided by the developer or manufacturer, or they can be generic models that are trained for implementation in specific types of devices and/or environments. The clean speech <b>802</b> can be generic or it can be specifically selected base on phrases that are expected to be received by the user devices <b>302</b>, <b>304</b>, <b>306</b>. For example, different training speech is needed for different languages. Similarly, the noise <b>804</b> can be generic or it can be selected based on types of noises that would be expected in the operating environment of the user devices <b>302</b>, <b>304</b>, <b>306</b>. The noise <b>804</b> can be provided by the developers and/or manufacturers. The developers and/or manufacturers can supply the noise <b>804</b> in the form of customized noise data or even a customized noise model that generates noises accordingly. The developers and/or manufacturers can also supply a customized acoustic model for immediate implementation as part of the acoustic models <b>322</b> and the developers and/or manufacturers can supply a customized acoustic model for further model training <b>808</b>. Furthermore, clean speech <b>802</b> and/or noise <b>804</b> can be supplied to the developers and/or manufacturers so that the developers and/or manufacturers can train the acoustic models themselves and then eventually supply the customized and trained acoustic models for implementation from the acoustic models <b>322</b>. The repository of acoustic models <b>322</b> can be parsed or separated to prevent security concerns one of developer's and/or manufacturer's model being implemented by another developer and/or manufacturer.

0064The developer and/or manufacturer can also train the acoustic models using an interface that is similar to the acoustic model selection interface <b>320</b>. Once the acoustic models are trained, they can be selected using the interface <b>702</b> as discussed above with reference to <figref idref="DRAWINGS">FIG. <b>7</b></figref>.

0065Some embodiments are devices or serve devices, such as mobile phones, that can run in different software conditions such as by running different apps. The status of what app or apps are running is one type of condition that can be useful for selecting an appropriate acoustic model. For example, an email app is most often used in relatively low-noise environments. A navigation app might indicate the likely presence of vehicle road or street noise. An app that outputs audio such as a music player, video player, or game would favor an acoustic model that is resilient to background musical sounds.

0066Some embodiments are devices or serve devices that run in different physical or hardware conditions. For example, the geolocation or type of motion of mobile and portable devices is useful for guiding the selection of acoustic models. For example, devices in stadiums will favor acoustic models trained for background voices and devices in motion at high speed will favor acoustic models trained for road noise.

0067Essentially any type of sensor found in mobile phones such as light level sensors, accelerometers, microphones, cameras, satellite navigation (such as GPS) receivers, and Bluetooth receivers and any type of sensor found in automobiles such as cameras, LIDAR, geolocation, light level, traction level, and engine condition, can provide information useful for acoustic model selection.

0068Some embodiments gather commonly detectable device condition data and apply it to speech recordings, either online or offline, using either supervised or unsupervised machine learning algorithms to train models for selecting or adapting acoustic models for best accuracy in given device conditions.

0069Some embodiments select an acoustic model according to a device type, as encoded by metadata associated with speech audio. Some such embodiments have a general code for each of multiple types of devices, such as ones distinguishing between washing machine, coffee machine, and automobile. Some embodiments encode a model number as metadata, which is useful for distinguishing between a home coffee machine and a commercial coffee machine or distinguishing between sporty and luxury vehicles. Some embodiments encode a serial number that uniquely identifies each manufactured instance of a device. This can be useful for personalizing the selection of an acoustic model for the actual typical usage of the device. For example, some models of coffee maker are useful for home and office, which have different noise environments. Some devices are purchased by consumers with different accents. Selecting an acoustic model based on a device serial number can improve speech recognition accuracy if the chosen acoustic model favors people with the consumer's particular accent.

0070Some embodiments, such as vehicles and wearable devices, either sometimes or always operate without access to a speech recognition server through a network. Such embodiments perform speech recognition locally using one or more of multiple locally-stored acoustic models <b>322</b>. Device type is not a particularly useful type of metadata, but device conditions are useful for the per-utterance selection of the best acoustic model for speech recognition.

0071Some embodiments that perform local speech recognition using a choice of acoustic model guided by metadata comprise storage elements that store metadata. For example, an automobile stores metadata indicating the position of the windows, status of the ventilation fan, and volume of its sounds system, all of which are useful in various embodiments for selection of an appropriate acoustic model. Using metadata that is stored within an embodiment for methods of selecting an acoustic model should be construed as being received for the purpose of carrying out the method.

0072Some network-connected server-based systems store appropriate acoustic models locally within devices and, for each utterance or for each detected change of condition choose a best acoustic model. When sending an utterance over the network to a speech recognition server, the system sends the acoustic model with the speech audio.

0073Some embodiments, such as ones for dictation, small vocabulary command recognition, keyword search, or phrase spotting perform speech recognition without natural language understanding and, in some embodiments, without using a language model.

0074Some embodiments are, or comprise, custom speech recognition platforms, such as SoundHound Houndify. These provide server-based speech recognition and, in some embodiments, also natural language processing and virtual assistant functionality. Platforms according to some embodiments provide interfaces for developers to customize the speech recognition for their particular devices. Some such platforms simply offer a selection of whether speech recognition should use near-field or far-field acoustic models. Some platforms offer numerous other configuration parameters such as selections of vocabulary size, numbers of microphones, application type, noise profile, and device price-point.

0075Some embodiments comprise methods of using such platform configuration interfaces to configure speech recognition for a type of device. Some companies developing speech-enabled systems use such platforms to configure the operation of server-based recognition for their client devices. Some companies use platforms to configure speech recognition software to run locally on devices. In either case, some platforms offer, and some developers use an ability to recognize test speech audio. Some such platforms and users provide test speech audio along with test metadata to observe and vary the intended performance of acoustic model selection for their devices and systems under development.

0076Some embodiments comprise one or more computer readable medium, such as hard disk drives, solid-state drives, or Flash RAM chips. Some devices designed to work with server systems comprise such computer readable medium that stores software to control the devices to make it perform detection of metadata useful for selection of acoustic models. This can be, for example, by reading from a device-local sensor or reading a stored device status value from a storage medium. Such software also controls the device to receive speech audio, transmit the speech audio and metadata to a server, and receive requested information back from the server. For example, the speech audio can be a request for a weather report and the received information would be a description of the weather report. For another example, the speech audio can be a request to send a text message and the received information would be a data structure that controls the device to perform a text message creation function.

0077For purposes of the present invention, the passive act of having data in a storage medium should be construed as an act of storing, regardless of who wrote the data to the storage medium and when or how the writing occurred.

0078<figref idref="DRAWINGS">FIG. <b>9</b></figref> is a block diagram of an example computer system that can implement various components of the environment <b>300</b> of <figref idref="DRAWINGS">FIG. <b>3</b></figref>. Computer system <b>910</b> typically includes at least one processor <b>914</b>, which communicates with a number of peripheral devices via bus subsystem <b>912</b>. These peripheral devices may include a storage subsystem <b>924</b>, comprising for example memory devices and a file storage subsystem, user interface input devices <b>922</b>, user interface output devices <b>920</b>, and a network interface subsystem <b>916</b>. The input and output devices allow user interaction with computer system <b>910</b>. Network interface subsystem <b>916</b> provides an interface to outside networks, including an interface to communication network <b>308</b>, and is coupled via communication network <b>308</b> to corresponding interface devices in other computer systems.

0079User interface input devices <b>922</b> may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as speech recognition systems, microphones, and other types of input devices. In general, use of the term “input device” is intended to include all possible types of devices and ways to input information into computer system <b>910</b> or onto communication network <b>308</b>.

0080User interface output devices <b>920</b> may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of the term “output device” is intended to include all possible types of devices and ways to output information from computer system <b>910</b> to the user or to another machine or computer system.

0081Storage subsystem <b>924</b> stores programming and data constructs that provide the functionality of some or all of the modules described herein. These software modules are generally executed by processor <b>914</b> alone or in combination with other processors.

0082Memory <b>926</b> used in the storage subsystem can include a number of memories including a main random access memory (RAM) <b>930</b> for storage of instructions and data during program execution and a read only memory (ROM) <b>932</b> in which fixed instructions are stored. A file storage subsystem <b>928</b> can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain embodiments may be stored by file storage subsystem <b>928</b> in the storage subsystem <b>924</b>, or in other machines accessible by the processor.

0083Bus subsystem <b>912</b> provides a mechanism for letting the various components and subsystems of computer system <b>910</b> communicate with each other as intended. Although bus subsystem <b>912</b> is shown schematically as a single bus, alternative embodiments of the bus subsystem may use multiple busses.

0084Computer system <b>910</b> can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computer system <b>910</b> depicted in <figref idref="DRAWINGS">FIG. <b>9</b></figref> is intended only as a specific example for purposes of illustrating the various embodiments. Many other configurations of computer system <b>910</b> are possible having more or fewer components than the computer system depicted in <figref idref="DRAWINGS">FIG. <b>9</b></figref>.

Some Particular Implementations

0085We describe various implementations for performing speech recognition.

0086The technology disclosed can be practiced as a system, method, or article of manufacture (a non-transitory computer readable medium storing code). One or more features of an implementation can be combined with the base implementation. Implementations that are not mutually exclusive are taught to be combinable. One or more features of an implementation can be combined with other implementations. This disclosure periodically reminds the user of these options. Omission from some implementations of recitations that repeat these options should not be taken as limiting the combinations taught in the preceding sections—these recitations are hereby incorporated forward by reference into each of the following implementations.

0087A system implementation of the technology disclosed includes one or more processors coupled to memory. The memory is loaded with computer instructions that perform various operations. A CRM implementation of the technology discloses includes a non-transitory computer readable medium storing code that, if executed by one or more computers, would cause the one or more computers to perform various operations. The system implementation and the CRM implementation are capable of performing any of the method implementations described below.

0088In one implementation a method of performing speech recognition for a plurality of different devices s provided. The method includes storing a plurality of acoustic models associated with different device conditions, receiving speech audio including natural language utterances, receiving metadata indicative of a device condition, selecting an acoustic model from the plurality of acoustic models, the acoustic model being selected in dependence upon the received metadata indicative of the device condition, and employing the selected acoustic model to recognize speech from the natural language utterances included in the received speech audio.

0089In another implementation, a method of performing speech recognition for a plurality of different devices is provided. The method includes storing a plurality of acoustic models associated with different device types, receiving speech audio including natural language utterances, receiving metadata indicative of a device type, selecting an acoustic model from the plurality of acoustic models, the acoustic model being selected in dependence upon the received metadata indicative of the device type, and employing the selected acoustic model to recognize speech from the natural language utterances included in the received speech audio.

0090Further, in a different implementation of a method of providing a platform for configuring device-specific speech recognition is provided. The method includes providing a user interface for developers to select a set of at least two acoustic models appropriate for a specific type of a device, receiving, from a developer, a selection of the set of the at least two acoustic models, and configuring a speech recognition system to perform device-specific speech recognition.

0091In another implementation, a method of configuring a speech recognition system to perform device-specific speech recognition is provided. The method includes receiving, from a device of a specific device type, speech audio including natural language utterances and metadata associated with the received speech audio, selecting one acoustic model of at least two acoustic models in dependence upon the received metadata, and using the selected acoustic model to recognize speech from the natural language utterances included in the received speech audio.

0092In a further implementation, a method of using a platform for configuring device-specific speech recognition is provided. The method includes selecting, through a user interface provided by a computer system, a set of at least two acoustic models appropriate for a specific type of a device, providing speech audio with metadata to a speech recognition system associated with the platform, and receiving meaningful instructions from the computer speech recognition system, wherein the meaningful instructions are created by the speech recognition system in dependence upon a selection of one of the acoustic models from the set.

0093In another implementation, a non-transitory computer readable medium storing code is provided. The code, if executed by one or more computers, would cause the one or more computers to detect information useful for selecting an acoustic model and indicative of a device condition, receive speech audio, transmit the detected information and the received speech audio, and receive information requested by speech in the speech audio, wherein the detected information is capable of being employed to select the acoustic model from a plurality of acoustic models associated with different device conditions, and wherein the selected acoustic model is used to recognize speech from the transmitted speech audio.

0094Features applicable to systems, methods, and articles of manufacture are not repeated for each statutory class set of base features. The reader will understand how features identified in this section can readily be combined with base features in other statutory classes.