Distributed speech recognition using one way communication

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from copending and commonly-owned U.S. Prov. Pat. App. Ser. No. 61/093,221, filed on Aug. 29, 2008, entitled, “Distributed Speech Recognition Using One Way Communication,” which is hereby incorporated by reference herein.

BACKGROUND

A variety of automatic speech recognizers (ASRs) exist for performing functions such as converting speech into text and controlling the operations of a computer in response to speech. Some applications of automatic speech recognizers require shorter turnaround times (the amount of time between when the speech is spoken and when the speech recognizer produces output) than others in order to appear responsive to the end user. For example, a speech recognizer that is used for a “live” speech recognition application, such as controlling the movement of an on-screen cursor, may require a shorter turnaround time (also referred to as a “response time”) than a speech recognizer that is used to produce a transcript of a medical report.

The desired turnaround time may depend, for example, on the content of the speech utterance that is processed by the speech recognizer. For example, for a short command-and-control utterance, such as “close window,” a turnaround time above 500 ms may appear sluggish to the end user. In contrast, for a long dictated sentence which the user desires to transcribe into text, response times of 1000 ms may be acceptable to the end user. In fact, in the latter case users may prefer longer response times because they may otherwise feel that their speech is being interrupted by the immediate display of text in response to their speech. For longer dictated passages, such as entire paragraphs, even longer response times of multiple seconds may be acceptable to the end user.

In typical prior art speech recognition systems, increasing response time while maintaining recognition accuracy requires increasing the computing resources (processing cycles and/or memory) that are dedicated to performing speech recognition. As a result, many applications which require fast response times require the speech recognition system to execute on the same computer as that on which the applications themselves execute. Although such colocation may eliminate the delay that would otherwise be introduced by requiring the speech recognition results to be transmitted to the requesting application over a network, such colocation also has a variety of disadvantages.

For example, colocation requires a speech recognition system to be installed on every end user device—such as every desktop computer, laptop computer, cellular telephone, and personal digital assistant (PDA)—which requires speech recognition functionality. Installing and maintaining such speech recognition systems on such a large number and wide variety of devices can be tedious and time-consuming for end users and system administrators. For example, such maintenance requires system binaries to be updated when a new release of the speech recognition system becomes available. User data, such as speech models, are created and accumulated over time on individual devices, taking up precious storage space, and need to be synchronized with multiple devices used by the same user. Such maintenance can grow particularly burdensome as users continue to use speech recognition systems on a wider number and variety of devices.

Furthermore, locating a speech recognition system on the end user device causes the speech recognition system to consume precious computing resources, such as CPU processing cycles, main memory, and disk space. Such resources are particularly scarce on handheld mobile devices such as cellular telephones. Producing speech recognition results with fast turnaround times using such devices typically requires sacrificing recognition accuracy and reducing the resources available to other applications executing on the same device.

One known technique for overcoming these resource constraints in the context of embedded devices is to delegate some or all of the speech recognition processing responsibility to a speech recognition server that it located remotely from the embedded device and which has significantly greater computing resources than the embedded device. When a user speaks into the embedded device in this situation, the embedded device does not attempt to recognize the speech using its own computing resources. Instead, the embedded device transmits the speech (or a processed form of it) over a network connection to the speech recognition server, which recognizes the speech using its greater computing resources and therefore produces recognition results more quickly than the embedded device could have produced with the same accuracy. The speech recognition server then transmits the results back over the network connection to the embedded device. Ideally this technique produces highly-accurate speech recognition results more quickly than would otherwise be possible using the embedded device alone.

In practice, however, this “server-side speech recognition” technique has a variety of shortcomings. In particular, because server-side speech recognition relies on the availability of high-speed and reliable network connections, the technique breaks down if such connections are not available when needed. For example, the potential increases in speed made possible by server-side speech recognition may be negated by use of a network connection without sufficiently high bandwidth. As one example, the typical network latency of an HTTP call to a remote server can range from 100 ms to 500 ms. If spoken data arrives at a speech recognition server 500 ms after it is spoken, it will be impossible for that server to produce results quickly enough to satisfy the minimum turnaround time (500 ms) required by command-and-control applications. As a result, even the fastest speech recognition server will produce results that appear sluggish if used in combination with a slow network connection.

Furthermore, conventional server-side speech recognition techniques assume that the network connection established between the client (e.g., embedded device) and speech recognition server is kept alive continuously during the entire recognition process. Although it may be possible to satisfy this condition in a Local Area Network (LAN) or when both client and server are managed by the same entity, this condition may be impossible or at least unreasonable to satisfy when the client and server are connected over a Wide Area Network (WAN) or the Internet, in which case interruptions to the network connection may be common and unavoidable.

Furthermore, organizations often restrict the kinds of communications that their users can engage in over public networks such as the Internet. For example, organizations may only allow clients within their networks to engage in outbound communications. This means that a client can contact an external server on a certain port, but that the server cannot initiate contact with the client. This is an example of one-way communication.

Another common restriction imposed on clients is that they may only use a limited range of outbound ports to communicate with external servers. Furthermore, outgoing communication on those ports may be required to be encrypted. For example, clients often are allowed to use only the standard HTTP port (port <b>80</b>) or the standard secure, encrypted HTTPS port (port <b>443</b>).

What is needed, therefore, are improved techniques for producing speech recognition results with fast response times without overburdening the limited computing resources of client devices.

SUMMARY

A speech recognition client sends a speech stream and control stream in parallel to a server-side speech recognizer over a network. The network may be an unreliable, low-latency network. The server-side speech recognizer recognizes the speech stream continuously. The speech recognition client receives recognition results from the server-side recognizer in response to requests from the client. The client may remotely reconfigure the state of the server-side recognizer during recognition.

Other features and advantages of various aspects and embodiments of the present invention will become apparent from the following description and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

<figref idrefs="DRAWINGS">FIG. 1</figref> is a dataflow diagram of a system for performing speech recognizing over a low-latency network according to one embodiment of the present invention;

<figref idrefs="DRAWINGS">FIG. 2A</figref> is a flowchart of a method performed by the system of <figref idrefs="DRAWINGS">FIG. 1</figref> according to one embodiment of the present invention;

<figref idrefs="DRAWINGS">FIG. 2B</figref> is a flowchart of a method performed by a server-side automatic speech recognizer to recognize a segment of speech according to one embodiment of the present invention;

<figref idrefs="DRAWINGS">FIG. 2C</figref> is a flowchart of a method performed by a server-side automatic speech recognizer as part of performing speech recognition on segments of speech according to one embodiment of the present invention;

<figref idrefs="DRAWINGS">FIG. 2D</figref> is a flowchart of a method performed by a server-side recognizer to ensure that the recognizer is reconfigured after certain recognition results are obtained and before further recognition is performed according to one embodiment of the present invention;

<figref idrefs="DRAWINGS">FIG. 3</figref> is a diagram of a speech stream according to one embodiment of the present invention; and

<figref idrefs="DRAWINGS">FIG. 4</figref> is a diagram of a command and control stream according to one embodiment of the present invention.

DETAILED DESCRIPTION

Referring to <figref idrefs="DRAWINGS">FIG. 1</figref>, a dataflow diagram is shown of a speech recognition system <b>100</b> according to one embodiment of the present invention. Referring to <figref idrefs="DRAWINGS">FIG. 2A</figref>, a flowchart is shown of a method <b>200</b> performed by the system <b>100</b> of <figref idrefs="DRAWINGS">FIG. 1</figref> according to one embodiment of the present invention.

A user <b>102</b> of a client device <b>106</b> speaks and thereby provides speech <b>104</b> to the client device <b>106</b> (step <b>202</b>). The client device <b>106</b> may be any device, such as a desktop or laptop computer, cellular telephone, personal digital assistant (PDA), or telephone. Embodiments of the present invention, however, are particularly useful in conjunction with resource-constrained clients, such as computers or mobile computing devices with slow processors or small amounts of memory, or computers running resource-intensive software. The device <b>106</b> may receive the speech <b>104</b> from the user <b>102</b> in any way, such as through a microphone connected to a sound card. The speech <b>104</b> may be embodied in an audio signal which is tangibly stored in a computer-readable medium and/or transmitted over a network connection or other channel. The speech <b>104</b> may, for example, include multiple audio streams, as in the case of “push to talk” applications, in which each push initiates a new audio stream.

The client device <b>106</b> includes an application <b>108</b>, such as a transcription application or other application which needs to recognize the speech <b>104</b>. Although the application <b>108</b> may be any kind of application that uses speech recognition results, assume for purposes of the following discussion that the application <b>108</b> is a “live” recognition application for transcribing speech. Portions of the speech <b>104</b> provided by the user <b>102</b> in this context may fall into one of two basic categories: dictated speech to be transcribed (e.g., “The patient is a 35 year-old male”) or commands (such as “delete this” or “sign and submit”).

The client device <b>106</b> also includes a speech recognition client <b>140</b>. Although the speech recognition client <b>140</b> is shown in <figref idrefs="DRAWINGS">FIG. 1</figref> as a separate module from the application <b>108</b>, alternatively the speech recognition client <b>140</b> may be part of the application <b>108</b>. The application <b>108</b> provides the speech <b>104</b> to the speech recognition client <b>140</b>. Alternatively, the application <b>108</b> may process the speech <b>104</b> in some way and provide the processed version of the speech <b>104</b>, or other data derived from the speech, to the speech recognition client <b>140</b>. The speech recognition client <b>140</b> itself may process the speech <b>104</b> (in addition to or instead of any processing performed on the speech by the application <b>108</b>) in preparation for transmitting the speech <b>104</b> for recognition.

The speech recognition client <b>140</b> transmits the speech <b>104</b> over a network <b>116</b> to a server-side speech recognition engine <b>120</b> located on a server <b>118</b> (step <b>204</b>). Although the client <b>140</b> may transmit the entire speech <b>104</b> to the server <b>118</b> using a single server configuration, doing so may produce suboptimal results. To improve recognition accuracy or change the context of the speech recognition engine <b>120</b>, the client <b>140</b> may instead reconfigure the speech recognition engine <b>120</b> at various points during transmission of the speech <b>104</b>, and therefore at various points during the speech recognition engine's recognition of the speech <b>104</b>. In general, configuration commands transmitted by the client <b>140</b> to the speech recognition engine <b>120</b> set the expectations of the recognizer <b>120</b> regarding the context and/or content of the speech that is to follow. Various prior art systems perform this configuration function by configuring the server-side recognition engine with an initial configuration, then sending some of the speech to the server, then reconfiguring the server-side recognition engine, then sending more of the speech, and so on. This enables the server-side recognition engine to recognize different portions of the speech with configurations and in contexts that are designed to produce better results for later portions of the speech than would have been produced using the initial configuration.

It is undesirable, however, to require the speech recognition client <b>140</b> to wait to receive an acknowledgement from the server <b>118</b> that the previous reconfiguration command has been processed by the server <b>118</b> before sending the next portion of the speech <b>104</b> to the server <b>118</b>, because such a requirement could introduce a significant delay into the recognition of the speech <b>104</b>, particularly if the network connection is slow and/or unreliable. It is also undesirable to stop server-side processing of the speech until the server receives instructions from the client-side application <b>108</b> on how to process subsequent speech. In prior art systems, however, the server needs to stop processing speech until it receives such instructions, such as reconfiguration commands, from the client.

Embodiments of the present invention address these and other problems as follows. The speech recognition client <b>140</b> transmits the speech <b>104</b> to the server <b>118</b> in a speech stream <b>110</b> over the network <b>116</b> (<figref idrefs="DRAWINGS">FIG. 2</figref>, step <b>204</b>). As shown in <figref idrefs="DRAWINGS">FIG. 3</figref>, the speech stream <b>110</b> may be divided into segments <b>302</b><i>a</i>-<i>e</i>, each of which may represent a portion of the speech <b>104</b> (e.g., 150-250 ms of the speech <b>104</b>). Sending the speech <b>104</b> in segments enables the speech recognition client <b>140</b> to transmit portions of the speech <b>104</b> to the server <b>118</b> relatively soon after those portions become available to the speech recognition client <b>140</b>, thereby enabling the recognizer <b>120</b> to begin recognizing those portions with minimal delay. The application <b>108</b> may, for example, send the first segment <b>302</b><i>a </i>immediately after it becomes available, even as the second segment <b>302</b><i>b </i>is being generated. Furthermore, the client <b>140</b> may transmit individual portions in the speech stream <b>110</b> to the server <b>118</b> without using a standing connection (e.g., socket). As a result, a connectionless or stateless protocol, such as HTTP, may be used by the speech recognition client <b>140</b> to transmit the speech stream <b>110</b> to the server <b>118</b>.

Although only five representative segments <b>302</b><i>a</i>-<i>e </i>are shown in <figref idrefs="DRAWINGS">FIG. 2A</figref> for ease of illustration, in practice the speech stream <b>110</b> may contain any number of segments, which may grow as the user <b>102</b> continues to speak. The application <b>108</b> may use any procedure to divide the speech <b>104</b> into segments, or to stream the speech <b>104</b> to the server <b>118</b> over, for example, an HTTP connection.

Each of the speech segments <b>302</b><i>a</i>-<i>e </i>contains data <b>304</b><i>a </i>representing a corresponding portion of the speech <b>104</b> of the user <b>102</b>. Such speech data <b>304</b><i>a </i>may be represented in any appropriate format. Each of the speech segments <b>302</b><i>a</i>-<i>e </i>may contain other information, such as the start time <b>304</b><i>b </i>and end time <b>304</b><i>c </i>of the corresponding speech data <b>304</b><i>a</i>, and a tag <b>304</b><i>d </i>which will be described in more detail below. The particular fields <b>304</b><i>a</i>-<i>d </i>illustrated in <figref idrefs="DRAWINGS">FIG. 3</figref> are merely examples and do not constitute limitations of the present invention.

In general, the server-side recognizer <b>120</b> queues segments from the speech stream <b>110</b> into a first-in first-out processing queue <b>124</b> at the server <b>118</b> (<figref idrefs="DRAWINGS">FIG. 2</figref>, step <b>216</b>). With certain exceptions that will be described in more detail below, the server-side recognizer <b>120</b> pulls segments from the processing queue <b>124</b> as soon as possible after they become available and performs speech recognition on those segments to produce speech recognition results (step <b>218</b>), which the server <b>120</b> queues into a first-in first-out output queue <b>134</b> (step <b>220</b>).

The application <b>108</b>, through the speech recognition client <b>140</b>, may also send a control stream <b>112</b> to the server-side recognizer <b>120</b> over the network <b>116</b> as part of step <b>204</b>. As shown in <figref idrefs="DRAWINGS">FIG. 4</figref>, the control stream <b>112</b> may include control messages <b>402</b><i>a</i>-<i>c</i>, transmitted in sequence to the recognizer <b>120</b>. Although only three representative control messages <b>402</b><i>a</i>-<i>c </i>are shown in <figref idrefs="DRAWINGS">FIG. 4</figref> for ease of illustration, in practice the control stream <b>112</b> may contain any number of control messages. As will be described in more detail below, each of the control messages <b>402</b><i>a </i>may contain a plurality of fields, such as a command field <b>404</b><i>a </i>for specifying a command to be executed by the server-side recognizer <b>120</b>, a configuration object field <b>404</b><i>b </i>for specifying a configuration object, and a timeout value field <b>404</b><i>c </i>for specifying a timeout value. The particular fields <b>304</b><i>a</i>-<i>d </i>illustrated in <figref idrefs="DRAWINGS">FIG. 3</figref> are merely examples and do not constitute limitations of the present invention.

As shown in <figref idrefs="DRAWINGS">FIG. 1</figref>, the speech recognition client <b>140</b> may treat the speech stream <b>110</b> and control stream <b>112</b> as two different streams of data (steps <b>206</b> and <b>208</b>), transmitted in parallel from the speech recognition client <b>140</b> to the engine <b>120</b>. However, assuming that only one output port is available to the speech recognition client <b>140</b> for communicating with the server <b>118</b>, the client <b>106</b> may multiplex the speech stream <b>110</b> and the control stream <b>112</b> into a single data stream <b>114</b> transmitted to the server <b>118</b> (step <b>210</b>). The server <b>118</b> demultiplexes the signal <b>114</b> into its constituent speech stream <b>110</b> and control stream <b>112</b> on the server side (step <b>214</b>).

Any multiplexing scheme may be used. For example, if HTTP is used as a transport mechanism, then an HTTP client <b>130</b> and HTTP server <b>132</b> may transparently perform the multiplexing and demultiplexing functions, respectively, on behalf of the client <b>106</b> and server <b>118</b>. In other words, the speech recognition client <b>140</b> may treat the speech stream <b>110</b> and control stream <b>112</b> as two separate streams even though they are transmitted as a single multiplexed stream <b>114</b> because the HTTP client <b>130</b> multiplexes these two streams together automatically and transparently on behalf of the speech recognition client <b>140</b>. Similarly, the server-side recognizer <b>120</b> may treat the speech stream <b>110</b> and control stream <b>112</b> as two separate streams even though they are received by the server <b>118</b> as a single multiplexed stream <b>114</b> because the HTTP server <b>132</b> demultiplexes the combined stream <b>114</b> into two streams automatically and transparently on behalf of the server-side recognizer <b>120</b>.

As mentioned above, by default the server-side recognizer <b>120</b> pulls speech segments from the processing queue <b>124</b> in sequence, performs speech recognition on them, and queues the speech recognition results into the output queue <b>134</b>. The speech recognition client <b>108</b> receives the speech recognition results as follows. The speech recognition client <b>140</b> sends, in the control stream <b>112</b>, a control message whose command field <b>404</b><i>a </i>calls a method referred to herein as “DecodeNext.” This method takes as parameters a configuration update object <b>404</b><i>b </i>(which specifies how a configuration state <b>126</b> of the server-side recognizer <b>120</b> is to be updated), and a real-time timeout value <b>404</b><i>c</i>. Although the speech recognition client <b>140</b> may send other commands in the control stream <b>112</b>, only the DecodeNext command will be described here for ease of explanation.

The server-side recognizer <b>120</b> pulls control messages from the control stream <b>112</b> in sequence, as soon as possible after they are received, and in parallel with processing the speech segments in the speech stream <b>110</b> (step <b>222</b>). The server-side recognizer <b>120</b> executes the command in each control message in sequence (step <b>224</b>).

Referring to <figref idrefs="DRAWINGS">FIG. 2B</figref>, a flow chart is shown of a method performed by the server-side recognizer <b>120</b> to execute a DecodeNext control message in the control stream <b>112</b>. If at least one speech recognition result is in the output queue <b>134</b> (step <b>240</b>), the recognizer <b>120</b> sends the next result(s) <b>122</b> in the queue <b>134</b> to the speech recognition client <b>140</b> over the network <b>116</b> (step <b>242</b>). If more than one result is available in the queue <b>134</b> at the time step <b>242</b> is performed, then all available results in the queue <b>134</b> are transmitted in the results stream <b>122</b> to the speech recognition client <b>140</b>. (Although the results <b>122</b> are shown in <figref idrefs="DRAWINGS">FIG. 1</figref> as being transmitted directly from the recognizer <b>120</b> to the speech recognition client <b>140</b> for ease of illustration, the results <b>122</b> may be transmitted by the HTTP server <b>132</b> over the network <b>116</b> and received by the HTTP client <b>130</b> at the client device <b>106</b>.) The DecodeNext method then returns control to the application <b>108</b> (step <b>246</b>), and terminates.

Recall that the recognizer <b>120</b> is continuously performing speech recognition on the speech segments in the processing queue <b>124</b>. Therefore, if the output queue <b>134</b> is empty when the recognizer <b>120</b> begins to execute the DecodeNext method, the DecodeNext method blocks until at least one result (e.g., one word) is available in the output queue <b>134</b>, or until the amount of time specified by the timeout value <b>404</b><i>c </i>is reached (step <b>248</b>). If a result appears in the output queue <b>134</b> before the timeout value <b>404</b><i>c </i>is reached, then the DecodeNext method transmits that result to the speech recognition client <b>140</b> (step <b>242</b>), returns control to the speech recognition client <b>140</b> (step <b>246</b>), and terminates. If no results appear in the output queue <b>134</b> before the timeout value <b>404</b><i>c </i>is reached, then the DecodeNext method informs the speech recognition client <b>140</b> that no results are available (step <b>244</b>), returns control to the speech recognition client <b>140</b> (step <b>246</b>), and terminates without returning any recognition results to the speech recognition client <b>140</b>.

Once control returns to the speech recognition client <b>140</b> (after the DecodeNext method either returns a recognition result to the speech recognition client <b>140</b> or informs the speech recognition client <b>140</b> that no such results are available), the speech recognition client <b>140</b> may immediately send another DecodeNext message to the server <b>120</b> in an attempt to receive the next recognition result. The server <b>120</b> may process this DecodeNext message in the manner described above with respect to <figref idrefs="DRAWINGS">FIG. 2B</figref>. This process may repeat for subsequent recognition results. As a result, the control stream <b>112</b> may essentially always be blocking on the server side (in the loop represented by steps <b>240</b> and <b>248</b> in <figref idrefs="DRAWINGS">FIG. 2B</figref>), waiting for recognition results and returning them to the client application <b>108</b> as they become available.

The timeout value <b>404</b><i>c </i>may be chosen to be shorter than the timeout value of the underlying communication protocol used between the client <b>140</b> and server <b>120</b>, such as the HTTP timeout value. As a result, if the client <b>140</b> receives notification from the server that no speech recognition results were produced before the timeout value <b>404</b><i>c </i>was reached, the client <b>140</b> may draw the conclusion that the timeout was the result of the inability of the server <b>120</b> to produce any speech recognition results before the timeout value <b>404</b><i>c </i>was reached, rather than as the result of a network communication problem. Regardless of the reason for the timeout, however, the client <b>140</b> may send another DecodeNext message to the server <b>120</b> after such a timeout.

The examples described above involve two fully unsynchronized data streams <b>110</b> and <b>112</b>. However, it may be desirable to perform certain kinds of synchronization on the two streams <b>110</b> and <b>112</b>. For example, it may be useful for the speech recognition client <b>140</b> to ensure that the recognizer <b>120</b> is in a certain configuration state before beginning to recognize the speech stream <b>110</b>. For example, the recognizer <b>120</b> may use the textual context of the current cursor position in a text edit window to guide recognition for text that is to be inserted at that cursor position. Since the cursor position may change frequently due to mouse or other keyboard events, it may be useful for the application <b>108</b> to delay transmission of the text context to the server <b>120</b> until the user <b>102</b> presses the “start recording” button. In this case, the server-side recognizer <b>120</b> must be prevented from recognizing speech transmitted to the server <b>120</b> until the correct text context is received by the server <b>120</b> and the server <b>120</b> updates its configuration state <b>126</b> accordingly.

As another example, some recognition results may trigger the need to change the configuration state <b>126</b> of the recognizer <b>120</b>. As a result, when the server-side recognizer <b>120</b> generates such a result, it should wait until it is reconfigured before generating the next result. For example, if the recognizer <b>120</b> produces the result, “delete all,” the application <b>108</b> may next attempt to verify the user's intent by prompting the user <b>102</b> as follows: “Do you really want to delete all? Say YES or NO.” In this case, the application <b>108</b> (through the speech recognition client <b>140</b>) should reconfigure the recognizer <b>120</b> with a “YES|NO” grammar before the recognizer <b>120</b> attempts to recognize the next segment in the speech stream <b>110</b>.

Such results may be obtained as follows, as shown by the flowchart of <figref idrefs="DRAWINGS">FIG. 2C</figref>, which illustrates a method which may be performed by the server-side recognizer <b>120</b> as part of performing speech recognition on the audio segments in the processing queue (<figref idrefs="DRAWINGS">FIG. 2A</figref>, step <b>218</b>). Each recognizer configuration state is assigned a unique configuration state identifier (ID). The speech recognition client <b>140</b> assigns integer values to configuration state IDs, such that if ID1&gt;ID2, then the configuration state associated with ID1 is more recent than the configuration state associated with ID2. As described above with respect to <figref idrefs="DRAWINGS">FIG. 3</figref>, the speech recognition client <b>140</b> also provides tags <b>304</b><i>d </i>within each of the speech stream segments <b>302</b><i>a</i>-<i>e </i>which indicate the minimum required configuration state ID number that is required before recognition of that segment can begin.

When the server-side recognizer <b>120</b> retrieves the next audio segment from the processing queue <b>124</b> (step <b>262</b>), the recognizer <b>120</b> compares the configuration state ID <b>136</b> of the recognizer's current configuration state <b>126</b> to the minimum required configuration ID specified by the retrieved audio segment's tag <b>304</b><i>d</i>. If the current configuration ID <b>136</b> is at least as great as the minimum required configuration ID (step <b>264</b>), then the server <b>120</b> begins recognizing the retrieved audio segment (step <b>266</b>). Otherwise, the server <b>120</b> waits until its configuration ID <b>136</b> reaches the minimum required ID before it begins recognizing the current speech segment. Since the method of <figref idrefs="DRAWINGS">FIG. 2C</figref> may be performed in parallel with the method <b>200</b> of <figref idrefs="DRAWINGS">FIG. 2A</figref>, the configuration ID <b>136</b> of the server-side recognizer <b>120</b> may be updated by execution of control messages <b>224</b> even while the method of <figref idrefs="DRAWINGS">FIG. 2C</figref> blocks in the loop over step <b>264</b>. Furthermore, note that even while the server <b>120</b> waits to process speech from the processing queue <b>124</b>, the server <b>120</b> continues to receive additional segments from the speech stream <b>110</b> and queue those segments into the processing queue <b>124</b> (<figref idrefs="DRAWINGS">FIG. 2A</figref>, steps <b>214</b>-<b>216</b>).

As another example of ways in which the speech stream <b>110</b> and control stream <b>112</b> may be synchronized, the application <b>108</b>, through the speech recognition client <b>140</b>, may instruct the recognizer <b>120</b> ahead of time to stop recognizing the speech stream <b>110</b>, or take some other action, upon producing any recognition result or upon producing a recognition result satisfying certain criteria. Such criteria may effectively serve as breakpoints which the application <b>108</b>, through the speech recognition client <b>140</b>, may use to proactively control how far ahead the recognizer <b>120</b> produces recognition results.

For example, consider a context in which the user <b>102</b> may issue any of the following voice commands: “delete,” “next,” “select all,” and “open file chooser.” In this context, a possible configuration, which may be specified by the configuration update object <b>404</b><i>b</i>, would be: &lt;delete, continue&gt;, &lt;next, continue&gt;, &lt;select all, continue&gt;, &lt;open file chooser, stop&gt;. Such a configuration instructs the server-side recognizer <b>120</b> to continue recognizing the speech stream <b>110</b> after obtaining the recognition result “delete,” “next,” or “select all,” but to stop recognizing the speech stream <b>110</b> after obtaining the recognition result “open file chooser.” The reason for configuring the recognizer <b>120</b> in this way is that production of the results “delete,” “next,” or “select all” do not require the recognizer <b>120</b> to be reconfigured before producing the next result. Therefore, the recognizer <b>120</b> may be allowed to continue recognizing the speech stream <b>110</b> after producing any of the results “delete,” “next,” or “select all,” thereby enabling the recognizer <b>120</b> to continue recognizing the speech <b>104</b> at full speed (see <figref idrefs="DRAWINGS">FIG. 2D</figref>, step <b>272</b>). In contrast, production of the result “open file chooser” requires the recognizer <b>120</b> to be reconfigured (e.g., to expect results such as “OK,” “select file1.xml,” or “New Folder”) before recognizing any subsequent segments in the speech stream <b>110</b> (see <figref idrefs="DRAWINGS">FIG. 2C</figref>, step <b>274</b>). Therefore, if the application <b>108</b>, through the speech recognition client <b>140</b>, is informed by the recognizer <b>120</b> that the result “open file chooser” was produced, the application <b>108</b>, through the speech recognition client <b>140</b>, may reconfigure the recognizer <b>120</b> with a configuration state that is appropriate for control of a file chooser. Enabling the application <b>108</b> to pre-configure the recognizer <b>120</b> in this way strikes a balance between maximizing the recognizer's response time and ensuring that the recognizer <b>120</b> uses the proper configuration state to recognize different portions of the speech <b>104</b>.

Note that even if the recognizer <b>120</b> stops recognizing speech from the processing queue <b>124</b> as the result of a configuration “stop” command (step <b>274</b>), the recognizer <b>120</b> may continue to receive speech segments from the speech stream <b>110</b> and to queue those segments into the processing queue <b>124</b> (<figref idrefs="DRAWINGS">FIG. 2A</figref>, steps <b>214</b>, <b>216</b>). As a result, additional segments of the speech stream <b>110</b> are ready to be processed as soon as the recognizer <b>120</b> resumes performing speech recognition.

As mentioned above, the techniques disclosed herein may be used in conjunction with one-way communication protocols, such as HTTPS. Such communication protocols are simple to set up on wide area networks, but offer little guarantee against failures. Failures may occur during a request between the client <b>130</b> and server <b>132</b> that may leave the application <b>108</b> in an ambiguous state. For example, a problem may occur when either party (client application <b>108</b> or server-side recognizer <b>120</b>) fails while in the midst of a call. Other problems may occur, for example, due to lost messages to or from the server <b>118</b>, messages arriving at the client <b>106</b> or server <b>118</b> out of sequence, or messages mistakenly sent as duplicates. In general, in prior art systems it is the responsibility of the speech recognition client <b>140</b> to ensure the robustness of the overall system <b>100</b>, since the underlying communications protocol does not guarantee such robustness.

Embodiments of the present invention are robust against such problems by making all messages and events exchanged between the speech recognition client <b>140</b> and server-side recognizer <b>120</b> idempotent. An event is idempotent if multiple occurrences of the same event have the same effect as a single occurrence of the event. Therefore, if the speech recognition client <b>140</b> detects a failure, such as failure to transmit a command to the server-side recognizer <b>120</b>, the speech recognition client <b>140</b> may re-transmit the command, either immediately or after a waiting period. The speech recognition client <b>140</b> and recognizer <b>120</b> may use a messaging application program interface (API) which guarantees that the retry will leave the system <b>100</b> in a coherent state.

In particular, the API for the speech stream <b>110</b> forces the speech recognition client <b>140</b> to transmit the speech stream <b>110</b> in segments. Each segment may have a unique ID <b>304</b><i>e </i>in addition to the start byte index <b>304</b><i>b </i>(initially 0 for the first segment), and either an end byte index <b>304</b><i>c </i>or a segment size. The server-side recognizer <b>120</b> may acknowledge that it has received a segment by transmitting back the end byte index of the segment, which should normally be equal to the start byte plus the segment size. The end byte index transmitted by the server may, however, be a lower value if the server could not read the entire audio segment.

The speech recognition client <b>140</b> then transfers the next segment starting where the server-side recognizer <b>120</b> left off, so that the new start byte index is equal to the end byte index returned by the recognizer <b>120</b>. This process is repeated for the entire speech stream <b>110</b>. If a message is lost (on the way to or from the server <b>118</b>), the speech recognition client <b>140</b> repeats the transfer. If the server-side recognizer <b>120</b> did not previously receive that speech segment, then the server-side recognizer <b>120</b> will simply process the new data. If, however, the recognizer <b>120</b> previously processed that segment (such as may occur if the results were lost on the way back to the client <b>106</b>), then the recognizer <b>120</b> may, for example, acknowledge receipt of the segment and drop it without processing it again.

For the control stream <b>112</b>, all control messages <b>402</b><i>a</i>-<i>c </i>may be resent to the server <b>118</b>, since each of the messages may contain an ID for the current session. In the case of the DecodeNext method, the speech recognition client <b>140</b> may pass, as part of the DecodeNext method, a running unique identifier to identify the current method call. The server <b>118</b> keeps track of those identifiers to determine whether the current message being received in the control stream <b>112</b> is new or whether it has already been received and processed. If the current message is new, then the recognizer <b>120</b> processes the message normally, as described above. If the current message was previously processed, then the recognizer <b>120</b> may re-deliver the previously-returned results instead of generating them again.

If one of the control messages <b>402</b><i>a</i>-<i>c </i>is sent to the server <b>118</b> and the server <b>118</b> does not acknowledge receipt of the control message, the client <b>140</b> may store the control message. When the client <b>140</b> has a second control message to send to the server <b>118</b>, the client <b>140</b> may send both the first (unacknowledged) control message and the second control message to the server <b>118</b>. The client <b>140</b> may alternatively achieve the same result by combining the state changes represented by the first and second control messages into a single control message, which the client <b>140</b> may then transmit to the server <b>140</b>. The client <b>140</b> may combine any number of control messages together into a single control message in this way until such messages are acknowledged by the server <b>118</b>. Similarly, the server <b>118</b> may combine speech recognition results which have not been acknowledged by the client <b>140</b> into individual results in the results stream <b>122</b> until such results are acknowledged by the client.

Among the advantages of the invention are one or more of the following. Embodiments of the present invention enable speech recognition to be distributed anywhere on the Internet, without requiring any special network. In particular, the techniques disclosed herein may operate over a one-way communication protocol, such as HTTP, thereby enabling operation even in restrictive environments in which clients are limited to engaging only in outbound (one-way) communications. As a result, embodiments of the present invention are broadly useful in conjunction with a wide variety of networks without requiring security to be sacrificed. Furthermore, the techniques disclosed herein may reuse existing web security mechanisms (such as SSL and, by extension, HTTPS) to provide secure communications between client <b>106</b> and server <b>118</b>.

As mentioned above, one common restriction imposed on clients is that they may only use a limited range of outbound ports to communicate with external servers. Embodiments of the present invention may be implemented in such systems by multiplexing the speech stream <b>110</b> and the control stream <b>112</b> into a single stream <b>114</b> that can be transmitted through a single port.

Furthermore, outgoing communication may be required to be encrypted. For example, clients often are allowed to use only the standard secure, encrypted HTTPS port (port <b>443</b>). Embodiments of the present invention can work over either a standard (unsecured) HTTP port or a secured HTTPS port for all of its communication needs—both audio transfer <b>110</b> and control flow <b>112</b>. As a result, the techniques disclosed herein may be used in conjunction with systems which allow clients to communicate using unsecured HTTP and systems which require or allow clients to communicate using secured HTTPS.

The techniques disclosed herein are also resilient to intermittent network failures because they employ a communications protocol in which messages are idempotent. This is particularly useful when embodiments of the present invention are used in conjunction with networks, such as WANs, in which network drops and spikes are common. Although such events may cause conventional server-side speech recognition systems to fail, they do not effect results produced by embodiments of the present invention (except possibly by increasing turnaround time).

Embodiments of the present invention enable speech <b>104</b> to be transmitted from client <b>106</b> to server <b>118</b> as fast as the network <b>116</b> will allow, even if the server <b>118</b> cannot process that speech continuously. Furthermore, the server-side recognizer <b>120</b> may process speech from the processing queue <b>124</b> as quickly as possible even when the network <b>116</b> cannot transmit the results and/or the application <b>108</b> is not ready to receive the results. These and other features of embodiments of the present invention enable speech and speech recognition results to be transmitted and processed as quickly as individual components of the system <b>100</b> will allow, such that problems with individual components of the system <b>100</b> have minimum impact on the performance of the other components of the system <b>100</b>.

Furthermore, embodiments of the present invention enable the server-side recognizer <b>120</b> to process speech as quickly as possible but without getting too far ahead of the client application <b>108</b>. As described above, the application <b>108</b> may use control messages in the control stream <b>112</b> to issue reconfiguration commands to the recognizer <b>120</b> which cause the recognizer <b>120</b> to reconfigure itself to recognize speech in the appropriate configuration state, and to temporarily halt recognition upon the occurrence of predetermined conditions so that the application <b>108</b> can reconfigure the state of the recognizer <b>120</b> appropriately. Such techniques enable speech recognition to be performed as quickly as possible without being performed using the wrong configuration state.

It is to be understood that although the invention has been described above in terms of particular embodiments, the foregoing embodiments are provided as illustrative only, and do not limit or define the scope of the invention. Various other embodiments, including but not limited to the following, are also within the scope of the claims. For example, elements and components described herein may be further divided into additional components or joined together to form fewer components for performing the same functions.

As described above, various methods performed by embodiments of the present invention may be performed in parallel with each other, in whole or in part. Those having ordinary skill in the art will appreciate how to perform particular portions of the methods disclosed herein to achieve the stated benefits, in various combinations.

The techniques described above may be implemented, for example, in hardware, software, firmware, or any combination thereof. The techniques described above may be implemented in one or more computer programs executing on a programmable computer including a processor, a storage medium readable by the processor (including, for example, volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Program code may be applied to input entered using the input device to perform the functions described and to generate output. The output may be provided to one or more output devices.

Each computer program within the scope of the claims below may be implemented in any programming language, such as assembly language, machine language, a high-level procedural programming language, or an object-oriented programming language. The programming language may, for example, be a compiled or interpreted programming language.

Each such computer program may be implemented in a computer program product tangibly embodied in a machine-readable storage device for execution by a computer processor. Method steps of the invention may be performed by a computer processor executing a program tangibly embodied on a computer-readable medium to perform functions of the invention by operating on input and generating output. Suitable processors include, by way of example, both general and special purpose microprocessors. Generally, the processor receives instructions and data from a read-only memory and/or a random access memory. Storage devices suitable for tangibly embodying computer program instructions include, for example, all forms of non-volatile memory, such as semiconductor memory devices, including EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROMs. Any of the foregoing may be supplemented by, or incorporated in, specially-designed ASICs (application-specific integrated circuits) or FPGAs (Field-Programmable Gate Arrays). A computer can generally also receive programs and data from a storage medium such as an internal disk (not shown) or a removable disk. These elements will also be found in a conventional desktop or workstation computer as well as other computers suitable for executing computer programs implementing the methods described herein, which may be used in conjunction with any digital print engine or marking engine, display monitor, or other raster output device capable of producing color or gray scale pixels on paper, film, display screen, or other output medium.