Hybrid, offline/online speech translation system

BACKGROUND

Speech-to-speech translation (STS) systems are usually delivered in one of two different forms: online over the Internet or offline embedded on a user's device (e.g., smartphone or other suitable computing device). The online version has the advantage that it can benefit from significant processing resources on a large server (the cloud), and provides a data feed to the service provider that makes improvements and customization possible. However, online processing requires continuing network connectivity, which cannot be guaranteed in all locations or is not desirable in some instances due to roaming costs or privacy/security concerns. As alternative deployment, speech-to-speech translators, such as Jibbigo speech translation apps, can be delivered as software running embedded locally on the smartphone itself, and no network connectivity is needed after the initial download of the translation application. Such offline embedded speech translation capability is the preferred deployment for many if not most practical situations where language support is needed, as networks may not be available, intermittent or too expensive. Most travelers experience such intermittent or absent connectivity, for example, during airline flights, remote geographic locations, buildings, or simply because data roaming is turned off to avoid the associated roaming charges while traveling in a foreign country.

The way such speech translation se ices or software are delivered also has implications to what extent the software can/must operate in a domaindependent or -independent manner and whether it can adapt to the user's context. STS systems will usually work rather well for a domain and not so well for another domain (domain-dependence) if they have been closely optimized and tuned to a specific domain of use, or they attempt domain-independence by working more or less equally well for all domains. Either solution limits performance for all specific situations.

A user commonly runs an online client program on his/her computing device. This device typically digitizes and possibly encodes speech, then transmits samples or coefficients over a communication line to a server. The server then performs the heavy computation speech recognition and/or translation and sends the result back to the user via a communication line, and the result is displayed on the user's device. Different online designs have been proposed that move different parts of a processing chain off to the server and do more or less computing work on the device. In speech recognition, translation and translation systems, the user's device can be as simple as just a microphone, or an analog to digital converter, or provide more complex functions such as noise suppression, encoding as coefficients, one or more speech recognition passes, or one or more language processing steps. An off-line design by contrast runs the entire application on the device itself as an embedded application. All computation is done on the device locally and no transmission between a client and a server is needed during use.

Typically, an online design has the advantage that it needs only a very simple client and thus an application can be run on a very simple computing device or mobile phone, while all the heavy computations and processing are done on a large computing server. For speech and machine translation this can mean that more advanced but computationally intensive algorithms can be used, and up-to-date background information can be used. It also has the advantage that the developer or operator of the service can maintain/improve the service or capability on the server, without requiring the user to download or upgrade new system versions.

The disadvantage of an online design is the fact that it critically depends on network connectivity. As a user moves and travels to remote locations, however, connectivity can be intermittent and/or very expensive (roaming), and thus in many ways unavailable. For speech and speech translation systems this requirement is frequently unacceptable. Unlike text or email transmissions, voice cannot permit a temporary lapse of connectivity as it cannot permit a corresponding interruption of the speech stream without loosing information or real-time performance. An online design must therefore ensure continuous, real-time transmission and thus continuous connectivity during use.

SUMMARY

In one general aspect, the present invention is directed to a hybrid speech translation system whereby a wireless-enabled client computing device (e.g., a smartphone or a tablet computer) can translate input word phrases (e.g., input speech utterances or input text) from one language to another locally, e.g., an "offline" mode, and also, in an "online" mode when there is wireless network connectivity, have a remote computer, e.g., a back-end speech translation server system, perform the translation and transmit it back to the client computing device via the wireless network for outputting by client computing device (e.g., audibly via a speaker and/or via a text display field). In various embodiments, the user of the client computing device can transition between modes or the transition can be automatic-and transparent to the user-based on user preferences or settings. Further, the back-end speech translation server system can adapt the various speech translation models used by the client computing device in the offline mode based on analysis of user data over time, to thereby configure the client computing device with scaled-down, yet more efficient and faster, models than the back-end speech translation server system, while still being adapted for the user's domain.

Embodiments according to the invention are in particular disclosed in the attached claims directed to a speech translation system and speech translation method, wherein any feature mentioned in one claim category, e.g. method, can be claimed in another claim category, e.g. system, as well. The dependencies or references back in the attached claims and the embodiments listed below are chosen for formal reasons only. However any subject matter resulting from a deliberate reference back to any previous claim or embodiment (in particular multiple dependencies) can be claimed as well, so that any combination of claims and the features thereof is disclosed and can be claimed regardless of the dependencies chosen in the attached claims. Any combination of features of the embodiments listed below is also disclosed, irrespective of the back references mentioned therein.

In an embodiment according to the invention, a speech translation system comprises: <ul id="ul0001" list-style="dash" compact="compact"><li>a back-end speech translation server system; and</li><li>a client computing device that is configured for communicating with the back-end speech translation server system via a wireless network, wherein the client computing device comprises: <ul id="ul0002" list-style="dash" compact="compact"><li>a microphone;</li><li>a processor connected to the microphone;</li><li>a memory connected to the processor that stores instructions to be executed by the processor; and</li><li>a speaker connected to the processor, wherein:</li><li>the client computing device is for outputting via the speaker a translation of input word phrases from a first language to a second language; and</li><li>the memory stores instructions such that: <ul id="ul0003" list-style="none" compact="compact"><li>in a first operating mode, when the processor executes the instructions, the processor translates the input word phrases to the second language for output to a user; and</li><li>in a second operating mode: <ul id="ul0004" list-style="dash" compact="compact"><li>the client computing device transmits to the back-end speech translation server system, via the wireless network, data regarding the input word phrases in the first language received by the client computing device;</li><li>the back-end speech translation server system determines the translation to the second language of the input word phrases in the first language based on the data received via the wireless network from the client computing device; and</li><li>the back-end speech translation system transmits data regarding the translation to the second language of the input word phrases in the first language to the client computing device via the wireless network such that the client computing device outputs the translation to the second language of the input word phrases in the first language.</li></ul></li></ul></li></ul></li></ul>

The client computing device may have a user interface that permits a user to switch between the first operating mode and the second operating mode.

The client computing device may automatically select whether to use the first operating mode or the second operating mode based on a connection status for the wireless network.

Alternatively, the client computing device may automatically select whether to use the first operating mode or the second operating mode based on a user preference setting of the user for the client computing device.

In a further embodiment according to the invention, the input word phrases are input to the client computing through one of: <ul id="ul0005" list-style="dash" compact="compact"><li>input speech utterances captured by the microphone of the client computing device; or text input via a text input field on a user interface of the client computing device.</li></ul>

The client computing device may output the translations audibly via the speaker.

In the speech translation system of the present invention the client computing device may store in memory a local acoustic model, a local language model, a local translation model and a local speech synthesis model for, in the first operating mode, recognizing the speech utterances in the first language and translating the recognized speech utterances to the second language for output via the speaker of the client computing device.

The back-end speech translation server system may comprise a backend acoustic model, a back-end language model, a back-end translation model and a back-end speech synthesis model for, in the second operating mode, determining the translation to the second language of the speech utterances in the first language based on the data received via the wireless network from the client computing device.

Preferably, the local acoustic model may be different from the backend acoustic model; the local language model may be different from the back-end language model; the local translation model may be different from the back-end translation model; and the local speech synthesis model may be different from the back-end speech synthesis model.

Furthermore, the back-end speech translation server system may be programmed to: monitor over time speech utterances received by the client computing device for translation from the first language to the second language and update at least one of the local acoustic model, the local language model, the local translation model and the local speech synthesis model of the client computing device based on the monitoring over time of speech utterances received by the client computing device for translation from the first language to the second language, wherein updates to the at least one of the local acoustic model, the local language model, the local translation model and the local speech synthesis model of the client computing device are transmitted from the back- end speech translation server system to the client computing device via the wireless network.

The local acoustic model, the local language model, the local translation model and the local speech synthesis model of the client computing device can be updated based on analysis of translation queries by the user.

The client computing device may comprise a GPS system for determining a location of the client computing device.

The back-end speech translation server system may further be programmed to update at least one of the local acoustic model, the local language model, the local translation model and the local speech synthesis model of the client computing device based on the location of the client computing device, wherein updates to the at least one of local acoustic model, the local language model, the local translation model and the local speech synthesis model of the client computing device are transmitted from the back end speech translation server system to the client computing device via the wireless network.

Moreover, the back-end speech translation server system may be one of a plurality of a back-end speech translation server systems, and the client computing device may be configured for communicating with the each of the plurality of back-end speech translation server systems via a wireless network.

In the second operating mode, each of the plurality of back end speech translation server systems may be for determining a translation to the second language of the speech utterances in the first language based on the data received via the wireless network from the client computing device; and one of the plurality of back-end speech translation server systems may select one of the translations from the plurality of back-end speech translation server systems for transmitting to the client computing device.

Alternatively, one of the plurality of back end speech translation server systems merges two or more of the translations from the plurality of back-end speech translation server systems to generate a merged translation for transmitting to the client computing device.

According to another aspect of the present invention, a speech translation method is disclosed, the speech translation method comprising: <ul id="ul0006" list-style="dash" compact="compact"><li>in a first operating mode: <ul id="ul0007" list-style="dash" compact="compact"><li>receiving by a client computing device a first input word phrase in a first language;</li><li>translating by the client computing device the first input word phrase to a second language; and</li><li>outputting by the client computing device the first input word phrase in the second language;</li></ul></li><li>transitioning by the client computing device from the first operating mode to the second operating mode;</li><li>in the second operating mode: <ul id="ul0008" list-style="dash" compact="compact"><li>receiving by a client computing device a second input word phrase in a first language;</li><li>transmitting, by the client computing device, via a wireless network, data regarding the second input word phrase to a back-end speech translation server system;</li><li>receiving, by the client computing device, from the back-end speech translation server system via the wireless network, data regarding a translation by the back-end speech translation server system of the second input word phrase from the first language to the second language; and</li><li>outputting by the client computing device the second input word phrase in the second language.</li></ul></li></ul>

In a further embodiment of the invention, the client computing device stores in memory a local acoustic model, a local language model, a local translation model and a local speech synthesis model, for, in the first operating mode, recognizing the input speech utterances in the first language and translating the recognized input speech utterances to the second language for output via the speaker and the back-end speech translation server system preferably comprises a back-end acoustic model, a back-end language model, a back-end translation model and a back-end speech synthesis model for, in the second operating mode, determining the translation to the second language of the input speech utterances in the first language based on the data received via the wireless network from the client computing device.

The method may further comprise the steps of <ul id="ul0009" list-style="dash" compact="compact"><li>monitoring, by the back-end speech translation server system, over time speech utterances received by the client computing device for translation from the first language to the second language; and</li><li>updating, by the back-end speech translation server system, at least one of the local acoustic model, the local language model, the local translation model and the local speech synthesis model of the client computing device based on the monitoring over time of speech utterances received by the client computing device for translation from the first language to the second language, wherein updates to the at least one of the local acoustic model, the local language model, the local translation model and the local speech synthesis model of the client computing device are transmitted from the back-end speech translation server system to the client computing device via the wireless network.</li></ul>

The speech translation method may comprise a step of downloading by the client computing device application software for a language translation pair that comprises the first and second languages.

The step of downloading the application software for the language translation pair may comprise downloading the application software for the language translation pair when suitable connectivity between the client computing device and the back-end speech translation server system is available via the wireless network.

In a further embodiment of the invention, the speech translation method may comprise: <ul id="ul0010" list-style="dash" compact="compact"><li>determining by the client computing device a location of the client computing device; and</li><li>downloading by the client computing device the application software for the language translation pair based on the determined location of the client computing device and when suitable connectivity between the client computing device and the back-end speech translation server system is available via the wireless network.</li></ul>

Preferably, the client computing device may comprise a graphical user interface have a first language display section and a second language display section that are simultaneously displayed; and <ul id="ul0011" list-style="dash" compact="compact"><li>each of the first and second language display sections may comprise a user-accessible listing of a plurality of languages.</li></ul>

The method further may comprise the step of receiving by the client computing device via the graphical user interface a selection of the first language from the listing in the first language display section and the second language in the second language display section, such that the client computing device is thereby configured to translate the input speech utterances from the first language to the second languages.

Languages available in the first operating mode may be visually designated differently in the first and second language display sections of the graphical user interface from languages that are unavailable in the first operating mode.

The step of transitioning by the client computing device from the first operating mode to the second operating mode is responsive to an input via a user interface of the client computing device to transition from the first mode to the second mode.

These and other benefits of the present invention will be apparent from the description to follow.

FIGURES

Various embodiments of the present invention are described herein by way of example in connection with the following figures, wherein <dl id="dl0001" compact="compact"><dt>Figures 1 and 8</dt><dd>are block diagrams of the hybrid speech translation system according to various embodiments of the present invention;</dd><dt>Figures 2A-2B and 4A-4B</dt><dd>provide sample screen shots of graphical user interfaces through which a user of the client computing device may select a desired language translation pair, and transition from offline mode to online mode and vice versa, according to various embodiments of the present invention;</dd><dt>Figure 3</dt><dd>is a block diagram of the client computing device according to various embodiments of the present invention;</dd><dt>Figure 5</dt><dd>is a flow chart diagramming a process of the client computing device for automatically transitioning between the offline and online modes according to various embodiments of the present invention;</dd><dt>Figure 6</dt><dd>is a flow chart diagramming a process of updating various models and table of the offline speech translation of the client computing device according to various embodiments of the present invention; and</dd><dt>Figure 7</dt><dd>is a flow chart diagramming the speech translation processing in the offline and online modes according to various embodiments of the present invention.</dd></dl>

DESCRIPTION

The present invention is directed generally to a speech translation system where a wireless-enabled client computing device (e.g., a smartphone or a tablet computer) has both online (e.g., speech translation processing by a remote computer system) and offline (e.g., speech translation processing embedded on the client computing device) capabilities. <figref idref="f0001">Figure 1</figref> is a block diagram of an example embodiment of the speech translation system 10 according to various embodiments of the present invention. As illustrated in <figref idref="f0001">Figure 1</figref>, the system 10 comprises a wireless-enabled client computing device 12, a wireless network 14, a data communications network 15 (e.g., the Internet), a back-end speech translation server system 16, and an application store server system ("app store") 18. The client computing device 12 is used by a user to translate speech utterances received by the client computing device 12 in a first language to a second (or even other) languages. The client computing device 12 may be any suitable computing device, such as a desktop or laptop computer, but is more preferably a mobile, handheld computing device, such as smartphone or tablet computer. More details regarding an example client computing device 12 are described below in connection with <figref idref="f0003">Figure 3</figref>.

The client computing device 12 also preferably is capable of wireless data communication (i.e., the client computing device 12 is "wireless-enabled") via the wireless network 14. The wireless network 14 may be any suitable wireless network, such as a wireless LAN (WLAN) that uses the IEEE 802.11 WLAN standards, such as a WiFi network. The wireless network 14 may also comprise a mobile telecommunications network, such as a 3G or 4G LTE mobile phone mobile communication network, although other suitable wireless networks may also be used. The wireless network 14 preferably provides connection to the Internet 15, such as through an access point or base station. The back-end speech translation server system 16 and the app store 18 are connected to the Internet 15 and, hence, in communication with the client computing device 12 via the wireless network 14.

As described herein, the client computing device 12 is provisioned with software (including models) that permits the client computing device 12 to perform offline speech translation or to perform online speech translation, with the back-end speech translation server system 16 providing the computationally intensive speech recognition and/or translation processing steps. The back-end speech translation server system 16 may therefore comprise one or more networked computer servers that perform speech translation based on data received from the client computing device 12 via the wireless network 14. The back-end speech translation server system 16 may therefore comprise, for example: an automatic speech recognition (ASR) module 20 for recognizing the speech in the first language in the inputted speech utterance data; a machine translation (MT) module 22 that converts/translates the recognized in the first language to the selected second language; and a speech synthesis module 24 that synthesizes the translation into the second language to signals for an audible output in the second language. The ASR module 20 may employ, for example, (i) a language model that contains a large list of words and their probability of occurrence in a given sequence, and (ii) an acoustic model that contains a statistical representation of the distinct sounds that make up each word in the language model. The MT module may employ, for example, appropriate translation tables (or models) and language models. The speech synthesis module 24 may employ an appropriate speech synthesis model(s). Similarly, the speech translation software for the client computing device 12 may comprise an ASR module (with language and acoustic models), a MT module (with translation tables/models and language models), and a speech synthesis module (with speech synthesis models). More details for the ASR, MT and synthesis modules (or engines), for the online and offline modes, may be found in U.S. Patents 8,090,570 and 8,204,739, which are incorporated herein by reference in their entirety.

The user of the client computing device 12 may purchase speech translation software (or application or "app") via the app store 18. In various embodiments, an online version of the translation app, where the back-end speech translation server system 16 performs the bulk of the speech translation processing assuming a connection to the client computing device 12, is available for free downloading via the app store 18. The online translation app provides the client computing device 12 with a user interface, the capability of collecting input word phrases for translations, such as speech utterances (captured by a microphone on the client computing device 12) or text (via a text field provided by the user interface), and outputting the translation (via speakers of the client computing device 12 and/or textually via the user interface). In such an embodiment, the client computing device 12 may transmit to the back-end speech translation server system 16, via the wireless network 14, data regarding the input phrase to be translated, in the first language, recorded by a microphone of the client computing device 12 or input via the text input field, as the data including for example digital, encoded samples or feature vectors after pre-processing the input speech. Based on the received input data, the back-end speech translation server system 16 translates the speech utterances to the selected second language, and transmits data representative of the translation back to the client computing device 12 via the wireless network 14 for processing, synthesis, and audible outputting via the speakers of the client computing device 12.

The speech translation app may also be operated in an offline mode, where the client computing device 12 performs the speech translation locally, without connection to the back-end speech translation server system 16. In various embodiments, the user of the client computing device 12, while having connectivity to the wireless network 14, downloads the offline app software for a language pair of choice (e.g., English-Spanish, etc.), so that the offline system can run with network connectivity ceases. <figref idref="f0002">Figures 2A-B</figref> illustrate sample user interfaces for display on the client computing device 12 using the app that allows the user to select the desired language pair. The illustrated sample user interface also show that the user can select online or offline mode via a user control. For example, in <figref idref="f0002">Figure 2A</figref> the user switched the user control40 to online, as indicated by the cloud and/or the description "Online Translators"; in <figref idref="f0002">Figure 2B</figref> the user switched the user control 40 to offline, as indicated by the diagonal line through the cloud and the description "Offline Translators." In the examples of <figref idref="f0002">Figures 2A-B</figref>, the user can scroll the languages in the first and second columns 42, 44 up and down (much like a scroll wheel) until the user gets a desired language pair, indicated by the languages in the first and second columns at the highlighted selection area 46. In the example of <figref idref="f0002">Figure 2A</figref>, the selected language pair is English (International version) and Spanish (Mexico version). In the example of <figref idref="f0002">Figure 2B</figref>, the selected language pair is English (International version) and Spanish (Spain version).

In online mode, the user may access any language pair that is offered. This may be indicated to the user by showing in color the icons (e.g., nationality flags) for the available languages in the two columns 42, 44, as shown in <figref idref="f0002">Figure 2A</figref>. The user may then scroll the two columns up/down so that the desired language pair is show in the selection area 46. When wireless network connectivity is not available (such as because it was switched off by the user via user control40 or automatically switched off, as described below), only previously installed language pair on the client computing device 12 are available in various embodiments. Similarly, for the available languages for the offline mode, as shown in <figref idref="f0002">Figure 2B</figref>, may be indicated to the user by showing in color the icons (e.g., flags) for the installed languages in the two columns 42, 44, while showing all non-installed languages in grey.

<figref idref="f0007">Figure 7</figref> is a flowchart illustrating the hybrid online/offline process according to various embodiments. The client computing device 12 (e.g., the microphone thereof) captures an input voice utterance in the first language at step 70. If, step 72, the online mode is being used, at step 74 the client computing device 12 transmits, via the wireless network 14, data (e.g., samples or coefficients of the input speech utterance) regarding the input speech utterance to the back-end speech translation server system 16, which at step 76 translates the utterance to the second language. At step 77 the back-end speech translation server system 16 transmits data for the translation back to the client computing device 12 via the wireless network 14 so that, at step 79, the client computing device 12 (e.g., the speaker(s) thereof) may audibly output the translation in the second language of the input utterance. If at step 72 the offline mode is used, at step 78 the client computing device 12, executing the downloaded offline speech translation software stored in memory, translates the utterance to the second language, and it is output at step 79.

<figref idref="f0003">Figure 3</figref> is a block diagram of the client computing device 12 according to various embodiments. As shown in the example of <figref idref="f0003">Figure 3</figref>, the device 12 may comprise multiple processors 202, 204. A baseband processor 202 may handle communication over a mobile telecommunications network (e.g., cellular network) according to any suitable communications technology (e.g., 3G, 4G, etc.). The baseband processor 202 may comprise dedicated random access memory (RAM) 214. In various embodiments, the baseband processor 202 may be in communication with a transceiver 206. The transceiver 206 may, subsequently, be in communications with one or more power amplifiers 208 and an antenna 210. Outgoing signals for the mobile telecommunications network may be processed at baseband by the baseband processor 202 and provided to the transceiver 206. The transceiver 206 and/or the baseband processor 206 may modulate the outgoing signal to a carrier frequency. One or more of the amplifiers 208 may amplify the outgoing signal, which may be subsequently transmitted via antenna 210. Incoming signals for the mobile telecommunications network may be received by the antenna 210, amplified by one or more of the amplifiers 208 and provided to the transceiver 206. Either the transceiver 206 or the baseband processor 202 may demodulate the incoming signal to baseband.

An applications processor 204 may execute an operating system as well as software applications (e.g., downloaded via the app store 18), including the offline and online speech recognition and/or translation functionalities described herein. The applications processor 204 may also execute the software for the touch screen interface 232. The applications processor 204 may also be in communications with applications RAM 212, and non-volatile data storage (e.g., ROM) 216. The RAM 212 may store, for execution by the processor 204, among other things, the application software downloaded via the app store 18 for the offline and online speech translation, including the necessary automatic speech recognition, machine translation, and speech synthesis modules for the offline processing, and the software to communicate with the back-end speech translation server system 16 for the online processing.

The applications processor 204 may additionally be in communication with other hardware devices such as a combination WI-FI/BLUETOOTH transceiver 218. The WI- FI/BLUETOOTH transceiver 218 may handle radio frequency (RF) communication with a LAN (e.g., according to the WI-FI standard, or any suitable standard) or direct RF communications between the device 200 and another wireless device (e.g., according to the BLUETOOTH standard or any suitable standard). In various embodiments, the device 200 may also comprise a global positioning system (GPS) 222 that is in communication with a satellite-based GPS system via a GPS antenna 223 for providing the application processor 204 information describing the geographic location of the device 12. The touch screen 232 may both provide output to the user of the device 12 in visual form and receive input from the user. The input may be in the form of signals representing screen-touches by the user. An audio codec module 224 may provide hardware and/or software for decoding and playing audio signals. In some embodiments, the codec 224 may also comprise a digital-to-analog converter. Audio output signals may be provided to the device speaker 16 and/or a jack (not shown) that may receive a set of headphones and/or speakers for playing the audio output signal. Audio input signals may be provided via the device microphone(s) 18. The device may also comprise a digital camera 240.

Various other sensors may be included in certain embodiments. A magnetic sensor 226 may sense magnetic fields near the device. For example, the magnetic sensor 226 may be used by various apps and/or system functionality to implement a compass. An accelerometer 228 and gyroscope 230 may provide data describing movement of the device. For example, data from the accelerometer 228 and gyroscope 230 may be used to orient the display of the touch screen 232 (e.g., portrait versus landscape). The device 200 may be powered by a battery 234, which may, in turn, be managed by a power management integrated circuit (PMIC) 236. An I/0 transceiver 238 may manage wired communications between the device and other devices, for example, according to the Universal Serial Bus (USB) or any other suitable standard. A connector 239 may facilitate wired connections. In some embodiments, connections via the connector 239 and I/0 transceiver 238 may provide power for charging the battery 234.

As described above, in various embodiments the user can switch between online and offline modes, such as by activation of the user control 40 as shown in the examples of <figref idref="f0002">Figures 2A and 2B</figref>. Online processing preferably provides larger vocabularies in the language models than online processing, but online processing may provide enhanced user privacy and security as data for the user input utterances are not transmitted via the wireless network 14 and the Internet. The translation app software may also allow the client computing device 12 to automatically switch between online and offline modes according to various embodiments. For example, the user may provision the settings for the app so that if the wireless network 14 is available (e.g., network connectivity with suitable data rates/connection speed), the online operating mode is used; otherwise the offline operating mode. Accordingly, for such an embodiment, as shown in the example flow chart of <figref idref="f0005">Figure 5</figref>, if the client computing device 14 is in a wireless communication mode (e.g., WiFi or cell phone network, such as 3G or 4G) (step 50), the processor of the client computing device 12, executing the app software stored in the memory, may check the data rate/connection speed for the WiFi network (step 52), and if it is above a threshold data rate/connection speed, the online mode is used (step 54); otherwise the offline mode is used (step 56). This way, the user has automated, continuous translation capability and the use of either offline or online modes is transparent to the user. The client computing device 12 may visually display which of the modes is being used at a given time (such as with the cloud and no-cloud icons described above).

In other embodiments, the processor of the client computing device 12, executing the app software stored in the memory, may automatically switch between the online and offline operating modes based on other factors, such as: cost (e.g., if roaming charges apply, or if there is no network connectivity, the offline operating mode is used; otherwise online mode is used); quality (e.g., better translation, acoustic or language models, e.g., usage of offline speaker specific, or online general domain independent models); location (e.g., based on GPS coordinates as determined by the GPS system 222); privacy (e.g., use the online mode only if a secure wireless network is available); and/or time (e.g., specified modes during specified times of day). In various embodiments, the user of the client computing device 14 may configure the app through its settings to establish the applicable criteria for automatically transitioning between the online and offline operating modes. For example, the user could select, according to various embodiments, to: always use offline mode (in which case online mode is never used); prefer the fastest service (in which case online mode is used only if the connection speed of the wireless network exceeds a threshold); the most accurate translations (in which case online mode is used whenever available); limit costs (in which case, for example, offline mode is used when roaming charges are applicable). Such user preferences may be influenced by privacy (data transmission), quality (size and performance of speech translation models), or cost (data roaming) considerations.

Another aspect of the hybrid offline/online translation system is the language pairs made available on the client computing device 12 for offline mode. Because of memory size limitations for the client computing device 12, in most cases it is impractical to download all available language pairs to the client computing device 12. As such, the user of the client computing device 12 preferably downloads to the client computing device 12 only selected language pairs that the user needs. For example, in various embodiments, the user may chooses or purchases available offline language pairs through the app store 18. In various embodiments, the user could purchase a package that comprises several language pairs (such as languages for a geographic area, such as Europe, Southeast Asia, etc., or versions of the same language, such as Mexico and Spain versions of Spanish, Portugal and Brazil versions of the Portuguese, etc.), in which case the software apps for all of the language pairs in the package are available for downloading to the client computing device 18. For example, <figref idref="f0004">Figure 4A</figref> shows a sample screen shot where a user may choose to purchase various translation language pairs; and <figref idref="f0004">Figure 4B</figref> shows a sample screen shot for a package of the translation language pairs (in this example, a world package). If the user wishes to remove a language pair from the client computing device to same memory, the user is able, in various embodiments, to remove that pair and the models that correspond to it, without losing their availability. That is, the user may download the models again at a later date.

In one embodiment, the choice to download a language pair is left to the user, and the user selects the pairs to be installed on the client computing device for offline translation. If the user desired to install a selected language pair(s) but there is not satisfactory network connectivity, the client computing device stores and request and issues a reminder message to the user to download the pair when network connectivity is next available. The reminder message queries the user whether he/she wishes to download the offline version of the selected language pair(s) and proceeds with the download if confirmed by the user.

In another embodiment, the client computing device 12 itself may manage the offline translation pairs for the user. For example, the client computing device 12 may maintain data regarding languages used around the world, and may automatically download an offline language pair that is relevant to the location of the user. For example, if the GPS system 22 shows that the user is in Spain, the Spain version of Spanish may be downloaded, and so on. Also, the offline language pairs could be automatically downloaded based on, for example, calendar data for the user (e.g., a trip) or web search data that indicates a user's interest in or plain to travel to a particular region of the world.

Access to a user's location (e.g., based on GPS data) and/or interests (e.g., based on internet search data and/or speech translation queries) also offers customization of the speech translation system in its language behavior. Certain words, location names and food types may be preferred. Particularly, names (location names, person names) will likely be more or less relevant and likely dependent on location (e.g., Kawasaki and Yamamoto for Japan, versus Martinez or Gonzales for Spain, etc.). The modeling parameters of the recognition and translation models, most importantly their vocabularies and likely translations, may therefore be adjusted based on location and interests of the user. In an online mode, this can all be done dynamically during use, using established adaptation algorithms. But in an offline system, not all words should be stored and memory must be conserved to achieve efficiency on a mobile device. The system may therefore, in various embodiments, download customized parametric models even for offline/embedded systems from the back-end speech translation system 16, when network connectivity is available, and swap in and out vocabulary items, language models and modified probabilistic acoustic parameters.

The most memory intensive aspects of a speech translation system are typically given by the translation tables and language models of a machine translation engine, the acoustic and language models of the recognition engine, and the voice parameters of a speech synthesis engine. To reduce the size of the models for the offline translation app localized on the client computing device 12, different techniques depending on the type of the model can be used. Models having probabilities as model parameters such as acoustic models and language models can be reduced by quantizing the value range of the probabilities so that the value range can be mapped from a continuous to a discrete space with a fixed number of value points only. Depending on the quantization factor, the storage requirements can be reduced to a byte or a few bits only. Models storing phrases of words, such as translation tables and language models, can use efficiently implemented storing techniques such as prefix trees. In addition, memory- mapping techniques can be used which load only small parts of the models dynamically on demand into the RAM 212/214, while unneeded parts remain untouched in the non volatile storage 216.

Another, more sophisticated approach for reducing language models and/or translation models to a given size so that they run on an offline client computing device is to use special trimming/ expansion heuristics that either eliminates vocabularies and word N-grams or expanding a base model by adding additional information. Elimination may be done in an opportunistic manner so that a user's most likely words and expressions are still represented despite the resource limitations, e.g. by limiting the vocabulary to a user-specific subset only and selecting only those parts of a general model that is covered by this vocabulary or by automatically collecting user-specific information from the queries of the user and selecting those parts of the general model that are closely related to the user's queries. Expansion on the other hand may be done by selecting only user specific - e.g. based on the users speaking style - and/or domain specific - e.g. for tourism usage or humanitarian usage and/or situation specific - e.g. based on the GPS location - and/or general, not related to any of the above specific information on the server, transferring this (delta) information only from the server to the device and applying this information to a base model stored on the device.

For example, with reference to the flowchart of <figref idref="f0006">Figure 6</figref>, at step 60 the back-end speech translation server system 16 may analyze the user data to determine, at step 62, whether the user's offline language model and/or translation table should be updated to, for example, eliminate seldom used words or expressions, while maintaining commonly used words and expressions of the user or extracting commonly used translations and expressions on the server and applying them to a base model stored on the client computing device. As mentioned above, the back-end speech translation server system 16 may analyze the user's translation queries (e.g., expressions to be translated) and/or internet browsing history, to determine the words and expressions commonly (and not commonly) used. As such, in various embodiments, the user's translation queries in offline mode may be saved and stored by the client computing device 12, and uploaded to the back-end speech translation server system 16 during a period of network connectivity so that they can be analyzed by the back-end speech translation server system 16. Similarly, the user's internet browsing history (e.g. cookie data) may be uploaded to the back- end speech translation server system 16 during a period of network connectivity so that they can be analyzed by the back-end speech translation server system 16 to determine the words and expressions commonly (and not commonly) used by the user. If, through its analysis of the user data, the back-end speech translation server system 16 determines that the language models and/or translation tables of the client computing device's offline processing software should be updated, the updated software (e.g. models) are downloaded to the client computing device (e.g., from the back-end speech translation server system 16) at step 64. Instead of downloading the full models, also downloading just the (delta) information that is required to update the model on the client computing device is possible.

Similarly, user specific information might also be useful for reducing the size of the acoustic model by, for example, substituting a more general acoustic model by a smaller user- specific one. Depending on the amount of user-specific data, this can be achieved, for example, by either using acoustic model adaptation techniques such as MLLR or fully retraining the acoustic model using the new data in addition. Thus, for example, referring again to <figref idref="f0006">Figure 6</figref>, if at step 66 the back-end speech translation server system 16 determines that the offline-mode acoustic model for the user's client computing device 12 should be modified based on the analysis of the user data, the updated software (e.g., acoustic model) is downloaded to the client computing device (e.g., from the back-end speech translation server system 16) at step 68.

In a speech-to-speech translation system the most speed-limiting element are the speech recognition algorithms, as they perform searches over many acoustic hypotheses and many time slices of the speech signal. The speed of the algorithmic searches is affected predominantly by the size of the acoustic model set. In order to maintain the speed of the offline system when performing speech-to-speech translation on the client computing device 12, several techniques can be used. For example, in one embodiment, depending on the size of the model, look-up tables can be used for computing the Mahalanobis distances between the model and input speech instead of computing the distances on demand. In addition, Gaussian selection techniques may be used in the offline mode to reduce the overall amount of model parameters that need to be evaluated. As soon as user specific information is available, smaller, more efficient user specific models can be used instead, as described above in connection with <figref idref="f0006">Figure 6</figref>.

In addition, according to various embodiments, during online mode, the back-end speech translation system 16 may use and combine multiple speech recognition and translation engines (modules). These backend engines can be provided by the same speech translation provider and run on the same server, for example, or in other embodiments could be provided by independent speech translation providers at different locations, as illustrated in the example of <figref idref="f0008">Figure 8</figref>, which shows three separate and independent back-end speech translation systems 16, although in other embodiments fewer or more back-end speech translation systems 16 could be used. In such an embodiment, each of the back-end speech translation systems 16 may be requested to perform the translation simultaneously through the Internet 15. In such an embodiment, the back-end speech translation systems 16 are in communication (e.g., through the Internet 15) and one of the back-end speech translation systems 16 either chooses the better of the translations or combines them. To decide between multiple systems/translation and/or on how strongly to weight either system in combination, confidence measures in ASR and confidence measures for MT may be used. Such confidence measures are used to determine the reliability of an ASR or an MT hypothesis. If two or more ASR or MT engines are to be merged in such an embodiment, system combinations may be used, such as "ROVER" methods of combining ASR outputs (see e.g., <nplcit id="ncit0001" npl-type="s"><text>J. G. Fiscus, "A post-processing system to yield reduced error word rates: Recognizer output voting error reduction (ROVER)," IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 347-354, 1997</text></nplcit>), cross-adapting one system against the other, or MT system combination techniques (see, e.g., <nplcit id="ncit0002" npl-type="s"><text>Rosti et al., "Combining Outputs from Multiple Machine Translation Systems," Proc. of NAACL HLT, pp. 228-235, 2007</text></nplcit> and <nplcit id="ncit0003" npl-type="s"><text>K. Heafield et al., "Combining Machine Translation Output with Open Source," Prague Bulletin of Mathematical Linguistics, No. 93, pp. 27-36, 2010</text></nplcit>). In such an embodiment, the selected and combined hypotheses can compete in the backend to produce the best output for the user. Once this has been done in online mode, the system will remember the best choice arrived in this manner for introduction in the offline system. For learning of the offline system, the combined online systems 16 may retain the recognition hypotheses of multiple ASR engines and/or translation hypotheses from multiple MT engines in memory and uses the combination, or the better of these hypotheses, to adapt or to train new offline systems. Such retrained or adapted systems may be subsequently swapped back in on the offline systems, when the wireless network is available.

In one general respect, therefore, the present invention is directed to speech translation systems and methods for hybrid, offline and online speech translation. According to various embodiments, the system may comprise a back-end speech translation server system and a client computing device that is configured for communicating with the back-end speech translation server system via a wireless network. The client computing device may comprise a microphone, a processor connected to the microphone, a memory connected to the processor that stores instructions to be executed by the processor, a speaker connected to the processor. The client computing device is for outputting, e.g., via the speaker or a text display field, a translation of input word phrases for translation (e.g., speech utterances or input text) from a first language to a second language. The memory stores instructions such that, in a first operating mode (offline mode), when the processor executes the instructions, the processor translates the input word phrases to the second language for output (e.g., via the speaker). In a second operating mode (online mode): (i) the client computing device transmits to the back-end speech translation server system, via the wireless network, data regarding the input word phrases in the first language received by the microphone; (ii) the back-end speech translation server system determines the translation to the second language of the input word phrases in the first language based on the data received via the wireless network from the client computing device; and (iii) the back-end speech translation system transmits data regarding the translation to the second language of the input word phrases in the first language to the client computing device via the wireless network such that the client computing device outputs the translation to the second language of the speech utterances in the first language.

According to various implementations, the client computing device has a user interface that permits a user to switch between the first operating mode and the second operating mode. Alternatively, the client computing device automatically selects whether to use the first operating mode or the second operating mode based on a connection a user preference setting of the user for the client computing device. In addition, the client computing device may store in memory a local acoustic model, a local language model, a local translation model and a local speech synthesis model, for, in the first operating mode, recognizing the speech utterances in the first language and translating the recognized speech utterances to the second language for output via the speaker. Also, the back-end speech translation server system comprises a back-end acoustic model, a back-end language model, a back-end translation model and a back-end speech synthesis model for, in the second operating mode, determining the translation to the second language of the speech utterances in the first language based on the data received via the wireless network from the client computing device. The local models are different from the back-end models (e.g., a subset or other variation).

In addition, the back-end speech translation server system may be programmed to: (i) monitor over time speech utterances received by the client computing device for translation from the first language to the second language; and (ii) update at least one of the local acoustic model, the local language model, the local translation model and the local speech synthesis model of the client computing device based on the monitoring over time of speech utterances received by the client computing device for translation from the first language to the second language. The client computing device may also comprise a GPS system for determining a location of the client computing device. In such an embodiment, the back-end speech translation server system may also be programmed to update at least one of the local acoustic model, the local language model, the local translation model and the local speech synthesis model of the client computing device based on the location of the client computing device. Any such updates to the at least one of the models of the client computing device may be transmitted from the back-end speech translation server system to the client computing device via the wireless network.

Additionally, the client computing device may be configured to download application software (including the models) for a language translation pair that comprises the first and second languages, particularly when suitable connectivity between the client computing device and the back-end speech translation server system is available via the wireless network. Also, for embodiments where the client computing device comprises a GPS system, the client computing device may be configured to download the application software for the language translation pair based on the determined location of the client computing device and when suitable connectivity between the client computing device and the back-end speech translation server system is available via the wireless network.

Furthermore, the client computing device may comprise a graphical user interface having a first language display section and a second language display section that are simultaneously displayed. Each of the first and second language display sections may comprise a user-accessible listing of a plurality of languages, such that upon a user of the client computing device selecting the first language from the listing in the first language display section and the second language in the second language display section, the client computing device is thereby configured to translate the input speech utterances from the first language to the second languages. Languages available in the first (offline) operating mode may be designated differently in the first and second language display sections from languages that are unavailable in the first operating mode.

In addition, in various embodiments, the back-end speech translation server system is one of a plurality of a back-end speech translation server systems, and the client computing device is configured for communicating with the each of the plurality of back-end speech translation server systems via a wireless network. In the second (online) operating mode, each of the plurality of back-end speech translation server systems determines a translation to the second language of the input word phrases in the first language based on the data received via the wireless network from the client computing device. In such circumstances, one of the plurality of back-end speech translation server systems selects one of the translations from the plurality of back-end speech translation server systems for transmitting to the client computing device, or two or more of the translations from the plurality of back-end speech translation server systems are merged to generate a merged translation for transmitting to the client computing device.

In one general aspect, the speech translation method comprises, in a first (offline) operating mode: (i) receiving by a client computing device a first input word phrase in a first language; (ii) translating by the client computing device the first input word phrase to a second language; and (iii) outputting by the client computing device the first speech utterance in the second language (e.g., audibly via a speaker and/or visually via a text display field). The method further comprises transitioning by the client computing device from the first operating mode to the second operating mode, and then, in the second (online) operating mode: (iv) receiving by a client computing device a second input word phrase in a first language; (v) transmitting, by the client computing device, via a wireless network, data regarding the second input word phrase to a back-end speech translation server system; and (vi) receiving, by the client computing device, from the back-end speech translation server system via the wireless network, data regarding a translation by the back-end speech translation server system of the second input word phrase from the first language to the second language; and outputting by the client computing device the first speech utterance in the second language.

It will be apparent to one of ordinary skill in the art that at least some of the embodiments described herein may be implemented in many different embodiments of software, firmware, and/or hardware. The software and firmware code may be executed by a processor circuit or any other similar computing device. The software code or specialized control hardware that may be used to implement embodiments is not limiting. For example, embodiments described herein may be implemented in computer software using any suitable computer software language type, using, for example, conventional or object-oriented techniques. Such software may be stored on any type of suitable computer-readable medium or media, such as, for example, a magnetic or optical storage medium. The operation and behavior of the embodiments may be described without specific reference to specific software code or specialized hardware components. The absence of such specific references is feasible, because it is clearly understood that artisans of ordinary skill would be able to design software and control hardware to implement the embodiments based on the present description with no more than reasonable effort and without undue experimentation.

Moreover, the processes associated with the present embodiments may be executed by programmable equipment, such as computers or computer systems, mobile devices, smartphones and/or processors. Software that may cause programmable equipment to execute processes may be stored in any storage device, such as, for example, a computer system (nonvolatile) memory, RAM, ROM, Flash Memory, etc. Furthermore, at least some of the processes may be programmed when the computer system is manufactured or stored on various types of computer readable media.

A "computer," "computer system," "host," "server," or "processor" may be, for example and without limitation, a processor, microcomputer, minicomputer, server, mainframe, laptop, personal data assistant (PDA), wireless e mail device, cellular phone, smartphone, tablet, mobile device, pager, processor, fax machine, scanner, or any other programmable device configured to transmit and/or receive data over a network. Computer systems and computer-based devices disclosed herein may include memory for storing certain software modules or engines used in obtaining, processing, and communicating information. It can be appreciated that such memory may be internal or external with respect to operation of the disclosed embodiments. The memory may also include any means for storing software, including a hard disk, an optical disk, floppy disk, ROM (read only memory), RAM (random access memory), PROM (programmable ROM), EEPROM (electrically erasable PROM) and/or other computer- readable media. The software modules and engines described herein can be executed by the processor (or processors as the case may be) of the computer devices that access the memory storing the modules.

In various embodiments disclosed herein, a single component may be replaced by multiple components and multiple components may be replaced by a single component to perform a given function or functions. Except where such substitution would not be operative, such substitution is within the intended scope of the embodiments. Any servers described herein, for example, may be replaced by a "server farm" or other grouping of networked servers (such as server blades) that are located and configured for cooperative functions. It can be appreciated that a server farm may serve to distribute workload between/among individual components of the farm and may expedite computing processes by harnessing the collective and cooperative power of multiple servers. Such server farms may employ load-balancing software that accomplishes tasks such as, for example, tracking demand for processing power from different machines, prioritizing and scheduling tasks based on network demand and/or providing backup contingency in the event of component failure or reduction in operability.

While various embodiments have been described herein, it should be apparent that various modifications, alterations, and adaptations to those embodiments may occur to persons skilled in the art with attainment of at least some of the advantages. The disclosed embodiments are therefore intended to include all such modifications, alterations, and adaptations without departing from the scope of the embodiments as set forth herein.