Semantic object synchronous understanding for highly interactive interface

CA 02467134 2003-05-13

SEMANTIC OBJECT SYNCHRONOUS

UNDERSTANDING FOR HIGHLY INTERACTIVE

INTERFACE

BACKGROUND OF THE INVENTION

The present invention relates to access and rendering of information in a computer system. More particularly, the present invention relates to access of information using recognition and understanding.

Recently, technology has been advanced to allow the user to access information on a computer system by providing speech commands. Upon receipt of a user command, the computer system performs speech recognition on the user input and further processes the input in order to ascertain the intent of the user in order that the computer system will perform a desired action.

In some situations, the input provided by the user is incomplete or indefinite, which will require the computer system to solicit further information 20 from the user either in the form of visual or audible prompts. A dialog can thus be user and the computer system, where each takes turns and/or acknowledgments action can be performed.

is

In creating such a dialog is the interacting with the computer ascertained and an preferred mode for system.

Speech (SALT) has been as a viable input/output modality for modern interface

CA 02467134 2003-05-13 design. The design goal for SALT is to make common speech tasks simple to program, yet allow advanced capabilities with straightforward realization. SALT was designed for many applications. One being, for example, a telephone-based, speech-only application that only interacts with users exclusively through spoken dialogue.

SALT includes speech input and output objects (listen and prompt), which have a mode design to incorporate technologies to detect the start and the end of the user’s turn. Accordingly, many speech applications employ user interfaces that require the user to signal the start of a user turn. Some computer systems include wearable computers, speech enabled modal or multimodal (speech input provided for fields selected by an input device such as a mouse) devices and other eyes-free applications. Nevertheless, in each of these environments, a clean cut definition on the user versus computer system turn in the dialog is still present.

Human conversation however does not generally follow a clean cut, turn-taking dialog participants. Rather, conversations can include acknowledgements, confirmations, questions by one participant, etc., while the other is providing information that may drastically affect, slightly which the speaker is providing information. Human speakers enjoy this natural form of conversation.

Likewise,

CA 02467134 2003-05-13 telephone systems employ full duplex technology in order to allow such conversations to take place.

In contrast, dialogue based interfaces employ _arigid turn-taking mode of operation between a user and a computer system, which causes the computer system to wait for the end of the user dialog before processing and taking subsequent action. Although simple feedback, such as visual indications like a series of dots progressing across a computer screen, may provide the user some assurance that the computer system is at least processing something, until the user finishes his/her turn and the computer system responds, the extent of understanding by the computer system is not known.

Accordingly, there is a need for improvements in a computer systems that is based on recognition and understanding. Such improvements would provide a system or method for accessing information that would be easier to use by being more natural for the user.

SUMMARY OF THE INVENTION

A method and system provide a speech input mode which dynamically reports partial semantic parses, while audio captioning is still in progress. The semantic parses can be evaluated with an outcome immediately reported back to the user. The net effect is that tasks conventionally performed in the system turn are now carried out in the midst of the user turn thereby presenting a significant departure from the turn-taking nature of a spoken dialogue.

CA 02467134 2003-05-13

Generally, one aspect of the present invention includes a computer implemented method for interacting with a computer system. The method includes receiving input from a user and capturing the input for processing. Recognition is then performed on the input, sequentially or simultaneously, to ascertain semantic information pertaining to a first portion of the input and outputting the semantic object. The semantic object includes data in a format to be processed by computer application that is in accordance with the input that has been recognized (e.g. text, encrypted text, compressed text, etc . ) as well as semantic information for the first portion.

As indicated

<img file="CA2467134C_D0001.tif" />

performed

<img file="CA2467134C_D0002.tif" />

<img file="CA2467134C_D0003.tif" />

semantic object is outputted while capturing continues for subsequent portions of the input. This method can be implemented for audible input such as speech as well as non-audible input, for example, visual input or handwriting recognition.

The above method can be implemented in a computer readable media that includes instructions readable by a computing device which, when implemented, cause the computing device to handle information and. perform the method. In a further embodiment, the computer readable medium can include instructions to establish a language model to perform recognition and understanding. The language model is adapted to provide data in a format in accordance

CA 02467134 2012-07-10

51039-8 with the recognized input and to provide semantic information for the input received.

According to one aspect of the present invention, there is provided a computer-implemented method for interacting with a computer system, the method comprising: receiving input comprising at least one command executable by an application from a user and capturing the input for processing; performing recognition on the input to ascertain semantic information pertaining to a first portion of the input and outputting a semantic object comprising data including data for executing the at least one command in a format to be processed by a computer application and being in accordance with the input that has been recognized and semantic information for the first portion, wherein performing recognition and outputting the semantic object are performed using a language model comprising a combination of an N-gram language model and a context free grammar and while capturing continues for subsequent portions of the input, the language model storing information related to words and semantic information to be recognized; and rendering information to the user while the user is providing the input, at least some of the information being responsive to the data regarding the at least one command within said semantic object.

According to another aspect of the present invention, there is provided a computer readable storage media including instructions readable by a computing device which, when implemented, cause the computing device to handle information by performing steps comprising: establishing a language model to perform recognition and understanding, the language model comprising a combination of an N-gram language model and a context free grammar and storing information related to words and semantic information to be recognized, the language model providing data in a format to be processed by a computer application and being in accordance with an input received and to provide semantic information for the input received; receiving input comprising at least one command executable by an application from a user and capturing the input for processing; processing the input with the language model to perform recognition on the input to ascertain semantic information pertaining to a first portion of the input and outputting a semantic object comprising data

CA 02467134 2012-07-10

51039-8

5a including data for executing the at least one command in a format to be processed by a computer application and being in accordance with the input that has been recognized and semantic information for the first portion, wherein performing recognition and outputting the semantic object are performed while capturing continues for subsequent portions of the input; and rendering information to the user while the user is providing the input, the information being a function of said semantic object wherein at least some of the information includes information regarding the at least one command.

According to still another aspect of the present invention, there is provided a computer readable storage media including instructions readable by a computing device comprising: a language model comprising a combination of an Ngram language model and a context-free grammar language model and storing information related to words and semantic information to be recognized; a module receiving input including commands executable by an application from a user and capturing the input for processing, the module performing recognition on the input by accessing the language model and ascertaining semantic information pertaining to a first portion of the input and outputting a semantic object comprising data including data for executing commands in a format to be processed by a computer application and being in accordance with the input that has been recognized and semantic information for the first portion, wherein performing recognition and outputting the semantic object are performed while capturing continues for subsequent portions of the input; and a second module performing a selected task, the second module receiving semantic objects synchronously in accordance with the user’s input, each semantic object comprising data including commands in a format to be processed by a computer application and in accordance with a portion of input from the user and semantic information pertaining to the portion of input from the user, the module taking action as a function of processing the semantic objects synchronously including providing information or utilizing other applications based on at least the data for executing the commands.

CA 02467134 2009-05-11

51039-8

5b

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a data presentation system.

FIG. 2 is a plan view of a computing device operating environment.

FIG. 3 is a block diagram of the computing device of FIG. 2.

FIG. 4 is a plan view of a telephone.

FIG. 5 is a block diagram of a general purpose computer.

FIG. 6 is a block diagram of an architecture for a client/server system.

FIG. 7 is a block diagram of a speech recognition and understanding module.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

FIG. 1 is a block diagram of a data presentation system 10 for rendering data based on voice input. System 10 includes a speech interface module 12, a speech recognition and understanding module 14 and a data rendering module 16. A user provides input in the form of a voice query to speech interface module 12. Speech interface module 12 gathers speech information from a user and provides a signal indicative thereof. After the input speech has been gathered by speech interface module 12, speech recognition and understanding module 14 recognizes the speech using a speech recognizer and also performs speech understanding, wherein in one aspect of the present

CA 02467134 2003-05-13 invention, provides partial semantic parses of the input so far received, while audio speech input is still being captured.

The partial semantic parses, which typically include the text for the input received (or other data indicative of rhe text of the input), but also ascertained semantic information, are provided to an application module 16 that can take many different forms. For instance, in one embodiment, the application module 16 can be a personal information manager that is used to send, receive and reply to e-mails, arrange meetings and the like. In this manner, the user can provide audible commands to perform these tasks. More importantly though, the application module 16 can provide interactive feedback and/or take actions on the partial semantic parses information as it is received, thereby providing the user with a highly interactive interface for the application module 16. For example, in a voice only mode of operation, an output 20 can include audible statements back to the user, while, of course, possibly performing other tasks related to the application.

The partial semantic parses or semantic objects can be used to execute dialog logic in the application. For instance, the dialog logic can present an option, or a plurality or list of options, to the user based on one or more semantic objects.

This enables the system 10 to immediately report outcomes based on a partial utterance, namely, before the user turn has ended. In other words, by using back channel communication to report and perform tasks

CA 02467134 2003-05-13 normally associated with a system cum, the definition of a user and a system turn is blurred. Most conventional dialog studies, especially those based on human to human dialogs, often view the back channel communications as non-intrusive feedback that conveys only simple signals such as positive, negative, or neutral acknowledgement. However, the feedback provided by output 20 can potentially carry more information so as to be somewhat intrusive to the ongoing user utterance, which may or may not cause the user to clarify the user's intention or direction. Nevertheless, this approach provides a more realistic human dialog between the user and the system 10, which in many cases will not be considered annoying, but rather be more comfortable to the user and instill confidence that the user's desires will be met.

At this point it should be noted that the present invention is not limited to a voice-only operating environment, but rather can include other forms of feedback to the user based on processing of partial semantic parses or objects. For example, in the application discussed above where the application module 16 performs e-mail tasks, the output 20 can include visual feedback such as activating an e-mail module based only on receipt of a phrase such as Send e-mail in an uninterrupted command from the user comprising Send e-mail to Bob, where processing of the phrase to Bob, may cause the application module to access further information in data store 18 and render a list of people having the first name Bob.

CA 02467134 2003-05-13

-8Whereupon seeing the list, the user simply identifies the intended recipient as Bob Green, which can then be selected because the system would have provided another semantic object for the partial utterance Bob Green, which upon receipt by the application and processed would cause Bob Green to be selected.

As indicated above, the application module 16 can take many forms where aspects of the present invention discussed further below may be beneficial. Without limitation, the application module 16 can also be a dictation module to provide a textual output of the user's spoken input. However, by processing semantic information for partial input or phrases of input as well, a more accurate transcription can be obtained.

Although described above with respect to an input from the user comprising voice commands, aspects of the present invention can also be applied to other forms of input such as handwriting, DTMF, gesture or visual indications .

Given the broad applicability of processing partial semantic parses or objects, it may be useful to describe generally computing devices that can function in system 10 described above. As appreciated by those skilled in the art, the components of system 10 may be located within a single computer or distributed across a distributed computing environment using network connections and protocols.

Referring now to FIG. 2, an exemplary form of a mobile device such as a data management device (PIM, PDA or the like) is illustrated at 30. However, it is

CA 02467134 2003-05-13 contemplated that the present invention can also be practiced using other computing devices discussed below. For example, phones and/or data management devices will also benefit from the present invention. Such devices will have an enhanced utility compared to existing portable personal information management devices and other portable electronic devices.

In the exemplary form of a data management mobile device 30 is illustrated in FIG. 2, the mobile device 30 includes a housing 32 and has an user interface including a display 34, which uses a contact sensitive display screen in conjunction with a stylus 33. The stylus 33 is used to press or contact the display 34 at designated coordinates to select a field, to selectively move a starting position of a cursor, or to otherwise provide command information such as through gestures or handwriting. Alternatively, or in addition, one or more buttons 35 can be included on the device 30 for navigation. In addition, other input mechanisms such as rotatable wheels, rollers or the like can also be provided. However, it should be noted that the invention is not intended to be limited by these forms

<td> of input mechanisms.</td><td colspan="2"> For instance, another</td><td> form of</td>

<td> input can include a</td><td> visual</td><td> input such as</td><td> through</td>

<td> computer vision.</td><td></td><td></td><td></td>

<td> Referring now</td><td> to FIG.</td><td> 3, a block</td><td> diagram</td>

illustrates the functional components comprising the mobile device 30. A central processing unit (CPU) 50 implements the software control functions. CPU 50 is coupled to display 34 so that text and graphic icons

CA 02467134 2003-05-13 generated in accordance with the controlling software appear on the display 34. Ά speaker 43 can be coupled to CPU 50 typically with a digital-to-analog converter 59 to provide an audible output. Data that is downloaded or entered by the user into the mobile device 30 is stored in a non-volatile read/write random access memory store 54 bi-directionally coupled to the CPU 50. Random access memory (RAM) 54 provides volatile storage for instructions that are executed by CPU 50, and storage for temporary data, such as register values. Default values for configuration options and other variables are stored in a read only memory (ROM) 58. ROM 58 can also be used to store the operating system software for the device that controls the basic functionality of the mobile device 30 and other operating system kernel functions (e.g., the loading of software components into RAM 54).

RAM 54 also serves as a storage for the code in the manner analogous to the function of a hard drive on a PC that is used to store application programs. It should be noted that although non-volatile memory is used for storing the code, it alternatively can be stored in volatile memory that is not used for execution of the code.

Wireless signals can be transmitted/received by the mobile device through a wireless transceiver 52, which is coupled to CPU 50. An optional communication interface 60 can also be provided for downloading data directly from a computer (e.g., desktop computer), or from a wired network, if desired. Accordingly,

CA 02467134 2003-05-13 var loss forms of communication devices, includes a example, in response commands to rom

3/, and an handwriting, store o4.

a user of

By gesture microphone 2 orovides which are digitized by perform normalization and/or feature extraction functions on the obtain intermediate speech or communication interface 60, speech data can be transmitted to a remote recognition server 204 discussed below and illustrated in the architecture of FIG. 6. Recognition results can then be returned to mobile device 30 for rendering (e.g. visual and/or audible) thereon, and eventual transmission to a web server 202 (FIG. 6), wherein the web server 202 and mobile device 30 operate in a client/server relationship.

Similar processing can be used for other forms of input. For example, handwriting input can be digitized with or without pre-processing on device 30. Like the speech data, this form of input can be transmitted to the recognition server 204 for recognition wherein the recognition results are returned to at feast one of the device 30 and/or web server 202. Likewise, DTMF data,

CA 02467134 2003-05-13 gesture data and visual data can be processed similarly. Depending on the form of input, device 30 (and the other forms of clients discussed below) would include necessary hardware such as a camera for visual input.

FIG. 4 is a plan view of an exemplary embodiment of a portable phone 80. The phone 80 includes a display 82 and a keypad 84. Generally, the block diagram of FIG. 3 applies to the phone of FIG. 4, although additional circuitry necessary to perform other functions may be required.

For instance, a transceiver necessary to operate as a phone will be required for the embodiment of FIG. 3; however, such circuitry is not pertinent to the present invention.

In addition to the portable or mobile computing devices described above, it should also be understood that the present invention can be used with numerous other computing devices such as a general desktop computer. For instance, the present invention will allow a user with limited physical abilities to input or enter text into a computer or other computing device when other conventional input devices, such as a full alpha-numeric keyboard, are too difficult to operate.

The invention is also operational with numerous other general purpose or special purpose computing systems, environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, regular telephones (without any screen) personal computers,

CA 02467134 2003-05-13 server computers, hand-held or laptop devices, tablet computers, multiprocessor systems, microprocessor-based sysoems, sev top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

The following is a brief description of a general purpose computer 120 illustrated in FIG. 5. However, the computer 120 is again only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computer 120 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated therein.

The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment., program modules may be located in both local and remote computer storage media including memory storage devices. Tasks performed by the programs and modules are described

CA 02467134 2003-05-13 below and with the aid of figures. Those skilled in be

With reference to

120 may include, a computer readable medium.

FiG. 5, components of computer but a.re not limited to, a processing unit 140, a system memory 150, and a system bus 141 bus 141 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a

By way of example, and not limitation, such architectures include industry

Standard Architecture (USB), Micro

Architecture (MCA) bus , Enhanced

ISA (EISA) bus,

Video

Association (VESA) local bus, and

Component Interconnect (PCI) known as

Mezzanine bus. Computer 120 typically includes a mediums .

Computer readable mediums can be any available media that can be accessed by computer 120 and includes both volatile and nonvolatile media, removable and nonremovable media.'

By way or example, and not computer and mediums may comprise communication media.

Computer includes both volatile and

CA 02467134 2003-05-13 of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computer 120.

Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term modulated data signal means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, FR, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.

The system memory 150 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 151 and random access memory (RAM) 152. A basic input/output system 153 (BIOS), containing the basic routines that help to transfer information between elements within

CA 02467134 2003-05-13 computer 120, such as during start-up, is typically stored in ROM 151. RAM 152 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 140, By way of example, and not limitation, FIG. 5 illustrates operating system 154, application programs 155, other program modules 156, and program data 157 .

The computer 120 may also include other removable/non-removable volatile/nonvolatile computer storage media. By way of example only, FIG. 5 illustrates a hard disk drive 161 that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive 171 that reads from or writes to a removable, nonvolatile magnetic disk 172, and an optical disk drive 175 that reads from or writes to a removable, nonvolatile optical disk 17 6 such as a CD ROM or other optical media. Other removable/nonremovable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive 161 is typically connected to the system bus 141 through a non-removable memory interface such as interface 160, and magnetic disk drive 171 and optical disk drive 175 are typically connected to the system bus 141 by a removable memory interface, such as interface 170.

CA 02467134 2003-05-13

-17Tne drives and their associated computer storage provide storage program modules and other data for the computer 120.

hard disk drive 161 is illustrated as storing operating system

164, application programs 165, other program modules 166, and program data 167. Note that these components can either be the same as or different from operating system 154, application programs 155, other program modules 156, and program data 157. Operating system 164, application programs 165, other program modules 166, and program data 167 are given different numbers here to illustrate that, at a minimum, they are different copies.

A user may enter commands and information into the computer 120 through input devices such as a keyboard 182, a microphone 183, and a pointing device 181, such as a mouse, trackball or touch pad. Other input devices (not shown) may include a joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 140 through a user input interface 180 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB) . A monitor 184 or other type of display device is also connected to the system bus 141 via an interface, such as a video interface 185. In addition to the monitor, computers may also include other

CA 02467134 2003-05-13 peripheral output devices such as speakers 187 and printer 186, which may be connected through an output peripheral interface 188.

The coirpurer 120 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 194. The remote computer 194 may be a personal computer, a hand-held device, a server, a router, a network PC, a peer device or other common network node, and or all of the described above relative to the computer

120.

The in

FIG. 5 include a local area

191 and a wide area network (WAN) 133, include other networks. Such networking environments are commonplace in offices, enterprise-wide networks, intranets and the

Internet.

When used in a LAN networking environment, the computer 120 is connected to the LAN 191 through a network interface or adapter 190. When used in a WAN networking environment, the computer 120 typically includes a modem 192 or other means for establishing communications over

WAN 193, such as the external, the user

The modem

192, which may be internal or may be connected to the system bus 141 via

180, or other appropriate a networked environment, program

120, or storage thereof, may be stored tne remote memory

By way of example, and not

CA 02467134 2003-05-13 limitation, FIG. 5 illustrates remote application programs 195 as residing on remote computer 194. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

FIG. 6 illustrates architecture 200 for web based recognition and data rendering, which is one exemplary environment for the present invention.

Generally, information stored in a web server 202 can be accessed through a client 100 such as mobile device 30 or computer 120 (which herein represent other forms of computing devices having a display screen, a microphone, a camera, a touch sensitive panel, etc., as required based on the form of input), or through phone 80 wherein information is requested audibly or through tones generated, by phone 8 0 in response to keys depressed and wherein information from web server 202 is provided only audibly back to the user.

In 'this embodiment, architecture 200 is unified in than whether information is obtained through client 100 or phone 80 using speech recognition, a single recognition server 204 can support either mode 25 of operation. In addition, architecture 200 operates using an extension of well-known mark-up languages (e.g. HTML, XHTML, cHTML, XML, WML, and the like).

Thus, information stored on web server 202 can also be accessed using well-known GUI methods found in 30 these mark-up languages. By using an extension of

CA 02467134 2003-05-13 well-known mark-up languages, authoring on rhe web server 202 is easier, and legacy applications currently existing can be also easily modified to include voice recognition.

Generally, client 100 executes HTML pages, scriprs, or the like, generally indicated at 206, provided by web server 202 using a browser. When voice recognition is required, by way of example, speech data, which can be digitized audio signals or speech features wherein the audio signals have been preprocessed by client 100 as discussed above, are provided to recognition server 204 with an indication of a grammar cr language model 220 to use during speech recognition, which may be provided by client 100. Alternatively, speech server 204 may include the

<td colspan="2"> language model</td><td> 220. The</td><td> implementation of</td><td> the</td>

<td> recognition</td><td> serve</td><td> r 204 can t</td><td> ake many forms, one</td><td> of</td>

<td> which is</td><td colspan="2"> illustrated, but</td><td> generally includes</td><td> a</td>

<td> recognizer</td><td> 211.</td><td> The result</td><td> s of recognition</td><td> are</td>

<td> provided ba</td><td> ck to</td><td> client 100</td><td> for local rendering</td><td> if</td>

<td> desired or</td><td colspan="2"> appropriate. If</td><td colspan="2"> desired, text-to-speech</td>

<td> module 222</td><td colspan="2"> can be used to</td><td> provide spoken text</td><td> to</td>

<td> client 100.</td><td> Upon</td><td> compilation</td><td> of information thrc</td><td> &gt;ugh</td>

recognition and any graphical user interface if used, client 100 sends the information to web server 202 for further processing and receipt of further HTML pages/scripts, if necessary.

As illustrated in FIG. 6, client 100, web server 202 and recognition server 204 are commonly connected, and separately addressable, through a

CA 02467134 2003-05-13

Internet.

these other .

server server this manner,

202 can oe focused on authors

LO know the of recognition

204. Rather, recognition server

204 can be

205, and thereby, be updated server 202.

mechanism that can dynamically generate client-side server 202, be combined depending implementing a further embodiment, on the., capabilities

For instance, if the the

100 of web may the client

100 comprises a general purpose computer, personal computer, the client the recognition server

204. Likewise, if desired, the web and can be incorporated single machine.

Access to web through phone includes or network 208, that in turn, connects

Gateway

210

212 .

media server

214 than provides a telephony interfac

CA 02467134 2003-05-13 and a voice browser 216. Like client 100, uelephony voice browser 212 receives HTML pages/scripts or the like from web server 202. In one embodiment, the HTML pages/scripts are' of the form similar to HTML pages/scripts provided to client 100. In this manner, web server 202 need not support client 100 and phone 80 separately, or even support standard GUI clients separately. Rather, a common mark-up language can be used. In addition, like client 100, voice recognition from audible signals transmitted by phone 80 are provided from voice browser 216 to recognition server 204, either through the network 205, or through a dedicated line 207, for example, using TCP/IP. Web server 202, recognition server 204 and telephone voice browser 212 can be embodied computing environment such as the desktop computer illustrated in FIG.

in any suitable o ..

and in system 10, a components and the function of system 10 is provided.

FIG. 7 illustrates speech understanding module speech interface module is sent to speech recognition and understanding module recognition and understanding module 14 recognition engine associated language model 310

306 language model 310 semantic to represent each of the phrases

CA 02467134 2003-05-13 forming the input, providing partial semantic parses or objects as the input is received. Unlike systems that wain for the user to complete the utterance and then process the complete input received, rhe module 5 14 continuously provides semantic ohjeers based only what has so far been received.

Recognition engine 306 provides at least one surface semantic output object based on the partial utterance. In some embodiments, the recognition 10 engine 306 is capable of providing more than one alternative surface semantic object for each alternative structure.

Although nllustrated in FIG. 7 wherein speech input is provided, the present invention can be used 15 with handwriting recognition, gesture recognition or graphical user interfaces (where the user interacts with a keyboard or other input device) . In these other embodiments, the speech recognizer 306 is replaced with a suitable recognition engine as is 20 known in the art. For graphical user interfaces, a grammar (having the language model) is associated with the user input such as through an input box. Accordingly, a user's input is processed in a consistent way without significant modification based 25 on the manner of input.

The interactive dialog discussed above, which also includes other forms of informative feedback by the system 10 based on partial semantic parses or objects can be implemented using SALT (Speech 30 Application Language Tags) or other speech,

CA 02467134 2003-05-13

-24handwriring, and pattern

<img file="CA2467134C_D0004.tif" />

APIs (Application Program Interfaces) chat support a language model construct that can provide semantic information based on a selected schema for a given aoplication and semantic objecc synchronous decoding.

SALT is a developing standard for enabling access to information, applications and web services from telephones, tables PCs and wireless mobile devices, for example, but can also be applied to application interfaces as well without interconnection across a network.

SALT extends existing markup languages such as HTML, XHTML and XML. The SALT 1.0 specification may be found online at http://www.SALTforum.org. It should be noted that SALT can provide semantic information based upon the user's input, for example from speech server 204, which such information forms the objects provided to data rendering module 16 after completion of an utterance; however as discussed further below SALT can be used in manner not previously contemplated to provide partial semantic parses or objects. Use of SALT extensions, or similar extensions in other APIs, provides support for highly interactive event driven user interaction.

Using SALT by way of example, the SALT listen object can be used to perform both speech recognition and understanding tasks. This is because the design follows the viewpoint and formulation that treats speech understanding as a pattern recognition problem, just like speech recognition. Both strive to

CA 02467134 2003-05-13 find a pattern from a collection cf possible outcomes that best match a given speech signal. For speech recognition, the pattern to be found is a string of

<td> words whe</td><td> reas, for understanding,</td><td> a tree of semantic</td>

<td> obj ects.</td><td> A traditional speech</td><td> recognition task</td>

<td> instructs</td><td> the search process with</td><td> a language model in</td>

<td> composing</td><td> the likely word stri</td><td> ngs. In a similar</td>

<td> fashion,</td><td> a speech understanding</td><td> task can guide the</td>

same search engine to compose the suitable semantic object trees with a semantic model.

Like a language model that often implies a lexicon and the rules of composing phrase segments entries, a objects and the rules of composing them. While the recognition outcome is text string, the understanding result is a tree

Although it is oossible to extend N-gram to return a structured search outcome, most typical speech understanding probabilistic context free grammar (PCFG) where the rules of composing semantic annotated training data. One method of specifying associate each PCFG rule with production directives for the search engine on how to transform the partia

PCFG parse tree into a semantic object tree. An example written interface (SAPi) in Microsoft Speech Application (which is also an example of a speech API that can be used in the present invention) format is shown below:

CA 02467134 2003-05-13

-2 6<rule name=nyc>

<list>

<phrase>new york ?city</'phrase> <phrase>?the big apple</'phrase> </list>

<output>

kcit y__locat ion>

<city>New York</city>

<state>New York</state>

<country>USA</country>

</city_location>

</output>

</rule>

krule name=NewMeeting> kruleref min=0 name kruleref max=inf name^:<output>

<NewMeeting>

<DateTime>

<xsl:apply-templates <xsl:apply-templates <xsl:apply-templates </DateTime>

CarrierPhrase/>

ApptProperty/>

selecL=//Date/>

select=//Time/>

select=//Duration/>

krnviteest <xsl:apply-templates select=//Person/>

</Invitees>

k/NewMeet inert

CA 02467134 2003-05-13

-27</output>

</rule>

Crule name=ApptProperty/>

<list>

<td> Cruleref</td><td> name=Date/&gt;</td>

<td> cruleref</td><td> name=Durâtion/&gt;</td>

<td> Cruleref</td><td> n anie= T ime / &gt;</td>

<td> Cruleref</td><td> name=Person max=inf/&gt;</td>

<td> Cruleref</td><td> name=ApptSubject/&gt;</td>

</list>

</rule>

The grammar one, a pre-terminal named nyc lists the expressions for

New York City.

for constructing semantic objects.

They are invoked when the search path exits the grammar node denoted by the token immediately case, a' semantic object, created when a semantic object composed of three semantic city name, state and country name abbreviations, respectively semantic a dynamic process, for example, in scheduling a new meeting. For instance, a NewMeeting semantic object

CA 02467134 2003-05-13 will be produced when the user finishes specifying the meeting properties such as date, time, duration and attendees. Templates can be used to paste other semantic objects as constituents into the NewMeeting semantic object. The same principle can also be applied to other rules not shown here. As an example, an utterance schedule a meeting with Li Deng and Alex Acero on January first for one hour will result in the following semantic object:

<NewMeeting>

<DateTime>

<Date>01/01/2003</Date>

<Duration>3600</Duration>

</DateTime>

<Invitees>

<Person>Li Deng</Person> <Person>Alex Acero</Person>

</Invitees>

</NewMeeting>

in real applications, improving PCFG coverage is a daunting task.

It is therefore desirable to be able to use an N-gram to model, among other things, functional phrases that do not carry critical semantic information but usually have sizeable variations in the syntactic structure (e.g., May

I..., Could you show me..., Please show me...) . In one embodiment, the language model 310 comprises a semantic language model that combines PCFG with N

CA 02467134 2003-05-13

4« gram. The technique is slightly different from a unified language model, which can also be used. The unified language model is a natural extension to the conventional class N-gram except it allows CFG fragments, not just a list of words, to be modeled as an individual token in N-gram. The recognizer 306 using this model still produces text string that has to be subsequently parsed. The unified language model thus is designed to incorporate certain linguistic structure to assist text transcription.

The semantic language model, on the other hand, aims at using the decoder or recognizer to search for the semantic structure, which is usually better captured by PCFG. Therefore,' instead of embedding CFG fragments into N-gram, the PCFG is used to contain Ngram by creating a special PCFG pre-terminal that corresponds to an N-gram of interests. In Microsoft SAPI grammar format, this can be denoted using a pre-terminal with an XML <dictation> tag, as in

LCFG <dictation max=inf/> RCFG where LCFG and RCFG denotes the left and right context of the embedded N-gram, respectively. The search process treats the <dictation> tag as a token and expands into the N-gram as if entering a regular non-terminal. The max attribute on the tag specifies the maximum number of words that can be consumed by the N-gram. Inside vhis N-gram, the word string probability is computed by interpolating the backoff

CA 02467134 2012-02-02

51039-8

-30N-gram with the PCFG, more specifically,

P(v>n I = I ^ram,w_n__pw_n_₂,..:) + (1 - I RCFG)P(RCFG K_j, w„_₂,...) ¹¹ >

where λ is the N-gram interpolation weight and P(RCFG

I uses the back-off probability of the N5 gram, i.e., w_n is treated as if it is an out of vocabulary word. In one embodiment, the term P{w_n | RCFG) assumes only binary value depending on whether the maximum N-gram word string size is reached and the word is in the coverage of the CFG fragment or 10 not. Since words drawn from PCFG often have a higher probability, paths that really belong to be covered by CFG have tendency to win out their N-gram counterparts even when the maximum N-gram word count is set to infinite. In addition to functional 15 phrases, the embbeded N-gram can also be used to model semantic object with a dictation like property. For example, the meeting subject is model in our task as

Crule name=ApptSubject>

<p> <dictation max=inf/> </p>

Further details regarding the semantic language model are described in K. Wang, Semantic modeling 25 for dialog systems in a pattern recognition framework, in Proc. ASRU-2001, Trento Italy, 2001.

CA 02467134 2012-02-02

51039-8

-31A further aspect of the present invention includes a new use of the SALT listen object . SALT provides a set of XML elements, with associated attributes and DOM object properties, events and methods, which may be used in conjunction with a source markup document to apply a speech interface to a source page. In general, the main elements include:

<prompt ...> for speech synthesis configuration and prompt playing <listen .„> for speech recognizer configuration, recognition execution and post15 processing, and recording <dtmf ..> for configuration and control of DTMF <smex ...> for general purpose communication with platform components

The listen and dtmf objects also contain grammar and binding controls:

<grammar ...> for specifying input grammar resources <bind ...> for processing of recognition results

The listen element can include a mode attribute to distinguish three modes of recognition, which instruct the recognition server (e.g. 204) how and when to return results. The return of results 30 implies providing the onReco event or activating the bind elements as appropriate.

In a first mode, automatic, the speech platform, rather than the application, is in control of when to stop the recognition process. This mode

CA 02467134 2003-05-13 telephony or scenarios.

available, advanced for result is time period has elapsed silence, the speech stops the its can the

In a second mode of opertion, single, the return of a recognition result is under the of an explicit stop call. The stop call would correspond to an event such as pen-up by the user, and this mode is advanced for being used in a multimodal environment where the device allows speech input, but the user is generally under control of which and when to select fields for example through the use of the stylus 33 (FIG.l).

A third mode of operation of the speech recognizer is a multiple mode. This mode of operation is used for an open-microphone or in a dictation scenario. Generally, in this mode of operation, recognition results are returned at intervals until an explicit stop call is received or the or other time periods associated with unrecognized input or a maximum time to listen are exceeded. Generally, in this mode of operation, for each phrase that is recognized, a onReco event is issued and the result is returned until the stop () call is received.

However this mode of operation, as aspect of the present invention, can provide another a means

CA 02467134 2003-05-13 for rhe search engines to expose more interactive capabilities to the users by allowing them to report immediately whenever a salient linguistic landmark is reached. Search algorithms based on time synchronous decoding are well known and can be employed for this mode in a straightforward manner. One such algorithm is described in H. Ney, S. Ortmanns, Dynamic programming search for continuous speech recognition,

IeEE Signal Processing Magazine, pp.

64-83, 1999.

For speech recognition, the linguistic landmark usually corresponds to a word or a phrase boundary. A SALT multiple mode recognition can therefore be used to display dynamically the word string hypotheses as soon as they are available, a UI effect commonly seen in many commercial dictation software. However, in the present invention, the multiple mode can treat the instantiations of semantic objects as linguistic landmarks and report, i.e. provide some response as a function of what was understood, back to the application on them in a dynamic manner as well. This appears to the application designer as if SALT is performing a semantic object synchronous understanding.

This mode of operation may be better understood by comparing it with a multimodal scenario. In the multmodal scenario, a user indicates a field, for example, by pointing and holding the stylus in an input field while speaking. While the user can tap onto a generic field and utter an elaborated sentence to fill many fields in a single sentence, the tap

CA 02467134 2003-05-13 and-talk interface nevertheless engages the eyes and hands of the user, a scheme not appropriate for many situations. In addition, although tap-and-talk features a rich back channel communication . that displays the volume and a progress bar of the underlying spoken language process, those feedbacks provide only very primitive clues to the quality of the spoken language processing in terms of speed and accuracy. This can be potentially more problematic for longer sentences in which errors can propagate to a wider scope that eventually requires more efforts in merely verifying and correcting the recognition and understanding outcomes. Since the usability studies seem to indicate that long sentences are a key differentiating factor that demonstrates the utility of speech as more than a keyboard enhancement or alternative, a satisfying UI experience is absolutely necessary to the success of using speech as a viable modality.

In order to promote the perception of human computer as achieving a common understanding, by collaborative partners in goal, semantic object synchronous reporting the partial semantic parses or objects as soon as they are available, is effective .

xn one embodiment, this is achieved by using the multiple mode of the element

SALT .

In listen element multiple mode is designated and the for the input speech to be recognized. The assignment of results is

CA 02467134 2003-05-13

<td> also</td><td> specified in the listen</td><td> object. For example,</td>

<td> HTML</td><td> input code to acquire</td><td> necessary information,</td>

<td> such</td><td> as daoe, time, location,</td><td> subject, and meeting</td>

<td> atte</td><td> ndees, etc., for creating</td><td> a new meeuing might</td>

take the form:

blister. mode=maltiple ...>

bgrammar src=subject.grm/>

bgrammar src=date.grm/> bgraramar src=time_duration . grin/> bgrammar src=attendees.grm/> <bind targetElement=subject value=//ApptSubject/>

<bind targetElement=date value=//DateTime/>

<bind targetElement=start_time value=//start_time targetElement=end_time value=//end—time targetElement=duration value-//DateTime/duration/>

</listen>

The multiple grammars compose a parallel search space for rhe recognition with a null transition looping back to entry point. In this mode, SALT allows the listen object to raise an event as soon as a grammar is exited. The event forks a parallel process to invoke the bind directives in sequence

CA 02467134 2003-05-13

-36while the underlying audio collection and recognition are ongoing, thus creating the effect to the user chat relevant fields on the form are being filled while a spoken command is still being uutered for an application that has a visual rendering of fields.

For user interface for eyes free applications, accompanied speech outputs might be desired. In that case, SALT prompt objects can be used to give immediate feedbacks. For example, the following SALT prompt object can be used to synthesize response based on the dynatdc contents in the date field, and the speech synthesis can be triggered with additional SALT bind directives as follows:

<prompt id=say_date>

on <value targetElement=date/>

</prompt>

<listen ...>

Cbind targetElement=date value=//date targe tElement= say__date targetMethod=Start />

</listen>

The net effect is the user feels like he/she is talking to another party that not only jots down but repeats what is heard, as in Schedule a meeting (new

CA 02467134 2012-02-02

51039-8

-37meeting) at two (starting at two o'clock PM) next Tuesday (on 10/29/02) for two hours (duration: two hours), where the phrases provided in parentheses represent the audible and/or visual prompts (which also can be synchronized) back to the user.

It should be noted that SALT allows designers to attach customized recognition event handlers that perform sophisticated computations beyond the simple assignments as with the SALT bind directives. In the above example, the date normalization can be accomplished in the semantic grammar which, however, cannot facilitate advanced reference resolution (e.g., Schedule a meeting with Li Deng and his manager). For such cases, algorithms can be implemented as script objects accessible to proper event handlers to access stored data to ascertain indefinite references. Such algorithms are described in K. Wang, A plan based dialog system with probabilistic inferences, in Proc. ICSLP-2000, Beijing China, 2000 and European Patent Application EP 1199630A2, published April 24, 2002.

It should be noted that although the multiple mode operation existed for the listen object, in current implementations, this mode provides only text for the received input such as in the dictation scenario. However, in this aspect of the present invention, the ’’partial results as input is being received is not only the text, but also includes the corresponding semantic information pertaining to the

CA 02467134 2012-02-02

51039-8

-38text, and thus the output comprises partial semantic parses or objects, which can be used as described above to provide the user more quality feedback that the computer has properly understood what has been received. Depending on the sophistication of the application receiving the partial semantic parses or objects, the system can provide back to the user confirmations, alternatives, corrections and clarifications based on the partial semantic parses received.

Although in multimodal applications it is known to include multiple grammars to allow the user to speak ahead, and thereby, provide the ability to provide information that has not been indicated, using the listen element in the multiple mode operation can be better because it provides to the user a higher indication of understanding. In a voice-only application, a natural form of dialog is generated, whereas if using visual renderings are employed, an application can begin processing (taking actions, displaying interim results or options e.g. through pop-up windows) based only on partial semantic, parses of what the user has so far provided and while the user continues speaking.

Although the present invention has been described with reference to particular embodiments, workers skilled in the art will recognize that changes may be made in form and detail without departing from the scope of the invention.