Intelligent automated assistant.
Abstract
An intelligent automated assistant system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact.

Term
4.3 yearsleft in the term
Expires 11 January 2031.
- Priority
- Filed
- Granted
- Today
- Expires
32 claims: 18 independent, 14 dependent
- 1REIVINDICACIONES 1. Un método para operar un asistente automatizado inteligente, caracterizado porque comprende:en un dispositivo electrónico que comprende un procesador y una memoria que almacena datos para habilitar el procesador: recibir una petición de usuario, la petición de usuario incluye por lo menos una entrada de habla recibida de un usuario, en donde la entrada de habla incluye al menos un valor de al menos un parámetro;presentar al usuario un eco de la entrada de habla basado en una interpretación textual de la entrada de habla;determinar una tarea que será realizada por el dispositivo electrónico, desde una pluralidad de tareas realizables por el dispositivo electrónico, en donde la tarea es determinada al realizar procesamiento de lenguaje natural en la entrada de habla;y presentar al usuario una paráfrasis de la tarea, incluyendo una formulación en lenguaje natural de la tarea y el al menos un valor del al menos un parámetro.
- 2El método de conformidad con la reivindicación 1, caracterizado porque una oración respectiva usada en la paráfrasis se adapta a una modalidad respectiva usada para presentar la paráfrasis al usuario, y en donde una oración 267 IMPI INSTITUTO MKICANO Dt LA PROnibM) INDUSTRIAL más completa se utiliza en la paráfrasis mando la paráfrasis se va a presentar en forma audible que cuando la paráfrasis se va a presentar en forma escrita.
- 3El método de conformidad con cualesquiera de las reivindicaciones 1-2, caracterizado porque el eco incluye una o más primeras palabras y la paráfrasis incluye una o más segundas palabras, y en donde la una o más segundas palabras representan una interpretación semántica respectiva de la una o más primeras palabras y asocian la una o más primeras palabras con un concepto de un dominio respectivo entre una pluralidad de dominios predefinidos.
- 4El método de conformidad con cualesquiera de las reivindicaciones 1-3, caracterizado porque el eco y la paráfrasis se presentan en una pantalla de conversación, y en donde la paráfrasis incluye el al menos un valor del al menos un parámetro presentado en un formato destacado.
- 5El método de conformidad con cualesquiera de las reivindicaciones 1-4, en donde la paráfrasis indica un progreso en tiempo real de la tarea.
- 6El método de conformidad con cualesquiera de las reivindicaciones 1-4, en donde la paráfrasis proporciona un resumen de un conjunto de resultados, y en donde la paráfrasis se presenta además del conjunto de resultados.
- 7El método de conformidad con la reivindicación 6, caracterizado porque la paráfrasis especifica unos o más 268 IMPI INSTITUTO MBXICANO <* LA RROfUDAD INDUSTRIAL parámetros usados para obtener el conjunto do -rec.n i-ado^.
- 8El método de conformidad con cualesquiera de las reivindicaciones 1-4, caracterizado porque la paráfrasis especifica los criterios de selección múltiples derivados de la petición de usuario, e indica que no se ha encontrado ningún resultado que cumplía todos los criterios de selección múltiples, y en donde la paráfrasis especifica una forma relajada de los criterios de selección múltiples y presenta un resumen de los resultados obtenidos basados en la forma relajada de los criterios de selección múltiples.
- 9Un sistema para operar un asistente automatizado inteligente, caracterizado porque comprende:uno o más procesadores;y memoria que tiene datos ahí almacenados, los datos hacen que el uno o más procesadores realicen operaciones que comprenden: recibir una petición de usuario, la petición de usuario incluye por lo menos una entrada de habla recibida de un usuario, en donde la entrada de habla incluye al menos un valor de al menos un parámetro;presentar al usuario un eco de la entrada de habla basado en una interpretación textual de la entrada de habla;determinar una tarea que será realizada por el dispositivo electrónico, desde una pluralidad de tareas 269 IMPI INSTITUTO MEXICANO DS LA PROPIEDAD INDUJTRJAL realizables por el dispositivo electrónico, en donde la tarea es determinada al realizar procesamiento de lenguaje natural en la entrada de habla;y presentar al usuario una paráfrasis de la tarea, incluyendo una formulación en lenguaje natural de la tarea y el al menos un valor del al menos un parámetro.
- 10El sistema de conformidad con la reivindicación 9, caracterizado porque una oración respectiva usada en la paráfrasis se adapta a una modalidad respectiva usada para presentar la paráfrasis al usuario, y en donde una oración más completa se utiliza en la paráfrasis cuando la paráfrasis se va a presentar en forma audible que cuando la paráfrasis se va a presentar en forma escrita.
- 11El sistema de conformidad con cualesquiera de las reivindicaciones 9-10, caracterizado porque el eco incluye una o más primeras palabras y la paráfrasis incluye una o más segundas palabras, y en donde la una o más segundas palabras representan una interpretación semántica respectiva de la una o más primeras palabras y asocian la una o más primeras palabras con un concepto de un dominio respectivo entre una pluralidad de dominios predefinidos.
- 12El sistema de conformidad con cualesquiera de las reivindicaciones 9-11, caracterizado porque el eco y la paráfrasis se presentan en una pantalla de conversación, y en donde la paráfrasis incluye el al menos un valor del al menos 270 un parámetro presentado en un formato destacAde- INSTITUTO MEXICANO DE LA PROPIEDAD INDUSTRIAL
- 13El sistema de conformidad con cualesquiera de las reivindicaciones 9-12, en donde la paráfrasis indica un progreso en tiempo real de la tarea.
- 14El sistema de conformidad con cualesquiera de las reivindicaciones 9-12, en donde la paráfrasis proporciona un resumen de un conjunto de resultados, y en donde la paráfrasis se presenta además del conjunto de resultados.
- 15El sistema de conformidad con la reivindicación 14, caracterizado porque la paráfrasis especifica unos o más parámetros usados para obtener el conjunto de resultados.
- 16El sistema de conformidad con cualesquiera de las reivindicaciones 9-12, caracterizado porque la paráfrasis especifica los criterios de selección múltiples derivados de la petición de usuario, e indica que no se ha encontrado ningún resultado que cumplía todos los criterios de selección múltiples, y en donde la paráfrasis especifica una forma relajada de los criterios de selección múltiples y presenta un resumen de los resultados obtenidos basados en la forma relajada de los criterios de selección múltiples.
- 17Un medio legible por computadora que tiene datos ahí almacenados, los datos hacen que uno o más procesadores realicen operaciones que comprenden:recibir una petición de usuario, la petición de 271 IMPI INSTITUTO MEXICANO Dt LA PROPIEDAD INDUSTRIAL usuario incluye por lo menos una entrada de bahía rpcíhiHa Hp un usuario, en donde la entrada de habla incluye al menos un valor de al menos un parámetro;presentar al usuario un eco de la entrada de habla basado en una interpretación textual de la entrada de habla;determinar una tarea que será realizada por el dispositivo electrónico, desde una pluralidad de tareas realizables por el dispositivo electrónico, en donde la tarea es determinada al realizar procesamiento de lenguaje natural en la entrada de habla;y presentar al usuario una paráfrasis de la tarea, incluyendo una formulación en lenguaje natural de la tarea y el al menos un valor del al menos un parámetro.
- 18El medio legible por computadora de conformidad con la reivindicación 17, caracterizado porque una oración respectiva usada en la paráfrasis se adapta a una modalidad respectiva usada para presentar la paráfrasis al usuario, y en donde una oración más completa se utiliza en la paráfrasis cuando la paráfrasis se va a presentar en forma audible que cuando la paráfrasis se va a presentar en forma escrita.
- 19El medio legible por computadora de conformidad con cualesquiera de las reivindicaciones 17-18, caracterizado porque el eco incluye una o más primeras palabras y la paráfrasis incluye una o más segundas palabras, 272 IMPI ΧΓΠτυτο mexicano Dt LA PROPItDAD INDUSTWAL y en donde la una o más segundas palabras rppi-pspntan una interpretación semántica respectiva de la una o más primeras palabras y asocian la una o más primeras palabras con un concepto de un dominio respectivo entre una pluralidad de dominios predefinidos.
- 20El medio legible por computadora de conformidad con cualesquiera de las reivindicaciones 17-19, caracterizado porque el eco y la paráfrasis se presentan en una pantalla de conversación, y en donde la paráfrasis incluye el al menos un valor del al menos un parámetro presentado en un formato destacado.
- 21El medio legible por computadora de conformidad con cualesquiera de las reivindicaciones 17-20, en donde la paráfrasis indica un progreso en tiempo real de la tarea.
- 22El medio legible por computadora de conformidad con cualesquiera de las reivindicaciones 17-20, en donde la paráfrasis proporciona un resumen de un conjunto de resultados, y en donde la paráfrasis se presenta además del conjunto de resultados.
- 23El medio legible por computadora de conformidad con la reivindicación 22, caracterizado porque la paráfrasis especifica unos o más parámetros usados para obtener el conjunto de resultados.
- 24El medio legible por computadora de 273 IMPÍ ustituto MsriCANo De LA PROPIEDAD INDUSTRIAL conformidad con cualesquiera de las reivindicaciones 17-20, caracterizado porque la paráfrasis especifica los criterios de selección múltiples derivados de la petición de usuario, e indica que no se ha encontrado ningún resultado que cumplía 5 todos los criterios de selección múltiples, y en donde la paráfrasis especifica una forma relajada de los criterios de selección múltiples y presenta un resumen de los resultados obtenidos basados en la forma relajada de los criterios de selección múltiples. 10
- 25El método de conformidad con la reivindicación 1, que además comprende presentar al usuario, concurrentemente con la paráfrasis de la tarea, un conjunto de resultados generado como un resultado de realizar la tarea. 15
- 26El método de conformidad con la reivindicación 1, en donde presentar la paráfrasis de la tarea incluye presentar un progreso en tiempo real de la tarea, el método además comprende:recuperar un conjunto de resultados;cesar de presentar la paráfrasis de la tarea;y después de cesar de presentar la paráfrasis de la tarea, presentar un resumen de un conjunto de resultados, en donde el resumen del conjunto de resultados incluye una formulación en lenguaje natural de la tarea y el al menos un valor del al menos un parámetro. 274 IMPI INSTITUTO MMiCaNO DE LA PROPIEDAD INDUSTRIAL
- 27El método de conformidad con la reivindicación 1, en donde el eco de la entrada de habla se presenta visualmente, y la paráfrasis se presenta a través de una salida de habla.
- 28El método de conformidad con la reivindicación 27, en donde el eco de la entrada de habla no se presenta a través de la salida de habla, y la paráfrasis se presenta visualmente.
- 29El método de conformidad con la reivindicación 27, que además comprende presentar visualmente un conjunto de resultados generado como un resultado de la realización de la tarea.
- 30El sistema de conformidad con la reivindicación 9, en donde las operaciones además comprenden presentar al usuario, concurrentemente con la paráfrasis de la tarea, un conjunto de resultados generado como un resultado de realizar la tarea. 31. El sistema de conformidad con la reivindicación 9, en donde presentar la paráfrasis de la tarea incluye presentar un progreso en tiempo real de la tarea, las operaciones además comprenden:recuperar un conjunto de resultados;cesar de presentar la paráfrasis de la tarea;y después de cesar de presentar la paráfrasis de la tarea, presentar un resumen de un conjunto de resultados, en donde 275 IMPI INSTITUTO MKCANO DE LA mOFWMD INDUSTRIAL el resumen del conjunto de resultados ínnlnya -una fnrmnian'ón en lenguaje natural de la tarea y el al menos un valor del al menos un parámetros.
- 3132. El medio legible por computadora de conformidad con la reivindicación 17, en donde las operaciones además comprenden presentar al usuario, concurrentemente con la paráfrasis de la tarea, un conjunto de resultados generado como un resultado de realizar la tarea.
- 3233. El medio legible por computadora de conformidad con la reivindicación 17, en donde presentar la paráfrasis de la tarea incluye presentar un progreso en tiempo real de la tarea, las operaciones además comprenden:recuperar un conjunto de resultados;cesar de presentar la paráfrasis de la tarea;y después de cesar de presentar la paráfrasis de la tarea, presentar un resumen de un conjunto de resultados, en donde el resumen del conjunto de resultados incluye una formulación en lenguaje natural de la tarea y el al menos un valor del al menos un parámetro. 276 •iSTlTUTO MEXICANO DE LA PROPIEDAD INDUSTRIAL
Independent claims32
2,691 paragraphs in 347 sections, as filed
(54) Title: USER APPLICATION PARAPHRASIS AND RESULTS BY DIGITAL AUTOMATED ASSISTANT. (54) Title: INTELLIGENT AUTOMATED ASSISTANT.
(57) Summary
An intelligent automated assistant system gathers with the user in an integrated way, as a conversation using natural language, and invokes external services when appropriate, to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the network, email, smartphone, and the like, or any combination thereof. In one embodiment, the system relies on correlated domain and task sets, and employs additional functional power by external services with which the system can interact.
(57) Abstract
An intelligent automated assistanl system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external Services when appropriate to obtain Information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external Services with which the system can interact.
<img file="MX338784B_D0001.tif" />
Π Mexican Institute of Property Q Industrial BS
MÍWIMÜADfHtMtMH
PATENT TITLE NO. 338784
Owner (s): APPLE INC.
Address: 1 Infinite Loop, Cupertino, California, 95014, USA
Denomination:
Classification:
Inventi
USER APPLICATION PARAPHRASIS
DIGITAL AUTOMATED ASSISTANT. *
TI '(ST?
lnt.CI.8: G06F17 / 28; G06F17 / 30; G06F3 / 16; G06F9 / 54; G10L15 / 00; G10L15 / 18; G10L15 / 22; G10L15 / 26; G10L21 / 06
THOMAS ROBERT GRUBER; HARRY JOSEPH SADDLER; ADAM JOHN CHARD
AND RESULTS BY
*
MX / a / 2012 / 01l |) 0
<img file="MX338784B_D0002.tif" />
REQUEST
International filing date: January 11, 2011 Patent Patent Number: 322428 s
<td> 1 <sup>Country:</sup> ></td><td></td>
<td> 1 <sup>us</sup></td><td></td>
<td>i <sup>us</sup></td><td></td>
<td>Security: Twenty</td><td>binos</td>
<td>'miss Vencí</td><td>nient</td>
<td>i reference patent sd</td><td>Morga co</td>
<td>e in accordance with article</td><td>the 23 of</td>
<td>exited from the fedj</td><td>of pres</td>
<td>stretches. i</td><td></td>
<td>who subscribes the presenti</td><td>title what</td>
<td>Industrial property (Daily</td><td>Officer d</td>
<td> 5/01/2004, 16/06/2005, 2^</td><td> 01/2006,</td>
PRIORITY
Date:
January 2010 January 10, 2011
Number:
61/295,774
12/987,982
<img file="MX338784B_D0003.tif" />
ptút'ij. ·., · ¡iSnlll.y 59 of 0 Ú Hiustrial Property.
I Law of the Pro] litation of the Sun people patented hg ^ UMlrigemla ^ jneinte Imprc ogables years, to keep the
Ions lll and 7 ° bis 2 of I based on I ________ the Federation (DOF) 06/27 / Τ9ΡΐΡ · Β · ΒΒ ··· ΙΒΗ594, 10/25/1996, 12/26/1997, 1 * 5/1999 , 05/01/2009, 01/06/2010, 06/18/2010, 06/28/2010, 01/27/2012 and 04/09/2012); articles 1, 3 action V (12/1999, amended on 07/01/2002, 07/15/2004, 07/28/2004 and 09/07/2007); Articles 1, 3, 4, ¿“fraction V subsection a), sub subsection iii), 16 sections I and III and 30 of the Organic Statute of the Mexican Institute of Industrial Property (DOF 12/27/1999, amended on 10/10/2002, 07/29/2004, 08/04/2004 and 09/13/2007); 1, 3 and 5 paragraph a) and antepenultimate paragraph of the Agreement that delegates powers to the Deputy Directors General. Coordinator. Divisional Directors, Regional Office Holders, Divisional Deputy Directors. Departmental Coordinators and other subordinates of the Mexican Institute of Industrial Property. (DOF 12/15/1999, amended on 02/04/2000, 07/29/2004, 08/04/2004 and 09/13/2007).
hey of the
Issue Date: May 2, 2016
DIVISIONAL DEPUTY DIRECTOR OF EXAMINATION OF PATENT FUND, ELECTRICAL AREAS AND PE PFfiiSTWQS OF INDUSTRIAL DESIGNS AND
<img file="MX338784B_D0004.tif" />
<img file="MX338784B_D0005.tif" />
<img file="MX338784B_D0006.tif" />
Arenal No. 550. Floor 1,
Coi. Sonta Mana Tepepan village.
Xochimilco. uP 16020,
Mexico City
Γ ei. (55; 53 34 0 '/ 00 www irnpj ¿x / b.nix
MX / 2016/34925 r3c?>
35818η
IMPI
MEXICAN INSTITUTE LH LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0007.tif" />
PARAGRAPH OF USER REQUESTS AND RESULTS
BY DIGITAL AUTOMATED ASSISTANT
CROSS REFERENCE TO RELATED REQUESTS
This application claims priority from US Provisional Patent Application Serial No. 61 / 295,774 for Intelligent Automated Assistant, agent file number SIRIP003P, filed on January 18, 2010, which is incorporated herein by reference.
This application further claims priority of US Utility Patent Application Serial No. DE 12 / 987,982 for Intelligent Automated Assistant, agent file number P10575US1, filed on January 10, 2011, which is incorporated herein by reference.
<td></td><td>This</td><td>request</td><td>also I know</td><td>refers</td><td>to</td><td>Request</td><td>of</td>
<td>Patent</td><td>the</td><td>USA No.</td><td>. serial</td><td colspan="2">11 / 518,292 for</td><td>Method</td><td>and</td>
<td>Apparatus</td><td>for</td><td>Building</td><td colspan="3">an Intelligent Automated</td><td colspan="2">Assistant,</td>
<td>filed</td><td>in</td><td>September</td><td> 8, 2006,</td><td>That</td><td colspan="2">incorporates here</td><td>by</td>
<td>reference.</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
<td></td><td>This</td><td>request</td><td>also I know</td><td>refers</td><td>to</td><td>Request</td><td>of</td>
US Provisional Patent Serial No. 61 / 186,414 to System and Method for Semantic Auto-Completion, filed June 12, 2009, which is incorporated herein by reference.
FIELD OF THE INVENTION
The present invention relates to intelligent systems, and more specifically for classes of
IMPI
INSTITUTO MEXICANO Di LA BROBIiDAD industrial
<img file="MX338784B_D0008.tif" />
smart automated wizard apps.
background of the invention
Today's electronic devices are capable of accessing a large, growing and diverse number of functions, services and information, both on the Internet and from other sources. The functionality of these devices is incrementally fast, as many consumer devices, smartphones, tablet computers and the like, are capable of running software applications to perform various tasks and provide different types of information. Often, each application, function, website, or feature has its own user interface and its own operational paradigms, many of which may be difficult or burdensome to learn, or overwhelming for users.
In addition, many users may have difficulty even discovering what functionality and / or information is available on their electronic devices or in various web sites; In this way, these users may become frustrated or overwhelmed, or simply unable to use the resources available to them effectively.
In particular, novice users or individuals who are handicapped or disabled in a certain way, and / or are elderly, busy, distracted and / or operate a vehicle, may have difficulty effectively interfacing with their electronic devices, and / or couple with online services
<img file="MX338784B_D0009.tif" />
ΪΜΡΙ
MEXICAN INSTITUTE
Dt LA RROFIIDAD
INDUSTRIAL indeed. These users in particular are likely to have difficulty with the large number of inconsistent and diverse functions, applications, and network sites that may be available for use.
Accordingly, existing systems are often difficult to use and navigate, and current users often have overwhelming and inconsistent interfaces that often prevent users from making effective use of technology.
COMPENDIUM
In accordance with various embodiments of the present invention, an automated electing int wizard is implemented in an electronic device, to facilitate user interaction with a device, and to assist the user in coupling more effectively with local and / or remote services . In various modalities, the intelligent automated wizard engages the user in an integrated form of conversation using natural language dialogue, and invokes external services when appropriate to obtain information or perform various actions.
In accordance with various embodiments of the present invention, the intelligent automated assistant integrates a variety of capabilities that are provided by different *
software or program components (for example, to support dialogue and natural language recognition, multimodal feeding, handling of personal information, handling of
<img file="MX338784B_D0010.tif" />
Furthermore, in order to offer intelligent interfaces and useful functionality for users, the intelligent automated assistant of the present invention can, in at least some embodiments, coordinate these components and services. The conversation interface, and the ability to obtain information and perform follow-up tasks, are implemented, in at least some modalities, by coordinating various components such as language components, dialogue components, task management components, management components. information and / or a plurality of external services.
In accordance with various embodiments of the present invention, intelligent automated assistant systems may be configured, designed, or operable, to provide various different types of operations, functionality, and / or features, and / or to combine a plurality of features, operations, and applications of an electronic device on which they are installed. In some embodiments, the intelligent automated assistant systems of the present invention may perform any or all of: actively producing a user's feed, interpreting the user's intention, clarifying or reducing ambiguity between competing interpretations, requesting and receiving clarification information as required and perform (or initiate) actions based on discerned intention. Actions can be performed, for example
<img file="MX338784B_D0011.tif" />
IMPI
INSTITUTO MEXICANO DE LA FROFIEDAD INDUSTRIAL by activating and / or interfacing. with any option BSi. .or. services that may be available on an electronic device, as well as services available on an electronic network such as the Internet. In various ways, this activation of external services can be done by APIs or by any other convenient mechanism. In this way, the intelligent automated assistant systems of various embodiments of the present invention can unify, simplify and improve the user experience with respect to many different applications and functions of an electronic device, and with respect to the services that may be available on the Internet. . In this way the user can be relieved of the burden of learning what functionality may be available in the device of the services connected to the network, how to interface with these devices to obtain what they want, and how to interpret the output received from these services; rather, the wizard of the present invention can act as an intermediary between the user and these various services.
Furthermore, in various embodiments, the wizard of the present invention provides a conversation interface that the user may find more intuitive and less troublesome than conventional graphical user interfaces. The user can participate in a form of conversation dialogue with the wizard using any of a number of available input and output mechanisms, such as speech,
MtXiCANO INSTITUTE
OF THE PROPERTY
INDUSTRIAL
<img file="MX338784B_D0012.tif" />
graphical user interfaces (buttons and link), diii'LiLiiluuÍLM>
text, and the like. The system can be implemented using any of a number of different platforms, such as API devices, the network, email, and the like, or any of their combinations. Requests for additional feed can be presented to the user in the context of such conversation. Short and long-term memory can be coupled in such a way that the user's power can be interpreted in the correct context given previous events and communications within a given session, as well as historical and profile information regarding the user.
In addition, in various embodiments, context information derived from user interaction with a feature, operation, or application on one device can be used to simplify the operation of other features, operations, or applications on the device or on other devices. For example, the smart automated assistant can use the context of a phone call (such as the person being called) to simplify the initiation of a text message (for example, to determine that the text message should be sent to the same person, without the user having to explicitly specify the recipient of the text message). The intelligent automated assistant of the present invention can thus interpret instructions such as sending you a text message, where it is interpreted according to the
IMPI
MÍXICANO INSTITUTE DS INDUSTRIAL PROPERTY
<img file="MX338784B_D0013.tif" />
context information derived from a current phone mTflddd, and / or any feature, operation, or application on the device. In various modalities, the intelligent automated wizard takes into account various types of context data available to determine which address book contact to use, which contact data to use, which telephone number for the contact, and the like, such that the user does not need to manually reenter this information.
In various modes, the wizard can also take external events into account and respond accordingly, for example to initiate action, initiate communication with the user, provide alerts, and / or modify a previously initiated action in view of external events. If power is required from the user, a conversation interface can again be employed.
In one embodiment, the system relies on interrelated games or sets of domains and tasks, and employs additional functional feeding by external services with which the system can interact. In various forms, these external services include network-enabled services, as well as functionally related to the physical equipment device itself. For example, in a mode where the Smart Automated Assistant is implemented on a smartphone, personal digital assistant, tablet computer, or
<img file="MX338784B_D0014.tif" />
INSTITUTO MEXICANO DE LA PROPIEDAD INDUSTRIAL another device, the assistant can control iHUdlldij 'qpidtau'iuinLj and device functions, such as dial a phone number, send a text message, make reminders, add events to a calendar, and the like.
In various embodiments, the system of the present invention can be implemented to provide assistance in any of a number of different domains. Examples include:
• Local Services (including specific time and location services such as restaurants, movies, automated teller machines (ATMs), events, and meeting places);
• Social and Personal Memory Services (including action items, notes, calendar events, shared links, and the like);
• Electronic commerce (including online purchases of items such as books, DVDs, music, and the like);
• Travel Services (including flights, hotels, attractions and the like).
A person skilled in the art will recognize that the list of previous domains is only exemplary. Furthermore, the system of the present invention can be implemented in any combination of domains.
In various modalities, the intelligent automated assistant systems described here can be configured
IMPI
INSTITUTO MWICANO DI LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0015.tif" />
or designed to include functionality for “* auLuiÍÍat'ÍLÍ! ii '1« e.
application of data and services available on the Internet, to discover, find, select between, purchase, reserve or order products and services. In addition to automating the process of using these data and services, at least one smart automated assistant system mode described here, you can also enable the combined use of multiple data sources and services right away. For example, you can combine product information from multiple review sites, check pricing and availability from multiple resellers and verify their location and time constraints, and help a user find a customized solution to their problem.
Additionally, at least one form of intelligent automated assistant system described herein can be configured or designed to include functionality to automate the use of data and services available on the Internet to discover, investigate, select between, book and otherwise learn about things by do (including but not limited to movies, events, performances, exhibits, shows, and attractions); places to go (including but not limited to travel destinations, hotels and other places to stay or to stay, highlights and other places of interest, etc.); places to eat or drink (such as restaurants and bars), hours and places to meet others, and any other source of entertainment or social interaction that can be found at
IMPI
INJTITUTO MEXICANO Dt LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0016.tif" />
Internet. Additionally, at least one form of smart automated attutiCe, described here, can be configured or designed to include functionality to allow the operation of applications and services through natural language dialog, which may otherwise be provided by dedicated applications with graphical user interfaces including search (including location-based search); navigation (maps and directions); database search (such as finding businesses or people by name or other properties); obtain weather conditions and forecasts, check the price of items on the market or the status of financial transactions; check traffic or flight status; access and update calendars and programs; manage reminders, alerts, tasks and projects;
communicate by email or other messaging platforms; and operate devices locally or remotely (for example, dial phones, control lighting and temperature, control home security devices, play music or video, etc.). Furthermore, at least one type of intelligent automated assistant system described herein can be configured or designed to include functionality to identify, generate and / or provide personalized recommendations for activities, products, services, source of entertainment, time management, or any other type of recommendation service that benefits from interactive dialogue in natural language and automated access to data and services.
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0017.tif" />
In various embodiments, the intelligent automated assistant of the present invention can control many features and operations of an electronic device. For example, the Smart Automated Assistant can call services that interface functionality and applications on a device through APIs or by other means, to perform functions and operations that may otherwise be initiated using a conventional user interface on the device. These features and operations may include, for example, setting an alarm, making a phone call, sending a text message or email, adding an event to the calendar, and the like. These functions and operations can be performed as additional functions in the context of a conversation dialogue between a user and the assistant. These functions and operations can be specified by the user in the context of this dialog, or can be performed automatically based on the context of the dialog. A person skilled in the art will recognize that the assistant in this way can be used as a control mechanism to initiate and control various operations on the electronic device, which can be used as an alternative to conventional mechanisms such as graphical user interfaces or buttons. .
BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings illustrate various embodiments of the invention and together with the description, serve to explain the
ΙΜΡΙ
MEXICAN INSTITUTE Ot THE INDUSTRIAL property
<img file="MX338784B_D0018.tif" />
principles of the invention according to ILlUlldlldaides. <sup>1</sup> A person skilled in the art will recognize that the particular embodiments illustrated in the drawings are only exemplary, and are not intended to limit the scope of the present invention.
Figure 1 is a block diagram illustrating an example of an embodiment of an intelligent automated assistant system.
Figure 2 illustrates an example of an interaction between
<td>a user and an assistant</td><td>automated</td><td>intelligent</td><td>of</td><td>agreement</td><td>with</td>
<td>at least one modality.</td><td></td><td></td><td></td><td></td><td></td>
<td>Figure 3 is</td><td>a diagram</td><td>of blocks</td><td>than</td><td>illustrates</td><td>a</td>
<td colspan="2">computing device suitable for</td><td>implement</td><td>to the</td><td>less</td><td>a</td>
portion of a smart automated wizard according to at least one modality.
Figure 4 is a block diagram illustrating an architecture for implementing at least a portion of an intelligent automated assistant in a stand-alone computing system in accordance with at least one embodiment.
<td>The</td><td>Figure 5 is a</td><td>diagram</td><td>of</td><td>blocks</td><td>illustrating</td><td>a</td>
<td>architecture</td><td>to implement</td><td>at least</td><td>a</td><td>portion</td><td colspan="2">from an assistant</td>
<td>automated</td><td>smart in</td><td>a network</td><td>of</td><td>calculation</td><td>distributed</td><td>of</td>
<td colspan="3">according to at least one modality.</td><td></td><td></td><td></td><td></td>
<td>The</td><td>Figure 6 is a</td><td>diagram</td><td>of</td><td>blocks</td><td>illustrating</td><td>a</td>
system architecture illustrating several different types of clients and modes of operation.
MEXICAN INSTITUTE
OF THE PROPERTY
INDUSTRIAL
<img file="MX338784B_D0019.tif" />
Figure 7 is a diagram of hlng »ae. gno iinai-ra nn client and a server, which communicate with each other to implement the present invention according to one embodiment.
Figure 8 is a block diagram illustrating a fragment of an active ontology according to one embodiment.
Figure 9 is a block diagram illustrating an example of an alternate embodiment of an intelligent automated assistant system.
Figure 10 is a flow chart illustrating a method of operating one or more components that produce active power in accordance with one embodiment.
FIG. 11 is a flow chart illustrating a method of producing active type feed according to one embodiment.
Figures 12 to 21 are screen shots or snapshots illustrating some portions of some of the procedures for active type power production according to one embodiment.
Figure 22 is a flowchart illustrating a method of producing active power for voice or speech power according to one embodiment.
Figure 23 is a flowchart illustrating a method of producing active power for GUI-based power in accordance with one embodiment.
Figure 24 is a flow chart illustrating a
------- LX
MEXICAN INSTITUTE '¿¿• «SK'WfiSÍS ¿A OF THE PROPERTY
INDUSTRIAL level of a flow of flow that illustrates a relevant events of flow that illustrates a method to produce active feed to dialogue according to a modality.
Figure 25 is a method diagram for actively monitoring in accordance with one embodiment.
Figure 26 is a method diagram for producing multimodal active power according to one embodiment.
Figure 27 is a set of screen shots illustrating an example of various types of functions, operations, actions, and / or other features that can be provided by one or more components of domain models and services that are orchestrated according to a modality.
Figure 28 is a flowchart illustrating an example of a method for natural language processing according to one embodiment.
Figure 29 is a screen shot illustrating natural language processing according to one embodiment.
Figures 30 and 31 are screen shots illustrating an example of various types of functions, operations, actions, and / or other features that can be provided by one or more dialogue flow processor components in accordance with one embodiment.
Figure 32 is a flowchart illustrating an operating method for one or more processor components of
IMPI
INITITI ITO MSXICANO DE LA PROHEDAD INDUSTRIAL
<img file="MX338784B_D0020.tif" />
dialogue flow according to a modality.
<td>The</td><td>Figure</td><td> 33</td><td>is a</td><td>diagram of</td><td>flow</td><td>than</td><td>illustrates</td><td>a</td>
<td>process</td><td colspan="2">call</td><td colspan="3">and automatic response of</td><td colspan="2">agree with</td><td>a</td>
<td>modality.</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
<td>The</td><td>Figure</td><td> 34</td><td>is a</td><td>diagram of</td><td>flow</td><td>than</td><td>illustrates</td><td>a</td>
<td>example of</td><td>flow</td><td>of</td><td>chores</td><td>for one</td><td>homework</td><td>for</td><td colspan="2">selection</td>
restricted according to a modality.
Figures 35 and 36 are screenshots illustrating an example of the operation of the restricted selection task according to one embodiment.
Figure 37 is a flow chart illustrating an example of a procedure for executing a service orchestration procedure according to one embodiment.
Figure 38 is a flowchart illustrating an example of a service invocation process according to one embodiment.
Figure 39 is a flowchart illustrating an example of a multi-phase output procedure in accordance with one embodiment.
Figures 40 and 41 are screen shots illustrating examples of output processing according to one embodiment.
Figure 42 is a flowchart illustrating an example of multimodal output processing according to one embodiment.
Figures 43A and 43B are screen shots illustrating
<img file="MX338784B_D0021.tif" />
IMPI
MEXICAN INSTITUTE
OF THE PROPERTY
INDUSTRIAL an example of the use of one or more short-term memory components · to maintain dialogue context while changing location, according to a modality.
Figures 44A to 44C are screen shots illustrating an example of the use of one or more long-term personal memory components, in accordance with one embodiment.
Figure 45 illustrates an example of an abstract model for a restricted selection task.
<td colspan="2">The figure</td><td>46 illustrates</td><td colspan="2">an example of a flow model</td><td>of</td>
<td>dialogue for</td><td>help</td><td>to guide the</td><td>user through a</td><td>process</td><td>of</td>
<td>search.</td><td></td><td></td><td></td><td></td><td></td>
<td>The</td><td>Figure</td><td>47 is a</td><td>flow chart that</td><td>illustrates</td><td>a</td>
restricted selection method according to a modality.
DETAILED DESCRIPTION OF THE MODALITIES
Various techniques will now be described in detail with reference to a few exemplary embodiments thereof, as illustrated in the accompanying drawings. In the following description, numerous specific details are set forth to provide a complete understanding of one or more aspects and / or characteristics described or referred to herein. It will be apparent however, to a person skilled in the art, that one or more aspects and / or characteristics described or referred to herein may be practiced without some or all of these specific details. In other cases, well-known process steps and / or structures have not been described in detail so as not to block or obscure
IMPI
INSTITUTO mbxicano Dt LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0022.tif" />
some of the aspects and / or characteristics described referred to here.
One or more different inventions can be described in the present application. Furthermore, for one or more of the inventions described herein, numerous embodiments can be described in this patent application, and are presented for illustrative purposes only. The described modalities are not intended to be limiting in any way. One or more of the invention (s) can be broadly applicable to numerous embodiments, as will be readily apparent from the description. These modalities are described in sufficient detail to enable those skilled in the art to practice one or more of the inventions, and it will be understood that other modalities may be used and that structural, logical, software, electrical, and other changes may be carried out without departing from the scope of one or more of the invention (s). Accordingly, those skilled in the art will recognize that the one or more of the invention (s) can be practiced with various modifications and alterations. Particular features of one or more of the invention (s) may be described with reference to one or more particular embodiments or figures that are part of the present description, and where, by way of illustration, specific embodiments of one or more of the invention are shown. or inventions. It should be understood, however, that these features are not limited to use in one or more modalities
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0023.tif" />
individuals or figures with reference to the cUdl ^ and Sé describe. The present description is neither a literal description of all the modalities of one or more of the invention (s) nor is it a list of characteristics of one or more of the invention (s) that must be present in all the modalities.
The section headings provided in this patent application and the title of this patent application are for convenience only, and are not to be construed as limiting the description in any way.
Devices that are in communication with each other do not need to be in continuous communication with each other, unless expressly specified otherwise. Furthermore, devices that are in communication with each other can communicate directly or indirectly through one or more intermediaries.
A description of a modality with several components in communication with one another does not imply that all of these components are required. Rather, a variety of optional components are described to illustrate the wide variety of possible embodiments of one or more of the invention (s).
Furthermore, although process steps, method steps, algorithms, or the like may be described in sequential order, these processes, methods, and algorithms may be configured to work in alternate orders. In other words, any sequence or order of steps that can be described in this patent application does not, in and of itself, indicate a
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0024.tif" />
Requirement that the steps be carried out in r-'r- Jjüo otrpn ^ of described processes can be performed in any practical order. Furthermore, some stages may be performed simultaneously despite being described or implied that they occur non-simultaneously (for example, because one stage is described after the other stage). Furthermore, the illustration of a process when presented in a drawing does not imply that the illustrated process is exclusive of other variations and modifications to it, it does not imply that the illustrated process or any of its stages are necessary for one or more of the or inventions, and does not imply that the illustrated process is preferred.
When describing a single device or item, it will be readily apparent that more than one device / item (whether cooperating or not) can be used in lieu of a single device / item. Similarly, when more than one device or article is described (whether they cooperate or not), it will be readily apparent that a single device / article can be used in place of more than one device or article.
The functionality and / or features of a device may be alternately incorporated by one or more other devices that are not explicitly described as having such functionality / features. Thus, other embodiments of one or more of the invention (s) do not require the device itself to be included.
Techniques and mechanisms described or referred to here in «
t
MEXICAN INSTITUTE
DS THE PROPERTY
INDUSTRIAL • ·
<img file="MX338784B_D0025.tif" />
occasions will be described in the singular by clariTTyCh <sup>1</sup> Without uiuLa.i! And 'y it should be noted that particular modalities include multiple iterations of a technique or multiple instances of a mechanism unless noted otherwise.
Although described within the context of intelligent automated assistant technology, it can be understood that the various aspects and techniques described herein (such as those associated with active ontologies, for example) may also be deployed and / or applied in other fields of technology involving interaction with computerized and / or human software.
Other aspects regarding intelligent automated assistant technology (for example, which can be used by, provided by, and / or implemented in one or more modalities of intelligent automated assistant system described herein), are described in one or more of the following references :
• US Provisional Patent Application No.
Series 61 / 295,774 for Intelligent Automated Assistant, agent file number SIRIP003P, filed on January 18, 2010, the description of which is incorporated herein by reference;
<td> •</td><td>Application</td><td>Patent</td><td>the</td><td>USA</td><td>No.</td><td>serial</td>
<td> 11/518,292</td><td>for Method</td><td>And Apparatus</td><td>for</td><td>Building</td><td>an</td><td>Intelligent</td>
<td>Automated</td><td>Assistant,</td><td>presented in</td><td colspan="2">September</td><td> 8,</td><td>2006, the</td>
description of which is incorporated herein by reference; and • US Provisional Patent Application No.
IMPI
MÍXICANO INSTITUTE DS INDUSTRIAL PROPERTY
<img file="MX338784B_D0026.tif" />
Series 61 / 186,414 for System and Method tor Semant'lí! AUtO- 'Completion, filed June 12, 2009, the description of which is incorporated herein by reference.
Physical Equipment Architecture
In general, the intelligent automated assistant techniques described here can be implemented on hardware or a combination of software and hardware. For example, they can be implemented in an operating system kernel, in a separate user process in a linked library package in network applications, on a specially built machine, or on a network interface card. In a specific embodiment, the techniques described herein can be implemented in software, such as an operating system or an application running on an operating system.
One or more hybrid hardware / software implementations of at least some of the intelligent automated assistant mode (s) described herein may be implemented on a programmable machine activated or selectively reconfigured by a computer program stored in memory. These network devices can have multiple network interfaces that can be configured or designed to use different types of network communication protocols.
A general architecture for some of these machines may appear from the descriptions presented here. According to specific modalities, at least some of the characteristics
IMPI
<img file="MX338784B_D0027.tif" />
<img file="MX338784B_D0028.tif" />
and / or functionalities of the various modalities * d¿5 Ul! The intelligent automated ujiatonta described herein can be implemented in one or more general purpose network host machines such as end user computer system, computer, server or network server system, mobile computing device (eg personal digital assistant, telephone mobile, smartphone, laptop, tablet computer or the like), consumer electronic device, music player, or any other convenient electronic device, router, switch, or the like, or any combination thereof. In at least some modalities, at least some of the features and / or functionalities of the various intelligent automated assistant modalities described herein can be implemented in one or more virtualized computing environments (eg, network computing clouds, or the like) .
Now referring to Figure 3, a block diagram is shown illustrating a computing device 60 suitable for implementing at least a portion of the intelligent automated assistant features and / or functionality described herein. The computing device 60 for example may be an end user computer system, server or network server system, mobile computing device (eg personal digital assistant, mobile phone, smartphone, laptop, tablet computer or
<img file="MX338784B_D0029.tif" />
IMPI • βίΐτυτο MÍXICANO
Say THE PROPERTY
INDUSTRIAL SIMILAR), electronic consumer device, 'Jf &' piU (lLTt.'Lur of music or any other convenient electronic device or any combination or portion thereof. The computing device 60 can be adapted to communicate with other devices , such as clients and / or servers, over a communication network such as the Internet, using known protocols for said communication, whether wireless or wired.
In one embodiment, computing device 60 includes central processing unit, UPC (CPU = Central Processing Unit) 62, interfaces 68, and a busbar 67 (such as a busbar for peripheral component interconnect (PCI = Peripheral Component Interconnect) )). When operated under the control of software or firmware, UPC 62 may be responsible for implementing specific functions associated with the functions of a specifically configured computing device or machine. For example, in at least one mode, a user's Personal Digital Assistant (PDA) can be configured or designed to function as a smart automated assistant system using CPU 62, memory 61, 65, and interface (s). 68. In at least one embodiment, CPU 62 may be caused to perform one or more of different types of functions and / or intelligent automated wizard options under the control of software modules / components, which may include an operating system, for example. and any application software
ΙΜΡΙ
MEXICAN INSTITUTE
Dt THE INDUSTRIAL PROPERTY
<img file="MX338784B_D0030.tif" />
appropriate, controllers and the like.
CPU 62 may include one or more processors 63 such as for example a processor from the Motorola or Intel family of microprocessors or the MIPS family of microprocessors. In some embodiments, the processor (s) 63 may include specially designed hardware (eg, application specific integrated circuits (ASICs), electrically erasable programmable read only memories (EEPROMs), field programmable gate arrays (FPGAs = Field- Programmable Gate Arrays) and the like) to control the operations of the computing device 60. In a specific embodiment, memory 61 (such as nonvolatile random access memory (RAM) and / or read-only memory (ROM)) is also part of CPU 62. However, there are many different ways in which memory can be attached to the system. Memory block 61 can be used for a variety of purposes such as for example temporary memory storage and / or storing data, programming instructions and the like.
As used herein, the term processor is not limited only to those integrated circuits that refer to the art as a processor, but broadly refers to a microcontroller, a microcomputer, a programmable logic controller, an application-specific integrated circuit, and any another programmable circuit.
In one embodiment, interfaces 68 are provided as
IMPI
MEXICAN INSTITUTE Say THE INDUSTRIAL PROPERTY
<img file="MX338784B_D0031.tif" />
interface cards (sometimes referred to — eamo · ^ ¡nr ds line). In general, they control the sending and receiving of data packets over a computer network and sometimes support other peripherals used with the computing device 60. Among the interfaces that can be provided are Ethernet interfaces, frame relay interfaces, cable interfaces , DSL interfaces, token ring interfaces, and the like. In addition, various types of interfaces can be provided such as for example universal serial busbar (USB), Serial, Ethernet, Firewire, PCI, parallel, radio frequency (RF), Bluetooth ™, near-field communications (eg using magnetic components Near Field), 802.11 (WiFi), Frame Relay, TCP / IP, ISDN,
Fast Ethernet, Gigabit Ethernet interfaces, Asynchronous Transfer Mode (ATM) interfaces
Mode), interface for high-speed serial interfaces (HSSI = High-Speed Serial Interface), point-of-sale interfaces (POS =
Point of Sale), Fiber Data Distributed Interfaces (FUDIS) and the like. In general, these interfaces 68 may include appropriate gates for communication with the appropriate medium. In some cases, they can also include a standalone processor and in some cases, volatile and / or non-volatile memory (for example, RAM).
Although the system shown in Figure 3 illustrates a specific architecture for a computing device 60 for
IMPI
MEXICAN INSTITUTE DS INDUSTRIAL PROPERTY
<img file="MX338784B_D0032.tif" />
Implementing the techniques of the invention is hereby described, in no way is the only device architecture in which at least a portion of the features and techniques described here can be implemented. For example, architectures having one or any number of processors 63 may be employed, and these processors 63 may be present on a single device or distributed among any number of devices. In one embodiment, a single processor 63 handles communications as well as routes computations. In various modalities, different types' of intelligent automated assistant features and / or functionality can be implemented in an intelligent automated assistant system that includes a client device (such as a personal digital assistant or smartphone running client software), and one or more server systems (such as a server system described in more detail below).
Regardless of the configuration of the network device, the system of the present invention may employ one or more memories or memory modules (such as for example memory block 65), configured to store data, program instructions for network operations general purpose and / or other information regarding the functionality of the intelligent automated assistant techniques described here. Program instructions can control the operation of an operating system and / or one or more applications, for example. The
<img file="MX338784B_D0033.tif" />
IMPI
INSTITUTO MEXICANO Dt LA PROPIEDAD INDUSTRIAL memory or memories can also be used. specific program described here.
Because this program information and instructions can be used to implement the systems / methods described here, at least some network device modalities may include non-transient machine-readable storage media that, for example, can be configured or designed to store program instructions. , status information, and the like to perform various operations described herein. Examples of these non-transient machine-readable storage media include, but are not limited to, magnetic media such as hard drives, floppy drives, and magnetic tape; optical media such as CD-ROM discs; magneto-optical media such as rewritable optical discs, and hardware devices that are specially configured to store and perform program instructions, such as read-only memory (ROM) devices, flash memory, memristor memory, random access memory (RAM ) and the like. Examples of program instructions include both machine code, such as produced by a compiler, and files that contain higher-level code that can be executed by the computer using an interpreter.
IMPI
MEXICAN INSTITUTE Dt THE INDUSTRIAL PROPERTY
<img file="MX338784B_D0034.tif" />
In one modality, the system I introduced ínvérldlóll ¿SfcT implements in an autonomous computing system. Now referring to Figure 4, a block diagram is shown illustrating an architecture for implementing at least a portion of an intelligent automated assistant in a stand-alone computing system, according to at least one embodiment. The computing device includes one or more processors 63 that runs software to implement the intelligent automated assistant 1002. The power device 1206 may be of any type suitable for receiving user power, including for example a keyboard, touch screen, microphone (for example, for voice power), mouse, touch pad, bouncer, five-way switch , game controller (joystick), and / or any combination thereof. Output device 1207 can be a display, speaker, printer, and / or any combination thereof. Memory 1210 may be a random access memory having a structure and architecture as known in the art for use by one or more processors 63 in the course of executing software. The storage device 1208 can be any magnetic, optical and / or storage device for storing data in digital form; examples include flash memory, magnetic hard disk, CD-ROM and / or the like.
In another embodiment, the system of the present invention is implemented in a distributed computing network, such as that
<img file="MX338784B_D0035.tif" />
IMPI
MEXICAN INSTITUTE
OE THE PROPERTY
INDUSTRIAL that has any number of clients and / or <iorwidnrn<sub>and</sub>.
referring to Figure 5, a block diagram is shown illustrating an architecture for implementing at least a portion of an intelligent automated wizard in a distributed computing network, in accordance with at least one embodiment.
In the arrangement or arrangement shown in Figure 5, any number of clients 1304 are provided; every customer
1304 it can run software to implement client-side portions of the present invention. Furthermore, any number of servers 1340 can be provided to handle requests received from clients 1304. Clients 1304 and servers 1340 can communicate with each other via an electronic network 1361, such as the Internet. Network 1361 can be implemented using any known network protocols, including for example wired and / or wireless protocols.
Furthermore, in one embodiment, servers 1340 can call external services 1360 when required to obtain additional information or refer to storing data regarding previous interactions with particular users. Communications with external 1360 services can be carried out for example over the 1361 network. In various forms, external services 1360 include services and / or functionalities enabled by the network, related to or installed on the physical equipment device itself. For example, in a mode where wizard 1002 is implemented on a smartphone or other device
<img file="MX338784B_D0036.tif" />
IMPI
INSTITUTO MÍXICANO Oí LA PROPItOA »INDUSTRIAL
<td>electronic, stored in</td><td>the a</td><td colspan="2">assistant 1002 can calendar application</td><td>ob L CTigi<sup>1</sup>(app),</td><td>contacts and / or</td>
<td>other sources.</td><td></td><td></td><td></td><td></td><td></td>
<td>In</td><td colspan="2">various modalities,</td><td>the</td><td>assistant</td><td>1002 can</td>
control many features and operations of an electronic device on which it is installed. For example, wizard 1002 can call external 1360 services that interface functionality and applications on a device through APIs or by other means, to perform functions and operations that may otherwise be initiated using a conventional user interface on the device. . These features and operations may include setting an alarm, making a phone call, sending a text message or email, adding a calendar event, and the like. These functions and operations can be performed as add functions in the context of a conversation dialogue between a user and assistant 1002. These functions and operations can be specified by the user in the context of this dialog, or can be performed automatically based on the context of the dialog. A person skilled in the art will recognize that the assistant 1002 can thus be used as a control mechanism to initiate and control various operations on the electronic device, which can be used as an alternative to conventional mechanisms such as buttons or graphical user interfaces.
<img file="MX338784B_D0037.tif" />
For example, user can provide 'allausiifcaciót' -— to assistant 1002 such as I need to wake up tomorrow at 8 am.
Once wizard 1002 has determined the user's intention, using the techniques described herein, wizard 1002 can call external services 1340 to interface with an alarm clock function or application on the device.
Wizard 1002 sets the alarm on behalf of the user. In this way, the user can employ the assistant 1002 as a replacement for conventional mechanisms to set the alarm or perform other functions on the device. If the user's request is ambiguous or requires further clarification, the wizard 1002 can use the various techniques described here, including inducing or provoking an active reaction, paraphrasing, suggestions, and the like, to obtain the required information, so that the correct services 1340 are requested and the intended action is carried out. In one embodiment, wizard 1002 may ask the user for confirmation before calling a 1340 service to perform a function. In one embodiment, a user can selectively disable 1002 the ability of the wizard to call particular services 1340 or can disable all service calls if desired.
The system of the present invention can be implemented with many different types of clients 1304 and modes of operation. Now referring to Figure 6, a block diagram is illustrated showing a system architecture that
MEXICAN INSTITUTE W LA PXOPIFDA »
INDUSTRIAL
<img file="MX338784B_D0038.tif" />
illustrates several different types of clients ..... ”1 ^ 04 and <sup>1</sup> beggar <sup>1</sup> ele · operation. A person skilled in the art will recognize that the various types of clients 1304 and modes of operation illustrated in Figure 6 are exemplary only, and that the system of the present invention can be implemented using clients 1304 and / or different modes of operation. to those shown. Additionally, the system may include any or all of these clients 1304 and / or modes of operation, alone or in any combination. Illustrated examples include:
• Computer devices with I / O input and output (I / O) devices and / or sensors 1402.
• A client component can be deployed on any similar 1402 computing device.
• At least one modality can be implemented using a 1304A network viewer or other software application to allow communication with 1340 servers over the 1361 network. Input and output channels can be of any type, including for example visual channels and / or auditorium.
For example, in one embodiment, the system of the invention can be implemented using voice communication methods, allowing an assistant mode for the blind whose equivalent of a network viewer is speech driven and uses speech for output.
• Mobile devices with I / O (I / O) and 1406 sensors, for which the client can be implemented with an application in
<img file="MX338784B_D0039.tif" />
ΙΜΡΙ
HST1 MEXICAN TUTO
DB PROPERTY
INDUSTRIAL the 1304B mobile device. This includes but is not limited to mobile phones, smart phones, personal digital assistants, tablet devices, network game consoles, and the like.
• Consumer appliances with input / output, I / O (I / O) and 1410 sensors, for which the customer can be implemented as an embedded application in the 1304C appliance, • Automobiles and other vehicles with dashboard interfaces and 1414 sensors , for which the client can be implemented as an embedded system application 1304D. This includes but is not limited to car navigation systems, voice control systems, car entertainment systems, and the like.
• Network computing devices such as routers
1418 or any other device that resides in or interfaces with a network, for which the client can be implemented as a resident application of the 1304E device.
• 1424 email clients, for which a modality of the wizard is connected by means of a
E-mail Mode 1426. The Modality server of
Email 1426 acts as a communication bridge, for example taking user feed as email messages sent to the wizard and sending wizard output to the user in response.
• 1428 instant message clients, for which a
<img file="MX338784B_D0040.tif" />
The modality of the assistant is connected by the SerVttl'ui of the Modality___ ,.
Messaging 1430. The Messaging Mode 1430 server acts as a communications bridge, taking power from the user as messages that are sent to the wizard and sending output from the wizard to the user as messages in response.
• 1432 voice telephones, for which a wizard mode is connected by a Protocol Mode Server
Voice over Internet (VoIP = Voice over Internet Protocol) 1430. The VoIP Mode 1430 server acts as a communications bridge, taking power from the user as a spoken voice to the assistant and sending output from the assistant to the user, for example as synthesized speech, in answer.
For messaging platforms, including but not limited to email, instant messages, discussion boards, group chat sessions, customer support or live help sessions, and the like, attendee 1002 can act as a participant in conversations. Assistant 1002 can monitor the conversation and respond to individuals or the group using one or more of the techniques and methods described here for one-on-one interactions.
In various embodiments, the functionality to implement the techniques of the present invention can be distributed among any number of client and / or server components. For example, various software modules can be implemented to perform various functions in
<img file="MX338784B_D0041.tif" />
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0042.tif" />
connection with the present invention, and these modules can be implemented in various ways to run on client and / or server components. Now referring to Figure 7, an example of a client 1304 and a server 1340 are shown, communicating with each other to implement the present invention in accordance with one embodiment. Figure 7 illustrates a possible setup for which software modules can be distributed between client 1304 and server 1340. A person skilled in the art will recognize that the illustrated arrangement is simply exemplary, and that these modules can be distributed in many different ways. Furthermore, any number of clients 1304 and / or servers 1340 can be provided, and modules can be distributed among these clients 1304 and / or servers 1340 in any number of different ways.
In the example in Figure 7, the functionality to produce power and output processing functionality are distributed between client 1304 and server 1340, with the client producing power 1094a and the client output processing 1092a located in the client 1304, and the server part producing power 1094b and the output processing server part 1092b located on server 1340. The following components are located on the 1340 server:
complete vocabulary 1058b;
complete library of 1060b language pattern recognizers;
<img file="MX338784B_D0043.tif" />
IMPI ββΤΠΤΠΌ MEXICAN
Ot THE FSOffERTTY
INEFujnUAL • master version of personal memory to co'tfta plagp '-' íOS'Íb?
• long-term personal memory master version
1054b.
In one embodiment, client 1304 maintains subsets and / or portions of these components locally, to improve response and reduce dependency on network communications.
These subsets and / or portions can be maintained and updated according to well known cache or buffer management techniques. These subsets and / or portions include for example:
• vocabulary subset 1058a;
• 1060a language pattern recognizer library subset;
• 1052nd short-term personal memory cache;
• 1054a long-term personal memory cache.
Additional components can be implemented as part of the 1340 server, including for example:
• 1070 language interpreter;
• 1080 dialogue flow processor;
• 1090 output processor;
• 1072 domain entity databases;
•. 1086 task flow models;
• 1082 service orchestration;
• 1088 service capacity models.
Each of these components will be described more fully
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0044.tif" />
detail below. Server 1340 belongs<sup>1</sup> additional information when interfacing with external 1360 services, when required.
Conceptual Architecture
Referring now to Figure 1, a simplified block diagram of a specific exemplary embodiment of an intelligent automated assistant 1002 is illustrated. As · described in greater detail here, different modalities of intelligent automated assistant systems can be configured, designed and / or operated, to provide various different types of operations, functionalities and / or features that generally refer to automated assistant technology. intelligent. Furthermore, as described in greater detail here, many of these various operations, functionalities, and / or features of the intelligent automated assistant system (s) described herein may enable or provide different types of benefits and / or benefits to different entities interacting with the or intelligent automated assistant systems. The modality shown in Figure 1 can be implemented using any of the physical equipment architectures described above or using a different type of physical equipment architecture.
For example, according to different modalities, at least some or some intelligent automated assistant systems can be configured, designed and / or operated to
IMPI
ΙΝΤΠΤυΤΟ MEXICANO DI LA PROPIEDAD industrial
<img file="MX338784B_D0045.tif" />
provide various different types of operations, functionalities and / or characteristics such as for example one or more of the following (or their combinations):
• automate the application of data and services available on the Internet to discover, find, select between, purchase, reserve or order products and services. In addition to automating the process of using these data and services, the intelligent automated assistant 1002 can also allow the combined use of several data sources and services at the same time. For example, you can combine product information from multiple review sites, check pricing and availability from multiple resellers, and verify their locations and time constraints, and help a user find a customized solution to their problems.
• automate the use of data and services available in
Internet to discover, research, select between, book and otherwise learn about things to do (including but not limited to movies, events, performances, exhibitions, shows and attractions); places to eat or drink (such as restaurants and bars), times and places to meet others, and any other source of entertainment or social interaction that can be found on the Internet.
• enable the operation of applications and services through natural language dialogue that is otherwise provided by dedicated applications with user interfaces
<img file="MX338784B_D0046.tif" />
ΙΜΡΪ
MEXICAN INSTITUTE
1 «THE TROfl AGE
INDUSTRIAL graphics including search (comprising EÜSTJÜS & S<sup>1</sup> báüádd tíll location); navigation (maps and instructions); searching databases (such as finding businesses or people by name or other properties); obtain weather and forecast conditions, check the price of market items or the status of financial transactions; monitor traffic or flight status; access and update calendars and schedules; manage reminders, alerts, tasks and projects; communicate by email or other messaging platforms; and operate devices locally or remotely (for example dial phones, control lighting and temperature, control home security devices, play music or video, and the like). In one mode, wizard 1002 can be used to start, operate, and control many functions and applications available on the device.
• offers personal recommendations for activities, products, services, sources of entertainment, time management or management, or any other type of recommendation service that benefits from interactive dialogue in natural language and automated access to data and services.
According to different modalities, at least a portion of the various types of functions, operations, actions and / or other features that are provided by the intelligent automated assistant 1002 can be implemented in one or more client systems, in one or more systems. server and / or its
IMPI
MEXICAN INSTITUTE DS LA RROEIFPAD INDUSTRIAL
<img file="MX338784B_D0047.tif" />
combinations. ----------—
According to different modalities, at least a portion of the various types of functions, operations, actions and / or other features that are provided by the assistant 1002 can be implemented by at least one modality of an automated call and response procedure, such as it is illustrated and described for example with respect to Figure 33.
Additionally, various modalities of the assistant 1002 described herein may include or provide a number of different advantages and / or benefits over existing intelligent automated assistant technology such as for example one or more of the following (or combinations thereof):
• The integration of speech-to-text and natural language comprehension technology that is restricted by a set of explicit domain models, tasks, services, and dialogues. Unlike assistive technology that attempts to implement a general-purpose artificial intelligence system, the modalities described here can apply the multiple sources of constraints to reduce the number of solutions to a more treatable size. This results in fewer ambiguous language interpretations, fewer relevant domains or tasks, and fewer ways to operationalize service intent. The focus on specific domains, tasks and dialogues also makes it feasible to achieve coverage on domains and tasks with human-marked vocabulary and intent mapping to
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0048.tif" />
service parameters. . ...
• The ability to solve user problems by invoking services on their behalf over the Internet, using APIs. Unlike search engines that only return or provide links or links and content, some modalities of automated assistants 1002 described here can automate research and troubleshooting activities. The ability to invoke multiple services for a given request also provides the user with broader functionality than visiting a single site, for example to produce a product or service or find something to do.
• The application of personal information and personal interaction history in the interpretation and execution of user requests. Unlike conventional search engines or question answering services, the modalities described here use information from personal interaction history (eg, dialogue history, previous result selections, and the like), personal physical context (eg , location and time of the user), and personal information that is obtained in the context of interaction (that is, name, email addresses, physical addresses, phone number, account numbers, preferences, and the like). Using these sources of information allows, for example, a better interpretation of user feeding (for
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0049.tif" />
example, using personal history and physical context when interpreting language);
more personalized results (for example, that drift towards preferences or recent selections);
improved efficiency for the user (for example, by automating steps that involve registration in form-filling services).
• The use of dialogue history to interpret the user's natural feeding language. Because modalities can maintain personal history and apply natural language comprehension in user feeds, you can also use dialog context such as current location, time, domain, task stage, and task parameters to interpret new feeds. Conventional search engines and command processors interpret at least one independent query from a dialog history. The ability to use dialogue history can enable a more natural interaction, one that resembles normal human conversation.
• Produce active power, where assistant 1002 guides in an active way and restricts the user's power, based on the same models and information used to interpret their power. For example, Wizard 1002 can apply dialog templates to suggest next steps in a dialog with the user, where a request is refined; offer
IMPI
MEXICAN INSTITUTE Di LA MONEDAD INDUSTRIAL
<img file="MX338784B_D0050.tif" />
complete for partially written feeding based on context and domain specific possibilities; or use semantic interpretation to select between ambiguous interpretations of speech as text or text as intention.
• Explicit modeling and dynamic service management or management, with robust and dynamic service orchestration. The described modality architecture allows wizard 1002 to interface with many external services, dynamically determine which services can provide information for a specific user request, map parameters of the user request to different service APIs, call multiple services to the time, integrate results from multiple services, Gentle migration to failed services and / or efficiently maintain the implementation of services as their
APIs and capabilities.
• The use of active ontologies as a method and apparatus for building wizards 1002, which simplifies software engineering and data maintenance of automated wizard systems. Active ontologies are an integration of runtime environments and data modeling for attendees. They provide a framework for linking the various model and data sources together (domain concepts, task flows, vocabulary, language pattern recognizers, dialog context, user personal information, and domain and task request mapping to external services . The
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROWEOAD
<img file="MX338784B_D0051.tif" />
Active ontologies and other arCjü'ltéULUllitas innovations described here make it practical to build deep functionality within domains, unify multiple sources of information and services, and do this across a set of domains.
In at least one embodiment, the intelligent automated assistant 1002 may be operable to use and / or generate various different types of data and / or other types of information when performing specific tasks and / or operations. This may include for example, feeding data / information and / or data / information output. For example, in at least one embodiment, the intelligent automated assistant 1002 may be operable to access, process, and / or otherwise use information from one or more different types of sources, such as, for example, one or more memories, devices and / or local and / or remote systems. Additionally, in at least one embodiment, the intelligent automated assistant 1002 may be operable to generate one or more different types of output data / information, which for example may be stored in memory of one or more local devices and / or systems. and / or remote.
<td>Examples of</td><td colspan="2">different types</td><td>information / data</td><td>of</td>
<td>entrance to those who</td><td>they can have</td><td colspan="2">access and / or used by</td><td>the</td>
<td>automated assistant</td><td>intelligent</td><td> 1002</td><td>may include but</td><td>not</td>
<td>are limited to,</td><td>one or more</td><td>of</td><td>the following (or</td><td>their</td>
combinations):
• Voice feeding: from mobile devices such as
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0052.tif" />
mobile phones and tablets, computers on wireless phones, headsets, Bluetooth, car voice control systems, telephony systems, recordings in answering or answering services, audio voicemail in integrated messaging services, Voice-powered consumer applications such as clock radios, telephone stations, home theater or entertainment control systems, and game consoles.
• Text feed from keyboards on computers or mobile devices, numeric keypads on remote controls or other consumer electronic devices, email messages sent to the assistant, instant messages or similar short messages sent to the assistant, text received from players in multi-user gaming environments and text transmitted in real time in message feeds.
• Location information that comes from sensors or location-based systems. Examples include the Global Positioning System (GPS) and GPS
Assisted (A-GPS = Assisted GPS) on mobile phones. In one mode, location information is combined with explicit user feed. In one embodiment, the system of the present invention is capable of detecting when a user is home based on known address information and current location determination. In this way, certain inferences can be made regarding the type of information in which you may have
ΙΜΡΙ
MEXICAN INSTITUTE W THE INDUSTRIAL PRORIITY
<img file="MX338784B_D0053.tif" />
user interest when at home against 0 °<sup>4-A</sup> home, as well as the type of services and actions that must be invoked on behalf of the user depending on whether or not he is at home.
• Clock time information is on client devices. This may include, for example, the time of telephones or other client devices that indicate the local time and time zone. In addition, the time can be used in the context of user requests, such as for example, to interpret phrases such as one hour and tonight.
• Compass, accelerometer, gyroscope and / or travel speed data, as well as other sensor data from mobile or portable devices or embedded systems such as the car control system. This may also include location data from remote control devices to gadgets and game consoles.
• Clicking and selecting menus and other events from a Graphical User Interface (GUI) on any device that has a GUI. Additional examples include touching a touch screen.
• Events of sensors and other triggers displaced by data, such as alarm clocks, calendar alerts, price change triggers, location triggers, notification of device oppression from servers and the like.
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PBOPIITY
<img file="MX338784B_D0054.tif" />
The feed to the modalities agí »-i-amH-ián __ includes the context of the history of user interaction, including the history of requests and dialogues.
Examples of different types of data / output information that can be generated by intelligent automated wizard
<td>1002 may include, but not</td><td>is it so</td><td>limited</td><td>a, one or more</td><td>of</td><td>the</td>
<td colspan="2">following (or combinations thereof):</td><td></td><td></td><td></td><td></td>
<td>• Text output</td><td>than</td><td>It sends</td><td>directly</td><td>of</td><td>a</td>
<td>output device and / or</td><td>to</td><td>interface</td><td>of user</td><td>of</td><td>a</td>
<td>device</td><td></td><td></td><td></td><td></td><td></td>
<td>• Text and graphics</td><td colspan="2">sent to a</td><td>user by</td><td colspan="2">mail</td>
electronic • Text and graphics sent to a user over a messaging service • Speech output, may include one or more of the following (or combinations thereof):
Synthesized speech
Sample talk
Recorded messages • Distribution or graphic arrangement of information with photographs, rich text, videos, sounds, and hyperlinks. For example, content generated on a web viewer.
• Actuator output to control physical actions on a device, such as causing it to turn on or off, make a sound, change color, vibrate, control a lamp, or
ΙΜ ΡI
INSTITUTO MÍXICANO Di LA TKOFIÍOAD INDUSTRIAL similar. <i <
• Invoke other applications on a device, such as requesting a mapping application, voice dialing from a phone, sending an email or instant message, playing media, calendar feeds, task managers or administrators, and note and other apps.
• Actuator output to control physical actions to devices connected or controlled by a device, such as operating a remote camera, controlling a wheelchair, playing music on remote speakers, playing videos on remote displays, and the like.
It can be appreciated that the intelligent automated assistant 1002 of Figure 1 is but one example of a wide range of intelligent automated assistant system modalities that can be implemented. Other modalities of the intelligent automated assistant system (not shown) may include additional, fewer and / or different components / features than those illustrated, for example, in the exemplary intelligent automated assistant system mode of Figure 1.
User interaction
Now referring to Figure 2, an example of an interaction between a user and at least one embodiment of an intelligent automated assistant 1002 is illustrated. The sA
IMPI mbxicano institute DI THE INDUSTUAL PROHBDAD
<img file="MX338784B_D0055.tif" />
Example in Figure 2 considers that a user 'is speaking to the intelligent automated assistant 1002 using the power device 1206, which may be a speech power mechanism, and the output is a graphic layout or arrangement to the output device 1207, which it can be a scrolling screen. Conversation screen 101A features a conversation user interface that shows what user 101B said (I would like a romantic Italian food place near my office) and assistant response 1002, which is a summary of their findings 101C (Correct I found these Italian restaurants that the reviews say are romantic, close to your work :) and a 101D result set (the first three on a list of restaurants are displayed). In this example, the user clicks on the first result in a list, and the result automatically opens to reveal more information about the restaurant, illustrated on information screen 101E. Information display 101E and conversation display 101A may appear on the same output device, such as a touch screen or other display device; the examples shown in Figure 2 are two different output states for the same output device.
In one embodiment, information display 101E displays information that is obtained and combined from a variety of services, including for example any or all of the following:
IMPI «MEXICAN TITUTE OF THE INDUSTRIAL FROPISOAD
<img file="MX338784B_D0056.tif" />
• Directions and geolocation of negooioc, · ——. • Distance from a user's current location;
• Criticisms of a plurality of sources;
In one embodiment, information screen 101E also includes some examples of services that wizard 1002 can offer on behalf of the user, including:
• Dial a phone to call the business (call);
• Remember this restaurant for future reference (save);
• Send an email to someone with directions and information regarding this restaurant (share);
• Show the location of any directions to this restaurant on a map (cartography);
• Keep personal notes regarding this restaurant (my notes).
As shown in the Example in Figure 2, in one embodiment, wizard 1002 includes intelligence beyond simple database applications, such as, for example, • Processing a natural language intent statement 101B, not just keywords. ;
• Infer semantic intention of 'that feeding of language, such as interpreting Italian food sites as Italian restaurants;
• Conceptualize semantic intention in a strategy
<img file="MX338784B_D0057.tif" />
FROM INDUSTRIAL PROPERTY to use online services and execute that strategy on behalf of the user (for example, conceptualizing the desire for a romantic site in the strategy of checking online review sites for reviews that describes a site as romantic).
Smart Automated Assistant Components
In accordance with various modalities, the intelligent automated assistant 1002 can include a plurality of different types of components, devices, modules, processes, systems, and the like that, for example, can be implemented and / or present or instantiate by using hardware and / or combinations of hardware and software. For example, as illustrated in the exemplary embodiment of Figure 1, wizard 1002 may include one or more of the following types of systems, components, devices, processes, and the like (or combinations thereof):
• One or more active 1050 ontologies;
• One or more components producing active 1094 power (may include 1094a client part and server part
1094b);
• One or more 1052 short-term personal memory components (may include master version 1052b and 1052a cache or memory);
• One or more 1054 long-term personal memory components (may include 1052b master version and cache
1052a);
IMPI
MEXICAN INSTITUTE DR INDUSTRIAL PROPERTY
<img file="MX338784B_D0058.tif" />
• One 'or several components of rio ·. Diciminln models. 1Ω56: _ • One or more 1058 vocabulary components (may include complete vocabulary 1058b and subset 1058a);
• One or more components of one or more 1060 language pattern recognizers (may include the entire library
1060b and subset 1560a);
• One or more language interpreting components
1070;
• 1072 domain entity databases;
• One or more 1080 dialogue flow processor components;
• One or more service orchestration components
1082;
• One or more 1084 service components;
• One or more model components for task flow
1086;
• One or more model components for 1087 dialog flow;
• One or more components of 1088 service models;
• One or more 1090 output processor components.
As described in connection with Figure 7, in certain client / server based modalities, some or all of these components may be distributed between client 1304 and server 1340.
For illustration purposes, at least a portion of
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0059.tif" />
The different types of components of a specific Intelligent Automated Attendance Wini-IDNH 1002 will now be described in greater detail with reference to the exemplary Intelligent Automated Attendant Mode 1002 of Figure 1.
Active Ontologies 1050
Active 1050 ontologies serve as a unifying infrastructure that integrates models, components, and / or data from other modality parts of intelligent automated assistants 1002. In the field of computer and information science, ontologies provide structures for data from representation and knowledge such as classes / types, relationships, attributes / properties and their exemplification. Ontologies are used, for example to build data and knowledge models. In some embodiments of intelligent automated system 1002, ontologies are part of the modeling structure in which models such as domain models are built.
Within the context of the present invention, an active ontology 1050 can also serve as a runtime environment, where different processing elements are arranged in an ontology-like way (eg, they have different attributes and relationships with other processing elements). These processing elements perform at least some of the tasks of the Intelligent Automated Wizard 1002. Any number of active 1050 ontologies can
IMPI
ΙΝΪΤΠυΤΟ MKICANO M LA RROFIEdAL »INDUSTRIAL
<img file="MX338784B_D0060.tif" />
be provided.
In at least one embodiment, the active ontologies 1050 may be operable to perform and / or implement various types of functions, operations, actions, and / or other features such as, for example, one or more of the following (or combinations thereof):
• Act as an integrated modeling and development environment, models and data for various model and data components, including but not limited to
Domain Models 1056
Vocabulary 1058
Domain entity database 1072
1086 Task Flow Models
Dialogue Flow Models 1087
1088 Service Capability Models • Act as a data modeling environment where ontoiogy-based editing tools can operate to develop new models, data structures, database schemas, and representations.
• Act as a live runtime environment, exemplification values for 1056 domain elements, 1086 tasks, and / or 1087 dialog templates, 1058 language and / or vocabulary pattern recognizers, and user-specific information such as found in short-term personal memory 1052, long-term personal memory 1054 and / or orchestration results of
IMPI
MKICANU INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0061.tif" />
1182 services. For example, some nodes of an active ontology may correspond to domain concepts such as restaurant and its proprietary restaurant name. During live execution, these active ontology nodes can be installed with the identity of a particular restaurant entity and its name, and how its name corresponds to words in a natural language feed expression. Thus, in this modality, the active ontology serves both as a modeling environment that specifies the concept that restaurants are entities with identities that have names and to store dynamic links of those modeling nodes with data from entity databases and syntactic analysis of natural language.
• Enable communication and coordination between components and processing elements of a smart automated wizard, such as one or more of the following (or combinations thereof):
o One or more components to produce active power 1094
One or more components for language interpretation
1070
One or more 1080 dialog flow processor components
One or more components for service orchestration
1082
One or more service components 1084.
In one embodiment, at least a portion of the functions,
ΙΜΡΙ
WJTTTUTO MEXICANO DB LA WOFie »AD INDUSTRIAL
<img file="MX338784B_D0062.tif" />
operations, actions and / or other 'Uü' tm'tei'luyíaa · 1050 active features described herein can be implemented, at least in part using various methods and apparatus described in US Patent Application Serial Number 11 / 518,292 for Method and Apparatus for Building an Intelligent Automated
Assistant, introduced on September 8, 2006.
In at least one mode, a given instance of active 1050 ontology can access and / or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information can be accessed by communication with one or more local and / or remote memory devices. Examples of different types of data that can be accessed by active 1050 ontologies may include, but are not limited to one or more of the following (or combinations thereof):
• Available static data from one or more components of the intelligent automated assistant 1002;
• Data that is dynamically exemplified by user session, for example but not limited to, maintaining the state of user-specific feeds and outputs exchanged between components of the intelligent automated assistant 1002, the contents of short-term personal memory, inferences made from states of the user session and the like.
In this way, active 1050 ontologies are used to unify elements of various components in the wizard
IMPI
MKICANO INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0063.tif" />
intelligent automated 1002. An ncL'luj 1050 ontology allows an author, designer or system builder to integrate components in such a way that the elements of one component are identified with elements of other components. The author, designer or system builder can thus combine and integrate the components more easily.
Now referring to Figure 8, an example of a fragment of an active ontology 1050 is illustrated in accordance with one embodiment. This example is intended to help illustrate some of the various types of functions, operations, actions, and / or other features that can be provided by active 1050 ontologies.
The active ontology 1050 in Figure 8 includes representations of a restaurant and a food event. In this example, a restaurant is a 1610 concept with properties such as its name 1612, types of meals served 1615, and its location 1613, which in turn can be modeled as a structured node with properties for street address 1614. The concept of a food event can be modeled as node 1616 including a formal dinner event 1617 (with size 1619) and time period 1618.
• Active ontologies can include and / or refer to 1056 domain models. For example, Figure 8 illustrates a 1622 dining out domain model linked to the 1610 restaurant concept and the 1616 food event concept.
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL MONEDAD
<img file="MX338784B_D0064.tif" />
case, the active ontology 1050 includes the model<sup>1</sup> de ·· <knniniia - de.
dining out 1622; specifically, at least two nodes of active ontology 1050, that is, restaurant 1610 and food event 1616, are also included in and / or referred to by the domain model of dining out 1622. This domain model represents, among other things, the idea that dining out involves a food event that happens in restaurants. The active ontology nodes restaurant 1610 and food event 1616 are also included and / or referred to by other components of the intelligent automated assistant, as shown by the dotted lines in Figure
8.
• Active Ontologies can include and / or reference 1086 task flow models. For example, Figure 8 illustrates a 1630 event planning task flow model, which models domain independent event planning, applied to a domain-specific type of event: food event 1616. Here, Active Ontology 1050 includes 1630 General Event Planning Task Flow Model, comprising nodes representing events and other concepts involved in their planning. Active ontology 1050 also includes food event node 1616, which is a particular type of event. In this example, food event 1616 is included or referenced by both the 1622 domain model and the 1630 task flow model, and both of these models are included in and / or referenced by the active ontology 1050. again, the event of
<img file="MX338784B_D0065.tif" />
IMPI <sup>the</sup>,£!!?<sup>wid</sup>ao INDUSTiuai food 1616 is an example of how actives can unify elements of various components included and / or referred to by other components of the intelligent automated assistant, as shown by the dotted lines in Figure 8.
• Active Ontologies can include and / or reference 1087 Dialogue Flow Models. For example, Figure 8 illustrates a 1642 Dialogue Flow model to obtain the constraint values required for an exemplified or represented transaction size. restriction part as represented in concept 1619. Again, the active ontology 1050 provides a framework or structure for relating and unifying various components such as dialogue flow models 1087. In this case, dialogue flow model 1642 has a general concept of a constraint that is exemplified or represented in this particular example the active ontology node group size 1619. This particular dialog flow model 1642 operates on constraint abstraction, regardless of domain. The active ontology 1050 represents the group 1619 size property of the group node
1617, which is related to food event node 1616. In this mode, intelligent automated wizard 1002 uses active ontology 1050 to unify the concept of constraining in the 1642 dialog flow model with the group size property 1619 as part of a swarm of nodes representing the 1616 food event concept, which is part of the model of
IM? » Domain 1622 for dining out. ___ • Active ontologies may include and / or reference 1088 service models. For example, Figure 8 illustrates a model of a restaurant reservation service 1672 associated with the dialogue flow stage to obtain required values for that service make a transaction. In this case, the 1672 service model for a restaurant reservation service specifies that a reservation requires a value for group size 1619 (the number of people participating in a table to reserve). The group size concept 1619, which is part of the active ontology 1050, is also linked or related to a general dialogue flow model 1642 to ask the user about the restrictions for a transaction; in this case, group size is a required constraint for the 1642 dialog flow model.
• Active ontologies may include and / or refer to the domain entity database 1072. For example, Figure 8 illustrates a 1652 domain or restaurant entity database associated with restaurant node 1610 in the ontology. activates 1050. The active ontology 1050 represents the general concept of restaurant 1610, as it can be used by the various intelligent automated assistant components 1002, and is represented by data regarding specific restaurants in the restaurant 1652 database.
• Active ontologies can include and / or make
<img file="MX338784B_D0066.tif" />
IMPI
INSTITUTE M «ICA NO
FROM PROPERTY n »USTRJAL reference to vocabulary databases 1058. For example, Figure 8 illustrates a vocabulary database of 1662 types of cuisine, such as Italian, French, and the like, and the words associated with each cuisine such as French, continental, provincial, and the like. The active ontology
1050 includes restaurant node 1610, which is related to the cooking types service node 1615, associated with the representation of cooking types in the 1662 cooking types database. A specific entry in the 1662 database for a Type of cuisine, such as French, is thus related through the active 1050 ontology as an example of the concept of types of cuisine served 1615.
• Active ontologies can include and / or refer to any databases that can be mapped to concepts or other representations in 1050 ontology. Domain entity databases 1072 and vocabulary databases 1058 are only two Examples of how the Active Ontology 1050 can integrate databases with each other and with other components of the 1002 Automated Wizard. Active ontologies allow the author, designer, or system builder to specify nontrivial mapping between representations in the database and representations in the 1050 ontology. For example, database schemas for restaurant databases 1652 can represent a restaurant as a table of strings and numbers, or as a projection from a database of
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0067.tif" />
largest business, or any other represent'St'lóll dtltít'uadu paxu> database 1652. In this example active ontology 1050, restaurant 1610 is a concept node with properties and relationships, organized differently from those database tables. In this example, the 1050 ontology nodes are associated with elements of the database schemas. The 1050 ontology and database integration provides a unified representation for interpreting and acting on specific data entries in databases, in terms of larger data and model sets in the active 1050 ontology. For example, the French word It can be an entry in the 1662 kitchen types database. Because in this example, the 1662 database is integrated into the active 1050 ontology, the same French word also has an interpretation as a possible type of cuisine served in a restaurant, involved in planning meal events, and This type of cuisine serves as a restriction to use when using restaurant reservation services, and so on. Active ontologies can thus integrate databases into the modeling and execution environment to operate interactively with other components of the automated wizard 1002.
As described above, the active ontology 1050 allows the author, designer, or system builder to integrate components; thus, in the example in Figure 8, the elements of a component such as constraint in the model
IMPI MEXICAN INSTITUTE OF INDUSTRIAL INDUSTRY
<img file="MX338784B_D0068.tif" />
Flow Dialog 1642 can be identified from other components such as required parameters of the 1672 restaurant reservation service.
Active 1050 ontologies can be incorporated, for example, as model, database, and component configurations in which relationships between models, databases, and components are any of:
• containment and / or inclusion;
• relationship with links and / or indicators;
• API interface, both internal to a program and between programs.
For example, now referring to Figure 9, an example of an alternate embodiment of the intelligent automated assistant system 1002 is shown, where the domain model components 1056, vocabulary 1058, language pattern recognizers 1060, personal short memory term 1052 and long-term personal memory 1054, they are organized under a common container associated with the active ontology 1050 and other components such as one or more components to produce active power 1094, language interpreter 1070 and dialogue flow processor 1080 are associated with active ontology 1050 through API relationships.
One or more components to produce active power 1094
In at least one embodiment, one or more components to produce active power 1094 (which, as described
ΙΜΡΙ
MEXICAN INSTITUTE QE LA INDUSTRIAL MOPIÍDAD
<img file="MX338784B_D0069.tif" />
above, they can be implemented in '._nfigurami stand-alone or in a configuration that includes both server and client components) can be operable to perform and / or implement various types of functions, operations, actions and / or other features such as for example one or more of the following (or combinations thereof):
• Produce, facilitate and / or process the user's diet or the user's environment, and / or information regarding their needs or requests. For example, if the user searches for a restaurant, the module that produces food can obtain information regarding the restrictions or preferences of the user by location, time, type of cuisine, price and so on.
• Facilitate different types of power from various sources, such as, for example, one or more of the following (or their combinations):
keyboard power supply or any other text-generated power supply keyboard power supply at user interfaces that offer dynamically suggested termination of partial power supply voice or voice power supply systems Graphical user interface power (GUIs) where the users click, select, or otherwise directly manipulate graphic objects to indicate selections
IMPI
MSGCAN INSTITUTE OF PROPERTY MDUSTR1A1
<img file="MX338784B_D0070.tif" />
feed from other CfU £ gSñéTS'ñ 'TdíXiU applications and send it to the automated wizard, including email, text messaging or other text communication platforms
By performing active feed production, the assistant 1002 is able to clarify the intention at an earlier stage of feed processing. For example, in an embodiment where speech feed is provided, the waveform can be sent to a server 1340 where the words are extracted, and the semantic interpretation is performed. The results of this semantic interpretation can then be used to shift active feed production, which can offer the user alternate candidate words to select based on their degree of semantic adjustment as well as phonetic correspondence.
In at least one mode, the component or components for producing active power 1094, actively, automatically and dynamically guide the user to power that can be powered by one or more of the services offered by the modalities of assistant 1002. Now with reference To Figure 10, a flow chart is shown illustrating an operating method for the active feed producing component (s) 1094 according to one embodiment.
The procedure starts at 20. At step 21, assistant 1002 can offer interfaces on one or more channels of
IMPI
MKICANO INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0071.tif" />
feeding. For example, a user interface provides users with options to speak, write, touch, or tap at any stage of a conversation interaction. In step 22, the user chooses a feed channel when starting feed in a mode, such as pressing a button to start recording speech or taking out a write interface.
In at least one mode, wizard 1002 offers predefined hints for select mode 23. That is, it offers options 24 that are relevant in the current context before the user provides any feed in that mode. For example, in a text feed mode, wizard 1002 may offer a list of common words that start prompts or text commands such as, for example, one or more of the following (or combinations thereof): imperative verbs (for example, find, buy, book, get, call, check, schedule, and the like), pronouns (for example, restaurants, movies, events, business, and the like), or menu-type option, to name speech domains (eg weather, sports, news and the like).
If the user selects one of the preset options at 25, and sets a preference for auto-display 30, the procedure can return immediately. This is similar to the operation of a conventional menu selection.
However, the initial option can be taken as
<img file="MX338784B_D0072.tif" />
IMPI
INSTITUTO MMUCANC O »LA SROFISDAD INDUSTRIAL partial feeding ·, or the user can --h. '^ Brr <sub>to</sub> supplying a partial feed 26. At any point in the feed, in at least one mode, the user can select to indicate that the partial feed · is complete 22, causing the procedure to return.
At 28, the last feed, either select or supplied, is added to the cumulative feed.
In 29, the system suggests the following possible feeds that are relevant given the current feed and other sources of restrictions on what constitutes relevant and / or significant feed.
In at least one embodiment, the user feed restriction sources (eg, used in steps 23 and 29) are one or more of several models and data sources that can be included in wizard 1002, which may include, but are not limited to, one or more of the following (or combinations thereof):
• Vocabulary 1058. For example, words or phrases that correspond to the current diet may be suggested. In at least one embodiment, the vocabulary can be associated with any or one or more active ontology nodes, domain models, task models, dialog models, and / or service models.
• 1056 domain models, which can restrict feeds that can instantiate or otherwise be
<img file="MX338784B_D0073.tif" />
IMPI
MIXICAN INSTITUTE
OF THE PROPERTY
INDUSTRIAL consistent with the domain model. For example, in at least one modality, 1056 domain models can be used to suggest concepts, relationships, properties, and / or distances that will be consistent with current diet.
• 1060 language pattern recognizers, which can be used to recognize languages, phrases, grammatical constructions, or other patterns in the current feed and used to suggest pattern filling.
• Domain entity database 1072, which can be used to suggest possible entities in the domain that correspond to the feed (for example, business names, movie names, event names, and the like).
• Short-term memory 1052, which can be used to couple to any previous feeding or feeding portion
<td>previous,</td><td>me</td><td>any other</td><td>property</td><td>or made regarding</td><td>the</td>
<td>history</td><td>of</td><td colspan="3">interaction with a user. For example,</td><td>the</td>
<td colspan="2">feeding</td><td>partial can</td><td>mate</td><td>against cities that</td><td>the</td>
<td>user</td><td colspan="2">has found in a</td><td>session already</td><td colspan="2">be hypothetical</td>
<td colspan="2">(for example,</td><td>mentioned in</td><td>queries)</td><td>and / or in physical form</td><td>(by</td>
<td>example,</td><td>how</td><td>is determined by</td><td colspan="2">location sensors).</td><td></td>
<td> •</td><td>In</td><td colspan="2">at least one modality,</td><td>semantic paraphrases</td><td>of</td>
Recent feeds, request, or results can be matched against the current feed. For example, if the user has previously requested live music and obtained on concert lists, and then described music in an environment that produces
<img file="MX338784B_D0074.tif" />
IMPI
MEXICAN INSTITUTE
SAY THE PROPERTY
INDUSTRIAL active feeding, suggestions may include live MUSlta and / or concerts.
• Long-term personal memory 1054, which can be used to suggest coupling items from long-term memory. These link items may include, for example, one or more or any combination of: domain entities that are kept (for example, restaurant, movies, theaters, meeting places or favorite spaces and the like), things to do, cite items, Calendar entries, names of people in contacts / address books, names of streets or cities mentioned in contacts / address books and the like.
• 1086 Task Flow Models, which can be used to suggest feeds based on the next possible steps in a task flow.
• 1087 Dialogue Flow Models, which can be used to suggest feeds based on the next possible stages of a dialogue flow.
• 1088 Service Capability Models, which can be used to suggest possible services to use by name, category, capacity, or any other property in the model. For example, a user can type part of the name of a preferred review site, and wizard 1002 can suggest a complete command to query that review site, to review.
In at least one embodiment, the component or components for producing active power 1094 present the user with a
MIXICAN INSTITUTE
Say LA FROFIBDAO industrial
<img file="MX338784B_D0075.tif" />
conversation interface, for example a user interface and the wizard communicate by two-way expressions in a form of conversation. The component or components to produce active power 1094 may be operable to realize and / or implement. various types of conversation interfaces.
<td></td><td>In at least one modality, the or</td><td>the</td><td>components</td><td>for</td>
<td>produce</td><td>active power 1094 can</td><td>to be</td><td>operable</td><td>for</td>
<td>perform</td><td>and / or implement various types</td><td>of</td><td>interfaces</td><td>of</td>
conversation in which the wizard 1002 uses layers of the conversation to request information from the user according to dialogue patterns. Dialogue models can represent a procedure to execute a dialogue, such as for example a series of steps required to produce the information required to perform a service.
In at least one embodiment, the component (s) to produce active power 1094 offers restrictions and guides the user in real time, while the user is halfway through writing, speaking, or otherwise creating power. For example, active production can guide the user to write text feeds that are recognizable by an assistant mode 1002 and / or that can be serviced by one or more services offered by assistant modalities 1002. This is an advantage over passive waiting for unrestricted feeding from a user because it allows the user's efforts to be
<img file="MX338784B_D0076.tif" />
IMPI
MUiCANO INSTITUTE
SAY THE PROPERTY
INDUSTRIAL focuses on feeds that may or may be useful, and / or allow modalities of wizard 1002 to apply their feed interpretations in real time as they are supplied by the user.
At least a portion of the functions, operations, actions, and / or other features of active power production described herein can be implemented at least in part using various methods and apparatus described in US Patent Application Serial Number 11 / 518,292 for Method and Apparatus for Building an Intelligent Automated Assistant, filed September 8, 2006.
According to specific modalities, multiple instances or threads of the component (s) to produce active power 1094 can be implemented concurrently and / or initiated by the use of one or more processors 63 and / or other combinations of hardware and / or equipment physical and software.
According to different modalities, one or more different threads or instances of the component or components for active feed production 1094 may be initiated in response to detection of one or more conditions or events that satisfy one or more different types of minimum threshold criteria for enable startup of at least one instance of the component (s) to produce 1094 active power. Various examples of conditions or events that can trigger the start and / or implementation of one or more threads or different instances of the or
ΙΜΡΙ
MWICAN INSTITUTE OF INDUSTRIAL WORK
<img file="MX338784B_D0077.tif" />
The components of Produce Active Power 1094 may include, but are not limited to, one or more of the following (or combinations thereof):
• User login. For example, when the user session starts or starts an application that is a modality of the wizard 1002, the interface may offer the opportunity for the user to start feeding, for example, by pressing a button to start a speech feeding system or click on a text field to start a text feed session.
• User power detected.
• When wizard 1002 explicitly asks the user for power, such as when requesting an answer to a question or offering a menu of next stages from which to choose.
• When the wizard 1002 helps the user perform a transaction and collects data for this transaction, that is, filling in a form.
In at least one mode, a given instance of the 1094 active power production component (s) can access and / or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information can be accessed by communication with one or more local and / or remote memory devices. Examples of different types of data that can be accessed by the component or components to produce power
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0078.tif" />
Active 1094 may include, but are not limited to, one or more of the following (or combinations thereof):
• database of possible words to use in a textual feed;
• grammar of possible sentences to use in mention as text feed;
• database of possible interpretations of speech feeding;
• database of previous feeds of a user or other users;
• data from any of the various models and data sources that may be part of wizard 1002 modalities, which may include, but are not limited to, one or more of the following (or combinations thereof):
• 1056 domain models;
• Vocabulary 1058;
• 1060 language pattern recognizers;
• Database for domain entity 1072;
• Short-term memory 1052;
• Long-term personal memory 1054;
• 1086 Task Flow Models;
• 1087 Dialogue Flow Models;
• 1088 service capacity models.
According to different modalities, the component or components to produce active power 1094 can apply
IMPI nSTITUTO MWICANO DS LA MONEDAD INDUSTRIAL
<img file="MX338784B_D0079.tif" />
procedures for producing active feeding, for example one or more of the following (or combinations thereof):
• written feeding;
• speech feeding;
• graphical user interface feeding (GUIs), including gestures;
• feeding of suggestions offered in a dialogue; and • events from computing and / or detected environments.
Produce Active Written Food
Now referring to Figure 11, a flow chart is shown illustrating a method of producing active written feed in accordance with one embodiment.
The method starts at 110. The wizard 1002 receives partial text feed on it, for example via the feed device 1206. The partial text feed can include, for example, characters that have been written so far in a feed field of text. At any time, a user can indicate that the written feed is complete 112, such as by pressing the Enter key. If not complete, a hint generator 114 generates candidate hints 116. These hints may be syntactic, semantic, and / or other types of hints based on the information sources or restrictions described here. If the suggestion is chosen 118, the feed becomes
117 to include the selected suggestion.
IMPI
INSTITUTO MM1CANO DI LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0080.tif" />
In at least one modality, the suggestions <sup>1</sup> They can include extensions to the current feed. For example, a suggestion for rest may be restaurant.
In at least one modality, suggestions may include replacing parts of the current supply. For example, a suggestion for rest may be places to eat.
In at least one modality, suggestions may include replacing and rewriting parts of the current feed. For example, if the current food is finding restaurants a suggestion can be Italian and when the suggestion is chosen, all the food can be rewritten again to find Italian restaurants.
In at least one embodiment, the resulting feed returned is noted at 119, so the information for which selections were made at 118 is retained along with the text feed. This allows for example the concepts or semantic entities underlying a string associated with the string when it is returned, which improves the precision of subsequent language interpretation.
Now referring to Figures 12 through 21, screen shots are shown illustrating portions of some of the procedures for producing active written feed according to one embodiment. The screenshots illustrate an example of an assistant mode 1002 as implemented in a smartphone such as iPhone available from Apple Inc.,
IMPI
WSTITUTO MIXICANO Dt THE INDUSTRIAL PROPERTY
<img file="MX338784B_D0081.tif" />
from Cupertino, California. Power is provided to this device via a touch screen, including on-screen keyboard functionality. A person skilled in the art will recognize that the screen shots illustrate an exemplary embodiment only, and that the techniques of the present invention can be implemented on other devices and using other layouts and arrangements.
In Figure 12, screen 1201 includes a set of top-level hints 1202 that are illustrated when no power has been provided in field 1203. This corresponds to step no power 23 in Figure 10 applied to step 114 of Figure 11, where there is no power.
In Figure 13, screen 1301 illustrates an example of using vocabulary to offer suggested user partial feed endings 1303 1305 that are supplied in the
<td>field 1203 using</td><td>the</td><td>keyboard</td><td>in</td><td>screen</td><td>1304. These</td>
<td>suggested endings</td><td> 1303</td><td>they can</td><td>to be</td><td>part of</td><td>the function of</td>
<td colspan="2">produce active power</td><td> 1094 .</td><td>The</td><td>user</td><td>has supplied</td>
partial user power 1305 including com string. Vocabulary component 1058 has provided a mapping of this string in three different types of instances, which are cited as suggested endings 1303: the phrase community and local events is a category of the event domain; chambers of commerce is a category of the local business search domain, and Jewish Community Center is the name of an instance
<img file="MX338784B_D0082.tif" />
ΙΜΡΙ
MEXICAN INSTITUTE Say LA FFOFISDAO INDUSTRIAL of local businesses. The vocaBTl'lállU component IODO pi ^ cd ^ provide data search and namespace management or handling like these. The user can touch the Go button 1306 to indicate that the power supply has finished; this causes wizard 1002 to proceed with the entire text string as a user feed unit.
In Figure 14, screen 1401 illustrates an example where suggested semantic endings 1303 for a partial string wh (for the English version what or where, in Spanish would be equivalent to / where 1305 includes complete sentences with described parameters. These types Suggestions can be enabled by using one or more of several power restriction models and sources described here. For example, in a modality shown in Figure 14, what happens in the city is an active production of the location parameter of the Local Events domain; where business name is an active production of the Business Name restriction of. Local Business Search domain; shown in the meeting name is an active production of the Meeting Place constraint - from the Local Events domain; and that it is being presented in the cinema is an active production of the restriction Name of Cinema of the domain of Local Events. These examples illustrate that the suggested endings are generated by models rather than simply taken from a database of previously supplied queries.
MEXICAN INSTITUTE
Dt THE PROPERTY
INDUSTRIAL
<img file="MX338784B_D0083.tif" />
In Figure 15, screen f 501 llli and ITH a continuation of the same example, after the user has provided additional text 1305 in field 1203. Suggested endings 1303 are updated to correspond to additional text 1305. In this example, data from the domain entity database 1072 were used: meeting places whose name begins with f. Note that this is a significantly smaller and semantically more relevant set of suggestions than all words beginning with f. Again, suggestions are generated by applying a model, in this case the domain model that represents Local Events that happen in Meeting Places, which are Business with Names. Suggestions actively produce feeds that can make potentially significant inputs when using a
Local Events Service.
In Figure 16, screen 1601 illustrates a continuation of the same example, after the user has selected one or more suggested terminations 1303. Active production continues by prompting the user to further specify the type of information desired, here by presenting a number of specifiers 1602 from which the user can select. In this example, these specifiers are generated by domain, task flow, and dialog flow models. The Domain is Local Events, which includes Event Categories that occur on the Dates on Sites and have Event and Performer Names of
IMPI «τιτυτο MEXICAN OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0084.tif" />
Characteristics. In this modality, the fact that '(] Liy ”eaLdS — t'incu · options are offered to the user, is generated from the Dialogue Flow model that indicates that users will be asked for Restrictions that they have not yet provided and the Model Service that indicates that these five Restrictions are parameters to Local Events services available to the attendee. Even the selection of preferred phrases to use as specifiers, such as categorized and characterized, are generated from the Domain Vocabulary databases.
In Figure 17, screen 1701 illustrates a continuation of the same example, after the user has selected one of specifiers 1602.
In Figure 18, screen 1801 illustrates a continuation of the same example, where select specifier 1602 has been added to field 1203, and additional specifiers 1602 are presented. The user can select one of specifiers 1602 and / or provide power from Additional text by keyboard 1304.
In Figure 19, screen 1901 illustrates a continuation of the same example, where select specifier 1602 has been added to field 1203, and still more specifiers 1602 are not presented. In this example, previously supplied constraints do not occur in any way. activates redundantly.
In Figure 20, screen 2001 illustrates a continuation of the same example, where the user has touched the
ΙΜΡΙ
MEXICAN INSTITUTE DS INDUSTRIAL PROPERTY
<img file="MX338784B_D0085.tif" />
Go button 1306. User feed-SS 'Hiré'S'Lfd' in box 2002, and a message is illustrated in box 2003, providing feedback to the user regarding the query made in response to user feed.
In Figure 21, screen 2101 illustrates a continuation of the same example, where results have been found. The message is displayed in box 2102. Results 2103, including power items allow the user to view more details, save the identified event, purchase tickets, add notes or the like.
On a screen 2101, and other and other displayed screens, they are scrollable, allow the user to scroll forward to view screen 2001 or other previously displayed screens, and make changes to the query if desired.
Active Speech Feed Production
Now referring to Figure 22, a flowchart illustrating a method of producing active power for voice or speech power is illustrated, in accordance with one embodiment.
The method starts at 221. Wizard 1002 receives on
121 feeding voice or speaking in the form of an auditory signal. A speech-to-text service 122 or processor generates a set of candidate text interpretations 124 of the auditory signal
In one embodiment, the speech-to-text service 122 is
<img file="MX338784B_D0086.tif" />
MEXICAN INSTITUTE
FROM THE PROPERTY vVfcm-íJií INDUSTRIAL “implements using, for example, the Unanro Rcmgm available from Nuance Communications, Inc. of Burlington, MA.
In one embodiment, wizard 1002 employs statistical language models to generate interpretations of candidate text 124 from speech feed 121.
In addition, in one modality, the statistical language models are adjusted to search for words, names, and phrases that occur in the various wizard models 1002 illustrated in the
Figure 8. For example, in at least one modality, statistical language models are given words, names, and phrases of some or all of: 1056 domain models (for example, words and phrases referring to restaurant and food events), 1086 task flow models (for example, words and phrases referring to event planning), 1087 dialog flow models (for example , words and phrases related to the restrictions required to collect supplies for a restaurant reservation), 1072 domain entity databases (for example, restaurant names), 1058 vocabulary database (for example, cuisine types), 1088 service models (for example, service names provided such as OpenTable), and / or any words, names, or phrases associated with any active ontology node
1050.
In one modality, statistical language models are also adjusted to search for words, names, and phrases in the
IMPI
HST1 MKICANO INDUSTRIAL PROPERTY TUTE
<img file="MX338784B_D0087.tif" />
Long-term personal memory 1054. For example, statistical language models can give text of things to do, list items, personal notes, calendar entries, names of people in contacts / address books, email addresses, names street or city mentioned in contacts / address books, and the like.
A classification or ordering component, analyzes candidate interpretations 124 and classifies or sorts them into 126 according to which they also fit syntactic and / or semantic models of the intelligent automated assistant 1002. Any source of restrictions on the user's power can be used. For example, in one embodiment, wizard 1002 can classify the output of the speech-to-text interpreter according to which interpretations are also parsed in a semantic and / or syntactic sense, a domain model, tasks and / or model of dialogue, and / or the like: evaluates how various word combinations in text interpretations 124 will also fit the concepts, relationships, entities, and properties of active 1050 ontology and its associated models. For example, if the speech-atext service 122 generates the two interpretations candidate Italian food for lunch and Italian shoes for lunch, the semantic relevance ranking 126 can classify Italian food for lunch if it better corresponds to the active ontology 1050 of the new attendees 1002 (for example,
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0088.tif" />
the Italian words (o), food and 'aliuue-rre<sup>11</sup> all
they correspond to nodes in 1050 ontology and they are all connected by relationships in 1050 ontology, while the word shoes does not correspond to the 1050 ontology or correspond to a node that is not part of the domain network for dining out).
In various modalities, algorithms or procedures employed by wizard 1002 for interpretation of text feeds, including any natural language processing procedure modality shown in Figure 28, may be used to classify and qualify interpretations 124 generated by the speech service. a-text 122.
In one embodiment, if the ranking component 126 determines 128 that the highest-ranking speech performance of performances 124 is in range above a specified threshold, the highest-ranking performance can be automatically selected at 130. If no performance has a range above a specified threshold, possible candidate interpretations of speech 134 are presented at 132 to the user.
The user can then select 136 from the displayed selections.
In various embodiments, user selection 136 among the displayed selections can be accomplished by any power mode, including for example any of the multimodal power modes described in connection with Figure 16. These power modes include, without limitation, power
<img file="MX338784B_D0089.tif" />
IMPI
MEXICAN INSTITUTE
OF THE PROPERTY
INDUSTRIAL written actively produced 2610, alini'gTftaLiúii<sup>1</sup> -ele hobla -—— actively produced 2620, GUI actively presented for 2640 power, and / or the like. In one embodiment, the user can select from candidate 134 performances, for example by touching or speaking. In the case of speaking, the possible interpretation of the new speech feed is highly restricted by the small selection set offered 134. For example, if it is offered Do you mean Italian food or Italian shoes? the user can only say food and the wizard can couple this with the phrase italian food and not confuse it with other global interpretations of the input or food.
If the feed is automatically chosen 130 or
<td>Choose</td><td> 136</td><td>for him</td><td>user,</td><td>I know</td><td>came back</td><td>to food</td>
<td>resulting</td><td> 138.</td><td>In al</td><td>minus one</td><td colspan="2">modality,</td><td>Power supply</td>
<td>back it</td><td>write down</td><td>at 138,</td><td>by way of</td><td>such</td><td>that</td><td>information regarding</td>
what selections were made in step 136 with text feed. This allows, for example, the concepts or semantic entities underlying a string to be associated with the string when it is returned, which improves the precision of subsequent language interpretation. For example, if Italian food is offered as one of Candidate Interpretations 134 based on a semantic interpretation of Type of Cuisine = Italian Food, then the machine-readable semantic interpretation can be sent along with the selection of iiviri
INSTITUTO MEXICANO DI LA FROFIgDAD INDUSTRIAL
<img file="MX338784B_D0090.tif" />
Italian food string user like alimenLdtlún<sup>1</sup> 'de feente »—-———.
annotated 138.
In at least one embodiment, candidate text interpretations 124 are generated based on speech interpretations that are received as speech-to-text service output 122.
In at least one embodiment, candidate text interpretations 124 are generated by paraphrasing speech interpretations in terms of their semantic meaning. In some modalities, there may be multiple paraphrases of the same speech interpretation, offering different meaning to the word or alternatives of the same name. For example, if the speech-to-text service 122 indicates a meeting place, the candidate interpretations presented to the user may be paraphrased as a meeting place (local businesses) and food place (restaurants).
<td></td><td>In</td><td>to the</td><td>less</td><td>one modality,</td><td>interpretations of</td><td>text</td>
<td>candidate</td><td> 124</td><td colspan="2">include</td><td colspan="2">offers to correct sub-strings.</td><td></td>
<td></td><td>In</td><td>to the</td><td>less</td><td>one modality,</td><td>interpretations of</td><td>text</td>
<td>candidate</td><td colspan="2"> 124</td><td colspan="2">include offers for</td><td colspan="2">correct sub-strings of</td>
Candidate interpretations using synthetic and semantic analysis as described here.
In at least one modality, when the user chooses a candidate interpretation, it is returned.
In at least one mode, the user is offered an interface to edit the performance before returning it.
In at least one modality, the user is offered a
IMPI
INSTITUTO MSXICANO Di LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0091.tif" />
interface to continue with more voice feeding before Stí returns the feeding. This allows to increase a construction of a mention of feeding, obtaining corrections, suggestions and syntactic and semantic guides, in one iteration.
In at least one embodiment, the user is offered an interface to proceed directly from 136 to step 111 of an active written feed production method (described above in connection with Figure 11). This allows you to insert written and spoken food, obtaining syntactic and semantic corrections, as well as suggestions and guidance in one stage.
In at least one embodiment, the user is offered an interface to proceed directly from step 111 from a mode of producing active written power to a mode of producing active speech power. This makes it possible to insert written and spoken food, obtaining correction of suggestions and syntactic and semantic guides in one stage.
GüI-Activa Based Food Production
Now referring to Figure 23, a flowchart illustrating a method for active power production for GUI-based power according to one embodiment is illustrated.
The method starts at 140. Wizard 1002 presents at 141 the graphical user interface (GUI) on the device.
IΜ ΡI
INSTITUTOMEXICANr »Ai
M EXICANO INSTITUTE DS LA FROFIÍDAU INDUSTRIAL exit 1207, which may include, for example, and hotnnps.
User interacts at 142 with at least one GUI element. Data 144 is received, and converted to 146 in a uniform format. The converted data is then returned.
In at least one modality, some of the GUI elements are dynamically generated from the models of the active ontology, instead of being written in a computer program. For example, wizard 1002 may offer a set of constraints to guide a restaurant reservation service as regions to tap on a screen, with each region representing the name of the constraint and / or a value. For example, the screen may have rows of a dynamically generated GUI layout with regions for Kitchen, Location, and Price Range restrictions. If the models of the active ontology change, the GUI screen will automatically change without reprogramming.
Active Dialogue Suggestion Feed Production
Figure 24 is a flowchart illustrating a method of producing active feed at the level of a dialogue stream according to one embodiment. Wizard 1002 suggests 151 possible answers 152. The user selects 154 a suggested answer. The received power is converted to 154 to a uniform format. The converted data is then returned.
In at least one modality, the suggestions offered in
MtXiCANO INSTITUTE
OF THE EHOWBDAD
INDUSTRIAL
<img file="MX338784B_D0092.tif" />
stage 151 are offered as follow stagesL'd tífl ”Uñ 'fluju<sup>1</sup> dU tasks and / or dialogues.
In at least one modality, the suggestions offer options to refine a query, for example using parameters from a domain and / or task model. For example, you may offer to change the assumed location or time of a request.
In at least one modality, suggestions offer options for selecting between ambiguous alternate interpretations given by a language interpretation procedure or component.
In at least one modality, the suggestions offer options for selecting between ambiguous alternate interpretations given by the language interpretation procedure or component.
In at least one mode, the hints offer options to select from the following steps in a workflow associated with the 1087 dialog flow model. For example, the 1087 dialog flow model may suggest that after you collect the restricted domain ( for example, dinner in a restaurant), assistant 1002 will suggest other related domains (for example, a nearby movie).
Active Supervision For Relevant Events
In at least one modality, asynchronous events can be treated as feeds in a manner analogous to each other
<img file="MX338784B_D0093.tif" />
IMPI
INSTITUTO IMÍXICANO Dt LA ÍÍCftÉDAD INDUSTWAL feeding methods produced in -Ρητ-τηρ Activa. In this way, these events can be provided as feeds to wizard 1002. Once interpreted, these events can be treated in a similar way to any other feed.
For example, a flight status change can initiate an alert notification to send to a user. If a flight is indicated as delayed, wizard 1002 can continue the dialogue by presenting alternate flights, making other suggestions and the like, based on the detected event.
These events can be of any type. For example, wizard 1002 may detect that the user has just arrived home, or is lost (outside of a specified route), or that a stock price has reached a threshold value, or that a television program that the user is interested in, is about to start, or that a musician of your interest is touring the area. In any of these situations, wizard 1002 can proceed with a dialog in substantially the same way as if the user had initiated the query themselves. In one modality, events can even be based on data that is
<td>provide</td><td>of</td><td>other devices,</td><td colspan="2">for example to say</td><td>to the</td>
<td colspan="2">user when</td><td>a collaborator has</td><td>returned</td><td>of lunch</td><td>(the</td>
<td>device</td><td>of the</td><td>collaborator can</td><td>point</td><td>said event</td><td>to the</td>
<td>device</td><td>of the</td><td>user, in whose</td><td>weather</td><td>the assistent</td><td> 1002</td>
installed on the user's device responds accordingly).
In one mode, events can be notifications
IMPI
MEXICAN INSTITUTE D * THE PROPERTY
INDUSTRIAL
<img file="MX338784B_D0094.tif" />
or alerts from a calendar, clock, rRcordatnrio ^ o anl i pir? ------ do. For example, an alert from a calendar application regarding a dinner date may initiate a dialog with attendee 1002 regarding the dinner event. The dialogue can proceed as if the user had just spoken or written the information regarding the future dinner event, such as dinner for in San Francisco.
In one embodiment, the possible event trigger context 162 can include information regarding people, places, times, and other data. This data can be used as part of the feed to assistant 1002 for use in various stages of processing.
In one embodiment, this event trigger context data 162 can be used to reduce ambiguity of user speech or text feeds. For example, if a calendar event alert includes the name of a person invited to the event, that information can help reduce feed ambiguity that may correspond to multiple people with the same or similar names.
Referring now to Figure 25, a flowchart illustrating a method for active monitoring for relevant events according to one embodiment is illustrated. In this example, events that trigger events are sets of feeds 162. Wizard 1002 monitors 161 for these events. Detected events can be filtered and classified into
IMPI mucicano institute Dt THE INDUSTRIAL PROPERTY
<img file="MX338784B_D0095.tif" />
164 for semantic relevance using models, data and information available from other components in the intelligent automated assistant 1002. For example, an event reporting a change in flight status can be given higher relevance if the short-term or long-term memory records for a user they indicate that the user is on that flight and / or has made queries about it to the assistant 1002. This classification and filtering can then present only the top events for review by the user, who can then choose one or more and act on them.
Event data is converted to 166 into a uniform feed format and returned.
In at least one embodiment, wizard 1002 can proactively offer services, associated with events that are suggested for user attention. For example, if a flight status alert indicates that a flight may be missed, assistant 1002 may suggest a task flow to the user to re-plan the itinerary or book a hotel.
Example of one or more Components that Produce Food:
The following example is intended to help illustrate some of the various types of functions, operations, actions, and / or other features that may be provided by it or the components to produce 1094 active power.
Example: Command Termination (what can the user say to assistant 1002?)
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL MONEDAD
<img file="MX338784B_D0096.tif" />
The user is in front of a box & Γ1Ι11ϋΙ1Ι2 <ΑΐΜβΙ<sup>ι</sup>Γ dtí text with the general instruction to feed what would you like to do ?. Depending on the context and user power, any of several system responses are provided. Examples are shown below.
Case: Zero Power
IMPI
INSTITUTO MBXICANÜ Cfc LA PROPI1DAD INDUSTRIAL
<img file="MX338784B_D0097.tif" />
<td>Feed User</td><td>Response from System</td><td>Commentary</td>
<td>With an empty field and without context of dialogue write a question or a character of space or not feed nothing for certain period of weather one</td><td>• Find restaurants, Hotels, films,... • Reserve a table, a room, a seat,... • Order food, flowers,... • Buy a ticket, a book, a DVD,... • Program, one meeting a event,... • Remember one thing to do,... • Explain results.</td><td>Black text is will insert into the field of feeding. Text with italics would be the suggestions following a time i know insert the text.</td>
Case: First Word Feed
<img file="MX338784B_D0098.tif" />
IMPI ΐΝϊΓίτυτο Mexican Dt THE INDUSTRIAL PROPERTY
<td colspan="2">Feeding</td><td>of</td><td>Response from</td><td>Commentary</td>
<td>User</td><td></td><td></td><td>System</td><td></td>
<td>With the</td><td>context</td><td>of</td><td>• Find</td><td>Bold text</td>
<td>dialogue</td><td>empty yes</td><td>the</td><td>restaurants,</td><td>will be inserted into the</td>
<td>user</td><td>writes</td><td>the</td><td>Hotels,</td><td>field of</td>
<td>first</td><td>lyrics</td><td>of</td><td>films,...</td><td>feeding.</td>
<td colspan="3">any command,</td><td> •</td><td>Italicized text</td>
<td colspan="2">It is suggested.</td><td>By</td><td></td><td>it would be</td>
<td>example.</td><td></td><td></td><td></td><td>suggestions</td>
<td>The</td><td></td><td></td><td></td><td>following once</td>
<td></td><td></td><td></td><td></td><td>which is inserted the</td>
<td></td><td></td><td></td><td></td><td>text.</td>
Case: Keyword Feed
<td>Feeding User</td><td>of</td><td>Answer System</td><td>of the</td><td colspan="2">Commentary</td>
<td>With context</td><td>of</td><td>• Remember</td><td>a</td><td>Black text</td><td>will be</td>
<td>empty dialogue</td><td>Yes</td><td>thing</td><td>by</td><td>inserted into</td><td>the</td>
<td colspan="2">user type</td><td>do,...</td><td></td><td>countryside</td><td>of</td>
<td>the first</td><td>two</td><td colspan="2">• Find</td><td>feeding.</td><td>Text</td>
<td>letters</td><td>of</td><td>restaurants</td><td>by</td><td>with italics</td><td>would</td>
<td>any</td><td></td><td>Location,</td><td>type</td><td colspan="2">the suggestions</td>
<td></td><td></td><td>kitchen, ..</td><td> •</td><td></td><td></td>
<img file="MX338784B_D0099.tif" />
IMPI
MEXICAN INSTITUTE Say THE INDUSTRIAL PROPERTY
<td>word is suggested</td><td></td><td>tracking a</td>
<td>Like a</td><td></td><td>time it was inserted</td>
<td>command. By</td><td></td><td>the text.</td>
<td>example</td><td></td><td></td>
<td>Re |</td><td></td><td></td>
Case: Suggest arguments
<td>Feeding User</td><td>of</td><td>Answer System</td><td>of the</td><td colspan="2">Commentary</td>
<td>The user</td><td>he has</td><td>• Restaurants</td><td>by</td><td>Not just offer</td><td>the</td>
<td>fed</td><td> 0</td><td>location type</td><td>of</td><td colspan="2">selection class</td>
<td>selected</td><td>a</td><td>kitchen,...</td><td></td><td>but warn</td><td>the</td>
<td>verb</td><td>of</td><td>• Hotels</td><td>by</td><td>options</td><td>of</td>
<td>command</td><td>and</td><td>Location,</td><td></td><td colspan="2">restriction. Notice</td>
<td>nothing else.</td><td>By</td><td>availability,.</td><td></td><td>that the name</td><td>of the</td>
<td>example,</td><td></td><td>• Films</td><td>by</td><td>site is just</td><td>a</td>
<td>Find |</td><td></td><td>Location,</td><td></td><td>suggestion, and</td><td>not</td>
<td></td><td></td><td>gender,...</td><td></td><td>will insert text.</td><td></td>
<td></td><td></td><td colspan="2">• Name of the site</td><td></td><td></td>
Case: Suggestion criteria
<img file="MX338784B_D0100.tif" />
IMPI
MEXICAN INSTITUTE DAY 1A INDUSTRIAL PROPERTY
<td rowspan="2">Feeding</td><td rowspan="2">of</td><td rowspan="2">Answer</td><td rowspan="2">of the</td><td colspan="2"></td>
<td colspan="2">Commentary</td>
<td>User</td><td></td><td>System</td><td></td><td></td><td></td>
<td>The user</td><td>already</td><td colspan="2">• In the city,</td><td colspan="2">Bold text</td>
<td colspan="2">has supplied</td><td colspan="2">Postal Code</td><td>will be inserted into</td><td>the</td>
<td colspan="2">enough for</td><td>• Near</td><td>House,</td><td>countryside</td><td>of</td>
<td colspan="2">establish a</td><td>office,</td><td>Name</td><td colspan="2">feeding. Text</td>
<td>homework</td><td>and</td><td>of the site</td><td></td><td colspan="2">with italics they will be</td>
<td>domain,</td><td>and</td><td>• Name</td><td>of the</td><td>the suggestions</td><td>of</td>
<td>now I know</td><td>you</td><td>restaurant</td><td></td><td>track a</td><td>time</td>
<td>suggest</td><td></td><td>mentioned</td><td></td><td>which was inserted</td><td>the</td>
<td>restrictions</td><td></td><td> •</td><td></td><td>text.</td><td></td>
<td>in order.</td><td></td><td></td><td></td><td></td><td></td>
<td>Find</td><td></td><td></td><td></td><td></td><td></td>
<td>restaurants</td><td>i</td><td></td><td></td><td></td><td></td>
Case: Add criteria
<td>Feeding User</td><td>of</td><td>Answer System</td><td>of the</td><td colspan="2">Commentary</td>
<td>The user</td><td>he has</td><td>" City,</td><td>code</td><td>Until</td><td>than</td>
<td>said in</td><td>and</td><td>Postcard</td><td></td><td>supply</td><td>the</td>
<td>now I know</td><td>you</td><td>• stick</td><td>High</td><td>following</td><td></td>
<td>suggest</td><td></td><td>California</td><td></td><td>word, the</td><td></td>
<img file="MX338784B_D0101.tif" />
ΙΜΡΙ
INSTITUTO MIXICANO OS LA PROPIEDAD INDUSTRIAL
<td colspan="2">locations.</td><td colspan="3">• Menlo Park,</td><td>suggestions</td><td>are</td>
<td>Find</td><td></td><td></td><td></td><td></td><td>a notice,</td><td>a</td>
<td>restaurants</td><td>in</td><td></td><td></td><td></td><td>recommendation</td><td>by</td>
<td>_l</td><td></td><td></td><td></td><td></td><td>what kind of</td><td>the</td>
<td></td><td></td><td></td><td></td><td></td><td>no way</td><td>to</td>
<td></td><td></td><td></td><td></td><td></td><td>to write.</td><td>The</td>
<td></td><td></td><td></td><td></td><td></td><td>sites</td><td>are</td>
<td></td><td></td><td></td><td></td><td></td><td>cities of</td><td>the</td>
<td></td><td></td><td></td><td></td><td></td><td colspan="2">personal memory.</td>
<td>The user</td><td>he has</td><td>• me</td><td>House</td><td></td><td>The places</td><td>are</td>
<td>said close</td><td>and</td><td>• me</td><td>job</td><td></td><td colspan="2">one or more sites</td>
<td>now I know</td><td>you</td><td> •</td><td>Stick</td><td>High,</td><td colspan="2">of the memory</td>
<td>suggest</td><td></td><td colspan="2">California</td><td></td><td>personal</td><td></td>
<td>locations.</td><td></td><td> •</td><td>Menlo</td><td>Park,</td><td></td><td></td>
<td>Find</td><td></td><td colspan="2">California</td><td></td><td></td><td></td>
<td>restaurants</td><td></td><td></td><td></td><td></td><td></td><td></td>
<td>close |</td><td></td><td></td><td></td><td></td><td></td><td></td>
Case: Add location or other restrictions
<td>Feed User</td><td>Response from System</td><td>Commentary</td>
<td>Find</td><td>• who serve the</td><td>Others are suggested</td>
<td>restaurants in</td><td>type of food</td><td></td>
<img file="MX338784B_D0102.tif" />
IMPI
MEXICAN INSTITUTE M INDUSTRIAL PROPERTY
<td rowspan="2">Palo Alto |</td><td></td><td></td>
<td>O Li ^ UO of UUUllld • with availability today at night, morning,... • described as a good service, Romantic,</td><td>name of the site</td>
<td>Find restaurants in Palo Alto with availability |</td><td>• or at night • morning • at 7pm • At 9pm • another time or date</td><td>7:00 and 9:00 are our suggestions with base on time current</td>
Case: Harnessing from restriction, task or unknown domain
<td>Feeding of user</td><td>Answer System</td><td>of the</td><td colspan="2">Commentary</td>
<td>Romantic |</td><td>• restaurants</td><td>by</td><td>Value</td><td>of</td>
<td></td><td>location type</td><td>of</td><td>restriction of</td><td>the</td>
<td></td><td>kitchen,...</td><td></td><td>ontology that</td><td>He says</td>
<td></td><td>• Hotels</td><td>by</td><td>which classes</td><td>of</td>
<td></td><td>Location,</td><td></td><td>selection use</td><td></td>
<img file="MX338784B_D0103.tif" />
ΙΜΡΙ
MEXICAN INSTITUTE Ul INDUSTRIAL PROPERTY
<td rowspan="3"></td><td colspan="2" rowspan="2">availability,...</td><td></td>
<td rowspan="2"></td>
<td>• films Location, gender,...</td><td>by</td>
<td>Comedy |</td><td>• films</td><td>by</td><td>Comedy is a value</td>
<td></td><td>Location,</td><td></td><td>of restriction for</td>
<td></td><td>gender,...</td><td></td><td>genres in movies,</td>
<td></td><td>• events</td><td>by</td><td>a genre in events,</td>
<td></td><td>Location,...</td><td></td><td>and</td>
<td></td><td>• clubs</td><td>by</td><td>comedy clubs is</td>
<td></td><td>Location</td><td></td><td>a category of</td>
<td></td><td></td><td></td><td>business of</td>
<td></td><td></td><td></td><td>local directory</td>
Example: Complete Name
Here, the user has typed in some text without accepting any of the commands, or just extending a command with an identity name. The system may try to complete the names, depending on the context. It also reduces ambiguity in the domain.
Case: words without context
100
<img file="MX338784B_D0104.tif" />
IMPI Mexican institute
DC THE INDUSTRIAL PROPERTY
<td>Feeding of user</td><td>Response from System</td><td></td>
<td>il for</td><td>• II Fornaio (restaurant) • III Forgotten Gains (film)</td><td>May require name search of identity. Termination of multiple words. Show mastery like a suggestion.</td>
<td>tom cruise</td><td>• films</td><td>May require search on behalf of identity</td>
<td>romantic</td><td>• films • restaurants</td><td>Based only on ontology</td>
Case: name with context
<td>Feeding of user</td><td>Response from System</td><td>Commentary</td>
<td>Find</td><td>• II Fornaio</td><td>May require search</td>
<td>restaurants</td><td>(restaurant)</td><td>of identity name.</td>
<td>il for</td><td> •</td><td>Use selection class</td>
101
<img file="MX338784B_D0105.tif" />
IMPI
MEXICAN INSTITUTE Say INDUSTRIAL ROOTTY
<td></td><td></td><td>and location context current queries and passes to restrict the termination. Shows domain as a suggestion.</td>
<td>tom cruise</td><td>• films</td><td>May require search in the name of identity</td>
<td>Find</td><td>• Romantic</td><td>Based only on</td>
<td>rom movies</td><td>• Roman Holiday (movie)</td><td>ontology</td>
<td>With classification g</td><td>• films</td><td></td>
Example: Select values from a set
Here, the user responds to a system request to supply a value for a specific parameter, such as location, time, type of cuisine, or genre. The user can either select from a list or provide a value. As you type, matching items from the list are illustrated as options. Examples are shown below.
Case: value class selection
IMPI
MEXICAN INSTITUTE D £ LA FROPIIDAO INDUSTRIAL
<img file="MX338784B_D0106.tif" />
<td>Feeding of user</td><td>Response from System</td><td colspan="2">Commentary</td>
<td>Restaurants</td><td>• burgers</td><td>Users</td><td>they can</td>
<td>they serve</td><td>• frozen</td><td>to write</td><td>any</td>
<td></td><td>• dogs</td><td>thing more and</td><td>they do not have</td>
<td></td><td>hot</td><td colspan="2">what to select from</td>
<td></td><td>dogs)</td><td>menu.</td><td></td>
<td>tom cruise</td><td>• films</td><td>Can</td><td>to require</td>
<td></td><td></td><td>search in</td><td>name of</td>
<td></td><td></td><td>identity</td><td></td>
<td>Users who</td><td>• burgers</td><td>With base</td><td>only in</td>
<td>serve h</td><td>• Hot dogs</td><td>ontology</td><td></td>
<td></td><td>(dogs</td><td></td><td></td>
<td></td><td>hot)</td><td></td><td></td>
<td></td><td>• Hot sauce</td><td></td><td></td>
<td></td><td>(hot sauce)</td><td></td><td></td>
<td>Movies that</td><td>• today</td><td></td><td></td>
<td>are presented</td><td>• today in the</td><td></td><td></td>
<td></td><td>night</td><td></td><td></td>
<td></td><td>• Friday in</td><td></td><td></td>
<td></td><td>the night</td><td></td><td></td>
Example: Reusing previous commands
103
IMPI
MEXICAN INSTITUTE M LA MONEDAD INDUSTRIAL
<img file="MX338784B_D0107.tif" />
Previous queries are also nnrínnoj par = an automatic termination interface. They can only be coupled as strings (when the power field is empty and there are no known restrictions) or they can be suggested as relevant when it comes to certain situations.
Case: Termination in previous consultations
<td>Feeding of user</td><td colspan="2">System response</td><td>Commentary</td>
<td>Ital</td><td> •</td><td>restaurants</td><td>Using</td>
<td></td><td>Italians</td><td>(termination</td><td>correspondence</td>
<td></td><td>normal)</td><td></td><td>string for</td>
<td></td><td colspan="2">• movies with</td><td>recover</td>
<td></td><td>actors</td><td>Italians</td><td>queries</td>
<td></td><td>(recent</td><td>query)</td><td>previous</td>
<td>lunch</td><td>• Sites</td><td>for lunch</td><td></td>
<td></td><td>in Marin</td><td></td><td></td>
<td></td><td>(recent</td><td>query)</td><td></td>
<td></td><td colspan="2">• buy the book</td><td></td>
<td></td><td>lunch</td><td>naked</td><td></td>
<td></td><td colspan="2">(Naked lunch)</td><td></td>
Example: Retrieve items from Personal Memory
Assistant 1002 can remember certain events and / or entities in personal memory associated with the user. Automatic termination can be performed based on these items
104
ΙΜΡΙ
MEXICAN INSTITUTE
OS THE FROfIBDAC industrial remembered. Examples appear below. ——. _
Case: Termination in events and entities in personal memory
<img file="MX338784B_D0108.tif" />
<td>Feed User</td><td>Answer System</td><td>of the</td><td>Commentary</td>
<td>Mary</td><td>• Lunch</td><td>with Mary</td><td></td>
<td></td><td>Saturday</td><td>(of my</td><td></td>
<td></td><td>events)</td><td></td><td></td>
<td></td><td> •</td><td>Films</td><td></td>
<td></td><td>calls</td><td>I am</td><td></td>
<td></td><td colspan="2">crazy about mary</td><td></td>
<td></td><td>(Something</td><td>about</td><td></td>
<td></td><td>Mary)</td><td></td><td></td>
<td>lunch</td><td>• lunch</td><td>with Mary</td><td></td>
<td></td><td>Saturday</td><td>(of my</td><td></td>
<td></td><td>events)</td><td></td><td></td>
<td></td><td>• to buy</td><td>the book</td><td></td>
<td></td><td>lunch</td><td>naked</td><td></td>
<td></td><td colspan="2">(of my all ??)</td><td></td>
<td>Hob</td><td colspan="2">• Hobee restaurant</td><td></td>
<td></td><td>in Palo</td><td>High (from</td><td></td>
<td></td><td colspan="2">my restaurants)</td><td></td>
105
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
Produce multimodal active feeding __________
In at least one embodiment, the components for producing active power 1094 can process power from a plurality of power modes. At least one modality can be implemented with a procedure to produce active feeding that takes advantage of the particular types of feeding and methods to select from suggested options. As described herein, there may be procedural modalities for producing active feed for text feed, speech feed, GUI based feed, feed in the context of a dialog, and / or feed resulting from event triggers.
In at least one mode, for a single instance of the intelligent automated wizard 1002, there may be support for one or more (or any combination of) written feed, speech feed, GUI feed, dialog feed, and / or event feed.
Now referring to Figure 26, a flowchart illustrating a method of producing multimodal active power according to one embodiment is illustrated. The method starts at 100. The feeds can be received concurrently from one or more or any combination of the feed modes, in any sequence. Thus, the method includes actively producing the 2610 written feed, 2620 speech feed,
<img file="MX338784B_D0109.tif" />
106
<img file="MX338784B_D0110.tif" />
IMPI • «τιτυτυ méxicano DI LA RUOPlíDAD INDUSTRIAL
GUI 2640, power in the context of a 2650 v / o dialog power resulting from 2660 event triggers. Any or all of these power supplies are unified in the 2690 unified power format and returned. The 2690 Unified Power Format allows the other components of the 1002 Intelligent Automated Assistant to be designed and operate independently of the particular mode of power.
Offering active guidance for multiple modalities and levels, allows restriction and feeding guidance beyond those available to isolated modalities. For example, the types of suggestions offered to select between speech, text, and dialogue stages are independent in such a way that their combination is a significant improvement over adding techniques for active production to individual modalities or levels.
Combine multiple sources of constraints as described here (syntactic / linguistic, vocabulary, entity database, domain models, task models, service models, and the like) and multiple sites where these restrictions can be actively applied (speech , text, GUI, dialog, and asynchronous events) provides a new level of functionality for human-machine interaction.
Components of 1056 domain models
Components of 1056 domain models include representations of the concepts entities relationships,
107
IMPI
MIXICAN INSTITUTE DB INDUSTRIAL PROPERTY
<img file="MX338784B_D0111.tif" />
properties and instances of a domain. By ejenrgrlej<sup>1</sup>, 'The mpdebo'de · domain of dining out 1622 may include the concept of a restaurant as a business with a name and address and telephone number, the concept of a food event with a group size and date and time associated with the restaurant.
In at least one embodiment, the domain model components 1056 of the wizard 1002 may be operable to perform and / or implement various types of functions, operations, actions, and / or other features such as, for example, one or more of the following ( or their combinations):
• 1056 domain model components can be used by automated wizard 1002 for various processes, including: producing power 100, interpreting natural language 200, dispatching to services 400, and generating output 600.
• 1056 domain model components can provide word lists that can correspond to a domain concept or entity, such as restaurant names, that can be used for active feed production 100 and natural language processing 200.
• 1056 domain model components can classify candidate words in processes, for example to determine that a word is the name of a restaurant.
• 1056 domain model components can show the relationship between partial information to interpret natural language, for example what type of cuisine can be associated with
108
IMPI
INSTITUTO MSCICANO OS LA INDUSTRIAL PR PISDAD
<img file="MX338784B_D0112.tif" />
business entities (eg food mexlddlld lUCdl — can be interpreted as finding stylish restaurants = Mexican, and this inference is possible due to information in domain model 1056).
• 1056 domain model components can organize information regarding services used in orchestrating 1082 services, for example that a particular network service may provide restaurant reviews or reviews.
• 1056 domain model components can provide the information to generate natural language paraphrases and other output formatting, for example by providing canonical forms to describe concepts, relationships, properties, and instances.
According to specific modalities, multiple instances or threads of the 1056 domain model components can be concurrently implemented and / or initiated by the use of one or more processors 63 and / or other combinations of hardware and / or hardware and media logical. For example, in at least some embodiments, various aspects, characteristics, and / or functionalities of 1056 domain model components may be performed, implemented, and / or initiated by one or more of the following types of systems, components, systems, devices, procedures, processes and the like (or their combinations):
• Components of 1056 domain models can be implemented as data structures that represent concepts,
109
IMPI
MEXICAN INSTITUTE D »LA ΡΑβηΙΟΑί) INDUSTRIAL
<img file="MX338784B_D0113.tif" />
relationships, properties and instances. These structure, Ηρ. ά ^ της can be stored in memory, files or databases.
• Access to 1056 domain model components can be implemented through direct APIs, network APIs, query interfaces to databases and / or the like.
• Creation and maintenance of 1056 domain model components can be accomplished for example by direct file editing, database transactions and / or through the use of domain model editing tools.
• 1056 domain model components can be implemented as part of or in association with active 1050 ontologies, which combine models with representations or instances of the models for servers and users.
In accordance with various modalities, one or more different threads or instances of 1056 domain model components may be initiated in response to detection of one or more conditions or events that satisfy one or more different types of minimum threshold criteria to trigger start of at least one instance of 1056 domain model components. For example, trigger initiation and / or deployment of one or more different threads or instances of 1056 domain model components can be triggered when domain model information is required, including during feed production, feed interpretation, task identification and mastery, natural language processing, service orchestration and / or
110
INSTITUTO MFJtCANO Dt THE PROPERTY
INDUSTRIAL
<img file="MX338784B_D0114.tif" />
format output for users. .....
In at least one mode, a given instance of 1056 domain model components can access and / or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information can be accessed by communication with one or more local and / or remote memory devices. For example, data from the 1056 domain model components may be associated with other model modeling components including vocabulary 1058, language pattern recognizers 1060, dialog flow models 1087, task flow models 1086, capability models of service 1088, database of domain entity 1072 and the like. For example, businesses in the 1072 domain entity databases that are classified as restaurants may be known by type identifiers, which are kept in the domain model component of dining out.
Example of domain model components:
Now referring to Figure 27, a set of screen shots is shown illustrating an example of various types of functions, operations, actions, and / or other features that can be provided by 1056 domain model components in accordance with one embodiment.
In at least one embodiment, the 1056 domain model components are the unification data representation that
111
IMPI
INSTITUTO MlRlCRNÜ DP LA PRO «po IndOstom.
<img file="MX338784B_D0115.tif" />
allow the presentation of information displayed in .nantaiiag
103A and 103B for restaurants, which combines data from several different data sources and services, including, for example: name, address, business categories, telephone number, identifier to store in long-term personal memory, identifier for share over email, multi-source reviews, map coordinates, personal notes, and the like.
1070 Language Interpreter Components
In at least one embodiment, the wizard's 1070 language interpreter components 1002 may be operable to perform and / or implement various types of functions, operations, actions, and / or other features such as for example one or more of the following (or their combinations):
• Analyze user feed and identify a set of parsing results.
User feed may include any user information and its device context that may contribute to understanding the user's intent, which may include for example one or more of the following (or combinations thereof): word sequences, the identity of gestures or GUI elements involved in producing the feed, current context of the dialog, current device application and its current data objects and / or any other personal dynamic data that is obtained
112
ΙΜΡΙ
INSTITUTO MKICANU OSLA INDUSTRIAL PROPERTY
<img file="MX338784B_D0116.tif" />
regarding the user such as location, time and the like. For example, in one embodiment, user power is in the form of a uniform annotated power format 2690 resulting from active power output 1094.
o Syntactic analysis of results are associations of data in the user's feed with concepts, relationships, properties, instances, and / or other nodes, and / or data structures in models, databases and / or other representations of intention and / or or user context. Associations of parsing results can be complex cartographies of sets and sequences of words, signals and other elements of user feed to one or more concepts, relationships, properties, instances or other nodes and / or associated data structures described here.
• User feed analysis and identify a set of parsing results, which are parsing results that associate data in the user feed with structures representing syntactic parts of speech, clauses, and phrases, including multi-word names, structures of sentences and / or other grammatical graph structures. The syntactic analysis results are described in element 212 of the natural language processing procedure described in connection with Figure 28.
• Analyze user feed and identify a set of semantic analysis results, which are results of
113
ΙΜΡΙ
INSTITUTO MÍXICANO DB LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0117.tif" />
parsing associating data in l'd dl'i'm & lTL cfrtníyiT<sup>-</sup>rtc<sup>1</sup> user with structures that represent concepts, relationships, properties, entities, quantities, propositions and / or other representations of meaning and intention of the user. In one embodiment, these representations of meaning and intention are illustrated by sets of and / or elements of and / or instances of models or databases and / or nodes in ontologies, as described in element 220 of the natural language described in connection with Figure 28.
• Clarify or reduce ambiguity between alternate syntactic or semantic analysis results as described in item 230 of the natural language processing procedure described in connection with Figure 28.
• Determine if a partially written feed has syntactic and / or semantic meaning in an automatic termination procedure such as that described in connection with the
Figure 11.
• Helps generate suggested terminations 114 in an automatic termination procedure such as that described in connection with Figure 11.
• Determine whether spoken feeding interpretations are significant in a syntactic and / or semantic sense in a speech feeding procedure such as that described in connection with Figure 22.
According to specific modalities, multiple
114
<img file="MX338784B_D0118.tif" />
IMPI tfSTlTUTO MWICANO oe THE PROPERTY
INDUSTIML instances or threads of language interpreter components
1070 they can be implemented and / or started concurrently through the use of one or more processors 63 and / or other combinations of hardware and / or hardware and software.
According to different modalities, one or more different threads or instances of language interpreter components
1070 they can be initiated in response to detection of one or more conditions or events that satisfy one or more different types of minimum threshold criteria to trigger the start of at least one instance of 1070 language interpreter components. Various examples of conditions or events that can trigger start and / or implementation of one or more different threads or instances of 1070 language interpreter components may include, but are not limited to, one or more of the following (or combinations thereof):
• While feeding occurs, including but not limited to
Suggest possible endings to written feed 114 (Figure 11);
Classify speech interpretations 126 (Figure 22);
When ambiguities are offered as suggested responses in dialogue 152 (Figure 24);
• When the result of producing power is available, including when power is produced by
115
<img file="MX338784B_D0119.tif" />
IMPI
MEXICAN INSTITUTE
Dt LA RNOMEÜAD
INDUSTRIAL any active multimodal power production mode
100.
In at least one embodiment, a given instance of the 1070 language interpreter components can access and / or use information from one or more associated databases. In at least one embodiment, at least a portion of this database information can be accessed by communication with one or more local and / or remote memory devices. Examples of different types of data that can be accessed by Language Interpreter components may include but are not limited to one or more of the following (or combinations thereof):
• 1056 domain models;
• Vocabulary 1058;
• 1072 domain entity database;
• Short-term memory 1052;
• Long-term personal memory 1054;
• 1086 task flow models;
• Dialogue flow models 1087;
• 1088 service capacity models.
Now also referring to Figure 29, a screen shot illustrating natural language processing according to one embodiment is shown. The user has provided (by voice or text) 2902 language feed consisting of the phrase who will be at Filmore this weekend. This phrase<sup>116</sup> Wicked
Mexican institute
M INDUSTRIAL PROPERTY is echoed to the user on screen 2901. The compusersUS language interpreters 1070 feed the component process 2902 and generate a parsing result. The parsing result associates that feed with a request to demonstrate local events that are scheduled for any of the following weekend days at any event location whose name matches
Filmore. A paraphrase of the parsing results is shown as 2903 on screen 2901.
Now with reference and also to Figure 28, a flowchart illustrating an example of a method for natural language processing according to one embodiment is illustrated.
The method begins 200. The language feed 202 is received, such as the string who will be featured this weekend at the Filmore in the example in Figure 29. In one embodiment, the feed is augmented by current context information, such as the location and local time of the current user. In word / phrase coupling 210, the language interpreter components 1070 find associations between feeds and user concepts. In this example, associations are found between the string being presented and the concept of event site listings; the string this weekend (along with the user's current local time) and a representation of an approximate time period that represents
<img file="MX338784B_D0120.tif" />
<sup>117</sup> IMPI
INSTITUTE Μ K ICA NO
OF THE PROPERTY
INDUSTRIAL the following weekend; and Filmore's string with the name of a meeting place. Word / Phrase Coupling 210 can use data, for example, from language pattern recognizers 1060, vocabulary database 1058, active ontology 1050, short-term personal memory 1052, and long-term personal memory 1054.
The language interpreter components 1070 generate candidate parsing 212 that includes the select parsing result but can also include other parsing results. For example, other parsing results may include those where it will be associated with other domains such as games or with an event category such as sporting events.
Short and / or long-term memories 1052, 1054 can also be used by language interpreter components 1070 to generate candidate parsing 212. Thus, feed that is previously provided in the same session, and / or known user information , can be used to improve performance, reduce ambiguity, and reinforce the conversational nature of the interaction. Data from active 1050 ontology, 1056 domain models, and 1086 task flow models can also be used to implement evidence rationing in determining valid candidate parsing 212.
In semantic correspondence 220 the components
118
IMPI
MEXICAN INSTITUTE Say THE HOUSING PROPERTY
<img file="MX338784B_D0121.tif" />
1070 language interpreters consider 'LUlllllJ'l'flatiuiiea of possible parsing results according to what they also fit with semantic models such as domain and database models. In this case, the parsing includes the associations (1) will be presented (an expression in the user feed) as a Local Event at the Meeting Place ”(part of a 1056 domain model represented by a swarm of nodes in active ontology 1050) and (2) Filmore (another word in the feed) as a match to an entity name in the 1072 domain entity database for Local Event Gathering Locations, which is represented by an active ontology node and domain model element (Meeting Place Name).
Semantic mapping 220 can use data from eg active ontology 1050, short-term personal memory 1052, and long-term personal memory 1054. For example, semantic mapping 220 can use data from pre-meeting or local event references in dialogue (from short-term personal memory 1052) or personal favorite meeting places (from long-term personal memory 1054).
A set of candidate or potential semantic parsing results is generated at 222.
At the stage of reducing ambiguity 230, the interpreting components of language 1070 weigh the evidential force of
119
<img file="MX338784B_D0122.tif" />
IMPI
INSTITUTO MEXICANO DI LA FBOPIEDAD INDUSTRIAL the results of semantic parsing ^ .anH-iHai-n ???. In this example, the combination of parsing running as a Local Event at the Meeting Site and matching
Filmore as a Meeting Site Name is a stronger match to a domain model, than alternate combinations where for example it will be presented is associated with a sports domain model, but there is no association in the sports domain for Filmore.
Ambiguity reduction 230 can use data from, for example, active ontology structure 1050. In at least one embodiment, connections between nodes in an active ontology provide evidence support to decrease or reduce ambiguity between candidate semantic parsing results. 222. For example, in one modality, if three active ontology nodes are semantically coupled or matched and all are connected in the active ontology 1050, this indicates greater strength of evidence from semantic parsing than if these correspondence nodes were not connected or connected by Longest connection routes in the active 1050 ontology. For example, in a semantic correspondence mode 220, to the syntactic analysis that couples both Local Event in Site of
Meeting and Meeting Site Name, it is given increased evidence support due to the combined representations of these aspects of the user's intention are connected by links and / or relationships in active 1050 ontology: in this case, the
120
IMPI ustitutq mbxicano Dt THE INDUSTRIAL PROPERTY
<img file="MX338784B_D0123.tif" />
Local Event node is connected to the Site node - * íe-Jte «iiáa - que_<sub>-</sub>ae.
connects to the Meeting Site Name node that is connected to the entity name in the meeting site names database.
In at least one modality, connections between nodes in an active ontology that provide evidence support to clarify or lessen ambiguity between candidate semantic parsing results 222 are directed arcs, which form an inference network, where corresponding nodes provide evidence by nodes to which they are connected by direct arcs.
At 232, the 1070 language interpreter components classify and select the top semantic parsing at 232 as the user intent representation 290.
Domain Entity Database 1072
In at least one embodiment, the domain entity databases 1072 may be operable to perform and / or implement various types of functions, operations, actions, and / or other characteristics such as, for example, one or more of the following (or their combinations):
• Store data regarding domain entities. Domain entities are things in the world or computing environment that can be modeled in domain models. Examples may include, but are not limited to one or more of the following (or
121
IMPI
INSTITUTO MSXICANO Dt LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0124.tif" />
their combinations): <sup>1 1</sup>
Businesses of any kind;
Movies, videos, songs and / or other musical products, and / or any other named entertainment products;
Products of any kind;
Events;
Calendar entries;
Cities, states, countries, neighborhoods and / or other geographic, geopolitical and / or geospatial points or regions;
Named sites such as landmarks, airports, and the like;
• Provide database services, in these databases, including but not limited to simple and complex queries, transactions, triggered events, and the like.
According to specific modalities, multiple instances or threads of 1072 domain entry databases can be implemented and / or started concurrently by the use of one or more processors 63 and / or other combinations of hardware and / or hardware physical or software. For example, in at least some modalities, various aspects, characteristics and / or functionalities of 1072 domain entity databases may be performed, implemented and / or started by the database software and / or hardware residing in the or the
122
ΙΜΡΙ
MEXICAN INSTITUTE M INDUSTRIAL PROPERTY
<img file="MX338784B_D0125.tif" />
1304 clients and / or 1340 servers.
An example of a domain entity database 1072 can be used in connection with the present invention in accordance with one embodiment, it is a database of one or more business storages, for example, their names and locations. The database can be used, for example, to search for words contained in a feed request for corresponding businesses and / or to search for the location of a business whose name is known. A person skilled in the art will recognize that many other setups and implementations are possible.
Vocabulary Co-Speakers 1058
In at least one embodiment, vocabulary components 1058 may be operable to perform and / or implement various types of functions, operations, actions, and / or other features such as, for example, one or more of the following (or combinations thereof):
• Provide databases that associate words and strings with concepts, properties, relationships, or instances of domain models or task models;
• Vocabulary of vocabulary components can be used by automated assistant 1002 for various processes, including for example: producing power, interpreting natural language, and generating output.
According to specific modalities, multiple
123
ΙΜΡΙ TÍSTITUTO MHIICANO SEES INDUSTRIAL PROPERTY
<img file="MX338784B_D0126.tif" />
Instances or threads of vocabulary components 1058 may ϋΰί be implemented concurrently and / or initiated by the use of one or more processors 63 and / or other combinations of hardware and / or hardware and software. For example, in at least some modalities, various aspects, characteristics and / or functionalities of 1058 vocabulary components can be implemented as data structures that associate strings with the name of concepts, relationships, properties, and instances. These data structures can be stored in memory, files or databases. Access to 1058 vocabulary components can be implemented through direct APIs, network APIs, and / or database query interfaces. Creating and maintaining 1058 vocabulary components can be accomplished by direct file editing, database transactions, or through the use of domain model editing tools. 1058 vocabulary components can be implemented as part of or in association with active 1050 ontologies. A person skilled in the art will recognize that many other arrangements and implementations are possible.
According to different modalities, one or more different threads or instances of 1058 vocabulary components may be initiated in response to detection of one or more conditions or events that satisfy one or more different types of minimum threshold criteria to trigger the initiation of at least an instance of vocabulary components 1058. In one modality, the
124
IMPI
MÍXICANO INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0127.tif" />
Vocabulary Components 1058 is accessed when vocabulary information is required, including for example during produce feed, interpret feed and format output for users. A person skilled in the art will recognize that other conditions or events may trigger the initiation and / or implementation of one or more different 1058 vocabulary component threads or instances.
In at least one embodiment, a given instance of 1058 vocabulary components can access and / or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information can be accessed by communication with one or more local and / or remote memory devices. In one embodiment, the 1058 vocabulary components can access data from external databases, for example, from a dictionary or data warehouse.
1060 Language Pattern Recognition Components
In at least one embodiment, the language pattern recognition components 1060 may be operable to perform and / or implement various types of functions, operations, actions, and / or other features, such as, for example, searching for patterns in language feed or speaks that indicate grammatical, idiomatic, and / or other compounds of signs or feeding samples. These patterns correspond, for example, to one or more of the following (or their combinations):
125
<img file="MX338784B_D0128.tif" />
IMPI
MSKICAN INSTITUTE
Say THE PROHBDAO
INDUSTRIAL words, names, phrases, data, parameters, commands and / or tags of speech acts.
In accordance with specific embodiments, multiple instances or threads of pattern recognition components 1060 may be concurrently implemented and / or initiated by the use of one or more processors 63 and / or other combinations of hardware and / or hardware and software. For example, in at least some embodiments, various aspects, features, and / or functionalities of 1060 language pattern recognition components may be implemented, implemented, and / or started by one or more files, databases, and / or programs containing expressions. in a pattern matching language. In at least one embodiment, the 1060 language pattern recognition components are represented declaratively, rather than as a program code; This allows them to be created and maintained by editors and other tools other than programming tools. Examples of declarative representations can include, but are not limited to, one or more of the following (or combinations thereof): regular expressions, pattern matching rules, natural language grammars, state machine-based parsers, and / or other models of parsing.
A person skilled in the art will recognize that other types of systems, components, systems, devices, procedures, processes, and the like (or combinations thereof)
126
ΙΜΡΪ
INSTITUTO MWICANO M LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0129.tif" />
can they be used to implement components ten? ? i my 1060 language pattern.
According to different modalities, one or more different threads or instances of 1060 language pattern recognition components may be initiated in response to detection of one or more conditions or events that satisfy one or more different types of minimum threshold criteria to trigger start of at least one instance of 1060 language pattern recognition components. Various examples of conditions or events that may trigger the start and / or implementation of one or more different threads or instances of 1060 language pattern recognition components may include, but are not limited to, one or more of the following (or their combinations):
• during active production of rendering, where the structure of language pattern recognition agents can restrict and guide the user's diet;
• during natural language processing, where language pattern recognition agents help interpret feeding as language;
• during the identification of tasks and dialogues, where language pattern recognition agents can help to identify tasks, dialogues and / or stages there.
In at least one mode, a given instance of 1060 language pattern recognition components can access and / or use information from one or more databases
127
NS MEE INSTITUTE [CANO DE LA BIHISTÍIAL BROTHERHOOD
<img file="MX338784B_D0130.tif" />
associated. In at least one embodiment, at least a portion of the database information can be held by communication with one or more local and / or remote memory devices. Examples of different types of data that can be accessed by the 1060 language pattern recognition components may include, but are not limited to, data from any of the models, various models, and data sources that may be part of Wizard modes 1002, which may include, but are not limited to, one or more of the following (or combinations thereof):
• 1056 domain models;
• Vocabulary 1058;
• 1072 domain entity database;
• Short-term memory 1052;
• Long-term personal memory 1054;
• 1086 Task Flow Models;
• Dialogue flow models 1087;
• 1088 service capacity models.
In one mode, data access from other mode parts of the wizard 1002 can be coordinated by active 1050 ontologies.
Again with reference to Figure 14, an example of some of the various types of functions, operations, actions and / or other features that may be provided by the pattern recognition components is shown.
128
IMPI
MEXICAN INSTITUTE OF PROPERTY
INDUSTRIAL
<img file="MX338784B_D0131.tif" />
1060 language. Figure 14 illustrates patterns to léliij Lia that · can recognize the 1060 language pattern recognition components. For example, the passing expression (in a city) may be associated with the event planning task and the domain of local events.
1080 Dialogue Flow Processor Components
In at least one embodiment, the 1080 dialogue flow processor components may be operable to perform and / or implement various types of functions, operations, actions, and / or other characteristics such as, for example, one or more of the following (or their combinations):
• Given a representation of the user's intention 290 of language interpretation 200, identify the task that a user wants to perform and / or a problem that the user wants to solve. For example, a task may be to find a restaurant.
• For a given problem or tasks, given a representation of the user's intention 290, identify parameters for the task or problem. For example, the user can search for a recommended restaurant that serves Italian food near the user's home. The restrictions are that a restaurant is recommended, that it serves Italian food and close to home are parameters for the task of finding a restaurant.
Given the current task interpretation and dialogue with the
IMPI
MIXICAN INSTITUTE
OL THE MOMITY
INDUSTRIAL
129
<img file="MX338784B_D0132.tif" />
user, such that it can be represented in personal short-term memory 1052, select an appropriate dialog flow model and determine a stage in the flow model corresponding to the current state.
According to specific modalities, multiple instances or threads of the 1080 dialogue flow processor components can be implemented concurrently and / or initiated by the use of one or more processors 63 and / or other combinations of hardware and / or hardware and software.
In at least one embodiment, a given instance of 1080 stream flow processor components can access and / or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information can be accessed by communication with one or more local and / or remote memory devices. Examples of different types of data that can be accessed by 1080 dialog flow processor components may include, but are not limited to, one or more of the following (or combinations thereof):
• 1086 task flow models;
• 1056 domain models;
• 1087 dialogue flow models.
Now referring to Figures 30 and 31, screen shots are shown illustrating an example of various types of functions, operations, actions and / or other features that
130
IMPI
MWCfCANO D £ LA RftOPIKOAU INDUSTRIAL INSTITUTE
<img file="MX338784B_D0133.tif" />
they can be provided by process components of dialogue flow according to one modality.
As shown on screen 3001, the user requests a dinner reservation by providing speech or text feed and book me a dinner table. Wizard 1002 generates a notice 3003 asking the user to specify time and size of the group or number of participants.
Once these parameters have been supplied, screen 3101 is displayed. Wizard 1002 outputs a dialog box 3102 indicating that the results are presented, and a warning 3103 asking you to click a time. The lists
3104 are also exhibited.
In one embodiment, this dialog is implemented as follows.
The dialogue flow processor components 1080 give a representation of the user intention of the 1070 language interpreter component and determine that the appropriate response is to ask the user for information required to perform the next step in a task flow. In this case, the domain is restaurants, the task is to obtain a reservation, and the dialogue stage is to ask the user for the information required to achieve the next stage in the task flow. This stage of the dialogue is exemplified by message 3003 on screen 3001.
Now also with reference to Figure 32, a flowchart is illustrated illustrating a method for operating the 1080 dialogue flow processor components in accordance with
131 ___ '' T'TV'PMWKANL. ?
D »THE PROPIMMD industrial S ^ Ur-t '' a modality. The flow diagram of the F-ignr ** '__ ^^ ribA an connection with the example shown in Figures 30 and 31.
The method starts at 200. The representation of the user's intention 290 is received. As described in connection with Figure 28 in one embodiment, the user intention representation 290 is a set of semantic analyzes. For the example shown in Figures 30 and 31, the domain is restaurants, the verb is to book associated with restaurant reservations, and the time parameter is the afternoon or night of the current day.
At 310, the components of the 1080 dialogue flow processor determine whether this interpretation of the user's intention is strong enough supported to proceed and / or whether it is better supported than alternate ambiguous parsing. In the current example, interpretation is strongly supported, without competitive ambiguous parsing. If, on the other hand, there are competitive ambiguities or insufficient uncertainty, then step 322 is performed to adjust the dialogue flow stage, such that the execution phase causes the dialogue to output a request or notice for more information by the user.
At 312, the components of the 1080 dialogue flow processor determine the preferred interpretation of semantic analysis with other information to determine the task to be performed and its parameters. Information can be obtained for example from
132
<img file="MX338784B_D0134.tif" />
IMPI
MEXICAN INSTITUTE
SAY THE NOFUOAD
INDUSTRIAL domain models 1056, flow models from Lál'ffS i (J8'b, and / or dialogue flow models 1087 or any of their combinations. In the current example, the task is identified as obtaining a reservation, which It involves both finding a place that can be reserved and available and making a transaction to reserve a table.The task parameters are the time constraint along with others that are inferred in step 312.
At 320, the task flow model is consulted to determine an appropriate next step. Information can be obtained for example from the 1056 domain models, 1086 task flow models and / or 1087 dialog flow models or any of their combinations. In the example, it is determined that in this task flow the next stage is to produce the missing parameters to an availability search for restaurants, resulting in notice 3003 illustrated in Figure 30, requesting the number of people and the time for a reservation .
As described above, Figure 31 illustrates the screen 3101 on which it is shown that includes the dialog item 3102 that is presented after the user responds to the request for the group number or number of people for the table and time for reservation. In one embodiment, screen 3101 is presented as the result of another iteration through an automated call and answer procedure as outlined.
133
IMPI
INSTITUTO MKICANO • S THE INDUSTRIAL PROPERTY
<img file="MX338784B_D0135.tif" />
describes, in connection with Figure 33, what is another call to the dialogue and flow procedure illustrated in Figure 32. In this representation of the flow and dialogue procedure, after receiving user preferences, the flow processor components of Dialogs 1080 determine a different task flow stage in step 320: to perform an availability search. When request 390 is built, it includes enough task parameters for dialogue flow processor components 1080 and service orchestration components 1082 to dispatch to a restaurant reservation service.
Components of the 1087 Dialogue Flow Model
In at least one embodiment, the dialogue flow model components 1087 can be operable to provide dialogue flow models, representing the steps that are taken in a particular type of conversation between a user and an intelligent automated assistant 1002. For example, the dialog flow for the generic task of performing a transaction includes steps to obtain the necessary data for the transaction and confirm the transaction parameters before performing it.
Task Flow Model Condolients 1086
In at least one embodiment, the 1086 task flow model components can be operable to provide task flow models, representing the steps taken to solve a problem or to address or meet a need. By
134
IMPI • «τιτυτο MMlCANO
M THE INDUSTRIAL RROFIÉDAD
<img file="MX338784B_D0136.tif" />
For example, the task flow to obtain a 7 i.i__Liuuuion for dinner involves finding a desirable restaurant, checking availability, and making a transaction to obtain a reservation for a specific time with the restaurant.
According to specific modalities, multiple instances or threads of the 1086 task flow model components can be concurrently implemented and / or started by using one or more processors 63 and / or other combinations of hardware and / or hardware hardware and software or programs. For example, in at least some embodiments, various aspects, features, and / or functionalities of the 1086 task flow model components can be implemented as programs, state machines, or other ways to identify an appropriate stage in a flow diagram.
In at least one embodiment, 1086 task flow model components can employ a task modeling structure called generic tasks. Generic tasks are abstractions that model the stages in the task and their required feeds and generated outputs, without being domain specific. For example, a generic transaction task may include steps to collect data required for the transaction, execute the transaction, and output transaction results - all without reference to any particular transaction domain or service to implement it. It can be represented by a domain such as purchases, but it is
135
ΙΜΡΙ »« τιτυτο MEXICAN • ϊ INDUSTRIAL PROPERTY
<img file="MX338784B_D0137.tif" />
independent of the purchasing domain and can also filUll - apply to booking, programming and similar domains.
At least a portion of the functions, operations, actions, and / or other features associated with the 1086 task flow model components and / or procedures described here can be implemented at least in part using concepts, features, components, processes, and / or or other aspects described here in connection with the generic task modeling structure.
Additionally, at least a portion of the functions, operations, actions, and / or other features associated with the 1086 task flow model components and / or procedures described herein, can be implemented at least in part using concepts, features, components, processes and / or other aspects relating to restricted selection tasks, as described here. For example, a generic task modality can be implemented using a restricted selection task model.
In at least one embodiment, a given instance of 1086 task flow model components may be accessing and / or using information from one or more associated databases. In at least one embodiment, at least a portion of the database information can be accessed by communication with one or more local and / or remote memory devices. Examples of different types of data that can be accessed by
136
IMPI in'í n rtrro msxicano í, t INDUSTRIAL property
<img file="MX338784B_D0138.tif" />
1072;
1086 task flow model components are not limited to one or more of the combinations):
• 1056 domain models;
• Vocabulary 1058;
• Domain entity databases • Short-term memory 1052;
• Long-term personal memory 1054;
• 1087 Dialogue Flow Models;
• 1088 service capacity models.
Now referring to Figure 34, a flowchart is illustrated showing an example of a task flow for a narrow-select task 351 according to one embodiment.
Restricted selection is a type of generic task where the goal is to select some item from a set of items in the world, based on a set of restrictions. For example, a restricted selection task 351 can be represented by the restaurant domain. The 351 restricted selection task begins by requesting criteria and restrictions from user 352. For example, the user may be interested in Asian food and may want a place to eat near their office.
In step 353, assistant 1002 presents items that meet the criteria and restrictions established for the user to view. In this example, it can be a list of · ρλλβ4β '<sup>η 4 11 4 and</sup> following (or their
137
<img file="MX338784B_D0139.tif" />
IMPI
Xstituto Mexicano M LA PROPIEDAD INDUSTRIAL restaurants and their properties, which can iimp lengón pays ·, & or.
selected among them.
At step 354, the user is given an opportunity to refine criteria and constraints. For example, the user can refine the request by saying near my office. The system then presents a new set of results in step 353.
Now referring also to Figure 35, an example of the screen 3501 is shown including the list 3502 of items presented by the restricted selection task 351 according to an embodiment.
At step 355, the user can select from the corresponding items. Any of a number of 359 follow-up tasks can be made available such as booking 356, remembering 357, or sharing 358. In various modalities, monitoring tasks 359 may involve interacting with network-enabled services, and / or with local functionality for the device (such as setting a calendar appointment, making a phone call, sending an email, or a message text, place an alarm and the like).
In the example in Figure 35, the user can select an item from list 3502 to see more details and take additional actions. Now also with reference to the
Figure 36 shows an example on screen 3601 after the user has selected an item from list 3502.
138
ΙΜΡΙ.
INSTITUTO MEXICANO VM LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0140.tif" />
exhibit additional information and options corrgBpui'idiintco to lao>
follow-up tasks 359 related to the selected item.
In various modes, the flow steps can be offered to the user in any of several feed modes, including but not limited to any combination of explicit dialog prompts and / or GUI links.
Service Co-Speakers 1084
The service components 1084 represent the set of services that the intelligent automated assistant 1002 can call on behalf of the user. Any service that may be requested can be offered in a 1084 service component.
In at least one embodiment, service components 1084 may be operable to perform and / or implement various types of functions, operations, actions, and / or other features such as for example one or more of the following (or combinations thereof):
• Provide the functions over an API that are normally provided to a network-based user interface to a service. For example, a review site may provide an API service that automatically returns reviews of a given entity when requested by a program. The API provides the intelligent automated assistant 1002 with the services that a human would otherwise obtain by operating the web site user interface.
• Provide the functions on an API that normally
139
<img file="MX338784B_D0141.tif" />
IMPI
MEXICAN INSTITUTE
OF THE PROPERTY
INDUSTRIAL provide by a user interface to an applddülüll. For example, a calendar application can provide a service API that automatically returns calendar entries when requested by a program. The API offers the intelligent automated assistant 1002 the services that a human would otherwise obtain by operating the application's user interface. In one embodiment, wizard 1002 is capable of initiating and controlling any of a number of different functions available on the device. For example, if Assistant 1002 is installed on a smartphone, personal digital assistant, tablet computer, or other device, Assistant 1002 may perform functions such as: launching applications, making calls, sending emails and / or text messages , add calendar events, place alarms and the like. In one mode, these functions are activated using the 1084 service components.
• Provide services that are not currently implemented in a user interface, but are available through an API for the wizard on larger tasks. For example, in one mode, for an API to take a street address and return machine-readable geo-coordinates, it can be used by wizard 1002 as a 1084 service component even if it does not have a direct user interface on the network or a device .
According to specific modalities, multiple instances or threads of 1084 service components can be
140
IMPI
USTITUTO MÍXICANO Di LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0142.tif" />
implemented concurrently and / or startedAe-jaadiajaie ^ & Ljiaa. of one or more processors 63 and / or other combinations of hardware and / or hardware and software. For example, in at least some embodiments, various aspects, features, and / or functionalities of service components) 1084 may be performed, implemented, and / or initiated by one or more of the following types of systems, components, systems, devices, procedures, processes and the like (or their combinations):
• implementation of an API exposed by a service, locally or remotely or any combination;
• inclusion of a database within the automated assistant 1002 or a database service available to the assistant 1002.
For example, a web site that offers users an interface to view movies can be used by an intelligent automated assistant mode 1002 as a copy of the database used by the web site. 1084 service components will then offer an internal API to the data, as if it were provided over an API network, even when the data is held locally.
As another example, the 1084 service components for an intelligent automated assistant 1002 that assists with restaurant selection and meal planning may include any or all of the following sets of services that are available from third parties on the network:
ΙΜΡΙ
MIXICAN INSTITUTE
M LA WOMIDAD
INDUSTRIAL
141
<img file="MX338784B_D0143.tif" />
• a set of restaurant list services that cite restaurants that correspond to name, location, or other restrictions;
• a set of restaurant rating services that return ratings for named restaurants;
• a set of restaurant review services that return written reviews by named restaurants;
• a geocoding service to locate restaurants on a map;
• a reservation service that allows programmatic reservation of tables in restaurants.
Service Orchestration Components 1082
Service Orchestration Components 1082 of Intelligent Automated Assistant 1002 runs a service orchestration procedure.
In at least one embodiment, the service orchestration components 1082 may be operable to perform and / or implement various types of functions, operations, actions, and / or other features such as for example one or more of the following (or combinations thereof) :
• Determine dynamically and automatically which services can fulfill the user's request and / or domains and specified tasks;
• Dynamically and automatically request multiple services, in any combination of concurrent order and
142
IMPI
MEXICAN INSTITUTE OE INDUSTRIAL PROPERTY
<img file="MX338784B_D0144.tif" />
sequential;
• Dynamically and automatically transform task parameters and constraints, to meet the service API feed requirements;
• Dynamically and automatically monitor and collect results from multiple services;
• Dynamically and automatically merge service result data from various services into a simplified result model;
• Orchestrate a plurality of services to comply with the restrictions of a request;
• Orchestrate a plurality of services to record an existing set of results with auxiliary information;
• Output the result of requesting a plurality of services in a uniform, independent service representation that unifies the results of various services (for example, as a result of requesting multiple restaurant services to return restaurant lists, merging the data into at least one restaurant of the various services, removing redundancy).
For example, in some situations, there may be several ways to accomplish a particular task. For example, user feed such as remind me to leave for my meeting on the other side of town at 2 pm specifies an action to be accomplished in at least three ways: setting the alarm; create a calendar event;
143
IMPI
MIKtCANO INSTITUTE OF XDUSTJUAL PROPERTY
<img file="MX338784B_D0145.tif" />
or request a stuff manager for hacei ··! In one · modality ,.
the 1082 service orchestration components determine how the request is best met.
The 1082 service orchestration components can also make determinations of which combination of various services will be best to invoke to perform a given total task. For example, finding and reserving a dinner table, 1082 service orchestration components will make the determinations of which services to request in order to perform these functions such as searching for reviews, obtaining availability and making a reservation. Determining which services to use may depend on any number of different factors. For example, in at least one modality, information regarding reliability, serviceability to handle certain types of requests, user feedback, and the like, can be used as factors in determining which service or services are appropriate to invoke.
According to specific modalities, multiple instances or threads of 1082 service orchestration components may be concurrently implemented and / or initiated by the use of one or more processors and / or other combinations of hardware and / or hardware and software .
In at least one mode, a given instance of 1082 service orchestration components can use 1088 explicit service capability models to represent
144
IMPI
INSTITUTO MU, CANO DI LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0146.tif" />
the capabilities and other properties of services' gRLi3iliu & ratio with respect to these capabilities and properties while achieving the characteristics of 1082 service orchestration components. This produces advantages over manual programming of a set of services that may include, for example, one or more of the following (or their combinations):
• Ease of development;
• Robustness and reliability in execution;
• The ability to dynamically add and remove services without interrupt code;
• The ability to implement generally distributed query optimization algorithms that are displaced by properties and capabilities rather than being embedded in the source code or predefined in the code to specific services or APIs.
In at least one mode, a given instance of 1082 service orchestration components can access and / or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information can be accessed by communication with one or more local and / or remote memory devices. Examples of different types of data that can be accessed by 1082 service orchestration components may include, but are not limited to, one or more of the following (or combinations thereof):
Representations of domain models;
145
ΙΝ5ΤΓΠΙΪΟ MEXICAN
OF THE PROPERTY
INDUSTRIAL • Syntactic and semantic analysis of natural language feeding;
• Representations of task models (with values for parameters);
• Models of dialogue and task flow and / or selected stages within them;
• 1088 service capacity models;
• Any other information available in an active 1050 ontology.
Now referring to Figure 37, an example of a procedure for executing a service orchestration procedure according to one embodiment is shown.
In this particular example, a single user is considered who is interested in finding a good place to dine in a restaurant, and with the intelligent automated assistant 1002 he conducts a conversation to help him provide this service.
Consider finding restaurants that are high quality, have good reviews, near a particular location, available for reservation at a particular time, and serve a particular type of food. These domains and task parameters are given as feed 390.
The method starts at 400. At 402, it is determined whether the given request can require any services. In some situations, delegation of services may not be
146
IMPI
MIXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0147.tif" />
required, for example if the assistant 1002 pq-papa ·? Have the desired task done. For example, in one mode, wizard 1002 may be able to answer an objective question without invoking delegation of services. Accordingly, if the request does not require services, then an autonomous flow stage is executed at 403 and its result is returned at 490. For example, if the task request was to request information regarding automated wizard 1002 itself, then the dialog response can be handled without invoking external services.
If, in step 402, it is determined that service delegation is required, the service orchestration components 1082 proceed to step 404. At 404, the service orchestration components 1082 can match the task requirements with the declarative descriptions. of service capabilities and properties in 1088 service capability models. At least one service provider that can support the represented operation, provides declarative qualitative metadata details, for example, one or more of the following (or combinations thereof):
• the data fields that are returned with results;
• what kinds of parameters are known to be statically supported by the service provider;
• parameter policy functions that the service provider may be able to support after dynamic inspection of parameter values;
147
MRXICANO INSTITUTE
Dt THE PROPERTY
INDUSTRIAL
<img file="MX338784B_D0148.tif" />
• a performance rating that it defines as (eg relationship DB, network service, triple store, full-text index, or some combination thereof);
• property quality ratings that statically define the expected quality of property values returned with the resulting object;
• A total quality rating of the results that can be expected from the service.
For example, reasoning regarding the kinds of parameters that the service can support, a service model can establish that services 1, 2, 3, and 4 can provide restaurants that are close to a particular location (a parameter), the services 2 and 3 can filter or rate restaurants by quality (another parameter), services
3, 4, and 5 can return reviews by restaurant (a return data field), service 6 can cite the types of food served by restaurants (a returned data field), and service 7 can verify the availability of restaurants for particular time intervals (one parameter). Services 8 to 99 offer capabilities not required for this particular domain and tasks.
Using these qualitative, declarative metadata, the task, task parameters, and other information available from the wizard runtime environment, the components
148
IMPI ^ τπτυτοΜκιοΛΝο O »THE INDUSTRIAL FREEDOM
<img file="MX338784B_D0149.tif" />
Orchestration Service 1082 determine 'grT' 411¾ Lili ceii'i optimal board of service providers to invoke. The optimal set of service providers can support one or more task parameters (returning results that satisfy one or more parameters) and also considers the performance rating of at least one service provider and the overall quality rating of at least one provider of services.
The result of step 404 is a list of services dynamically generated to request this particular user and request.
<td>In</td><td>at least</td><td>a</td><td>modality,</td><td>the</td><td>components</td><td>of</td>
<td>orchestration</td><td>of services</td><td> 1082</td><td>consider</td><td>the</td><td>reliability</td><td>of</td>
<td>services like this</td><td colspan="2">like your ability</td><td colspan="2">in order to respond</td><td>to requests</td><td>of</td>
<td colspan="2">specific information.</td><td></td><td></td><td></td><td></td><td></td>
<td>In</td><td>at least</td><td>a</td><td>modality,</td><td>the</td><td>components</td><td>of</td>
<td>orchestration</td><td>of services</td><td> 1082</td><td>they cover</td><td colspan="2">against lack</td><td>of</td>
<td>reliability</td><td>when requesting</td><td colspan="3">overlapping services</td><td>or redundant.</td><td></td>
<td>In</td><td>at least</td><td>a</td><td>modality,</td><td>the</td><td>components</td><td>of</td>
Service Orchestration 1082 considers personal information regarding the user (from the short-term personal memory component) to select services. For example, the user may prefer some rating services over others.
At step 450, service orchestration components 1082 dynamically and automatically invoke multiple services on behalf of a user.
In at least one modality
IMPI
INJTITUTO MEXICANO Say LA «ORI INDUSTRIAL AGE
<img file="MX338784B_D0150.tif" />
these are dynamically invoked while -goopoÍidian -a- - a user request. According to specific modalities, multiple instances or threads of the services can be requested concurrently. In at least one embodiment, they are requested over a network using APIs, or over a network using network service APIs, or over the Internet using
Network service APIs, or any combination thereof.
In at least one modality, the speed at which services are requested is limited and / or managed programmatically.
Also now referring to Figure 38, an example of a service invocation procedure 450 according to one embodiment is shown. The invocation of services is used, for example, to obtain additional information or to carry out tasks for the use of external services. In one mode, request parameters are transformed as appropriate for the service API. Once results are received from the service, the results are transformed into a results representation for presentation to the user within wizard 1002.
In at least one embodiment, services invoked by service invocation procedure 450 may be a network service, an application running on device, operating system function, or the like.
The representation of request 390 is provided, including for example task parameters and the like. To the
<img file="MX338784B_D0151.tif" />
150
<img file="MX338784B_D0152.tif" />
minus one service available from the 'give' service capacity 1088 models, service invocation procedure 450 performs transformation steps 452, request 454, and output mapping 456.
In transformation step 452, the current task parameters of request representation 390 are transformed into a form that can be used by at least one service.
Parameters to services, which can be offered as APIs or databases, may differ from the data representation used in task requests, and also from each other as a minimum. Accordingly, the objective of step 452 is to map at least one task parameter into the one or more formats and corresponding values in at least one service being requested.
For example, the names of businesses such as restaurants may vary through services that deal with these businesses. Accordingly, step 452 will involve transforming any names into forms that are more suitable for at least one service.
As another example, locations are known at various levels of precision and using various units and conventions across services. Service 1 may require postal codes, service 2 GPS coordinates, and service 3 postal address.
The service is requested in 454 on an API and its data is obtained. In at least one modality, results are sent
151 to buffer or cache. In at least one mode, IOS services that do not return within specified level performance (for example, as specified in the Convention a
Service Level (SLA = Service Level Agreement) are withdrawn.
IMPI
NBUCANO INSTITUTE W LA MONEDAD
INDUSTRIAL
<img file="MX338784B_D0153.tif" />
In the output mapping stage 456, the data returned by a service is mapped back into a unified result representation 490. This stage may include dealing with different formats, units, and so on.
In step 412, results from multiple services are validated and merged. In one modality, if validated results are collected, an equality policy function - defined on a per domain basis - is requested in pairs through one or more results to determine which results represent identical concepts in the real world. When a pair of equal results is discovered, a set of property policy functions also defined on a per-domain basis - are used to merge property values into a merge result. The ownership policy function can use the property quality ratings of serviceability models, task parameters, domain context, and / or 1054 long-term memory to decide the optimal merge strategy.
For example, restaurant lists from different restaurant providers can be merged and duplicates
152
ΙΜΡΙ
MEXICAN INSTITUTE Dt LA INDUSTRIAL PROPERTY
<img file="MX338784B_D0154.tif" />
backing out. In at least one modality, loo ..... egá-t-egio · »'·· —pay to identify duplicates may include inaccurate name match (less than 100%), inaccurate location match, inaccurate match against multiple properties of Domain entities, such as names, location, telephone number and / or network site address, and / or any combination thereof.
At step 414, the results are sorted and trimmed to return a list of results of the desired length.
In at least one embodiment, a request relaxation loop is also applied. If, at step 416, the service orchestration components 1082 determine that the current list of results is not sufficient (for example, it has less than the desired number of matching items), then the task parameters should be relaxed 420 to allow more results. For example, if the number of restaurants of the desired classification found within N kilometers of the target location is very small, then relaxation will execute the request again, searching an area greater than N kilometers away and / or relaxing some other search parameter.
In at least one modality, the service orchestration method is applied in a second step to annotate results with auxiliary data that is useful for the task.
At step 418, service orchestration components 1082 determine whether annotation is required. Can
153
IMPI
MÍXICANO K INSTITUTE> THE INDUSTRIAL FROMÍDAD
<img file="MX338784B_D0155.tif" />
required if, for example, the task can request the results on a map, but the primary services do not return geo-coordinates required for mapping.
At 422, the 1088 service capacity models are queried again to find services that can return the desired extra information. In one modality, the annotation process determines whether better or additional data can be annotated to a merge result. They do this by delegating to a property policy function - defined on a per-domain basis - for at least one property of at least one merged result. The property policy function can use the merged property value and property quality rating, property quality ratings of one or more other service providers, domain context and / or user profile, to decide if better data can be obtained. If it is determined that one or more service providers can annotate one or more properties for a merged result, a cost function is invoked to determine the optimal set of service providers to annotate.
At least one service provider in the optimal set of annotation service providers is invoked at 450 with the list of results merged, to get results 424.
Changes made to at least one merged result in at least one service provider are tracked during this process, and the changes are then merged using the same
154
IMPI
MEXICAN INSTITUTE M industrial PROPERTY
<img file="MX338784B_D0156.tif" />
property policy function process which umplcu ti * —latapa 412. Your results are merged at 426 into the existing result set.
The resulting data is classified at 428 and unified into a 490 uniform representation.
It can be seen that an advantage of the methods and systems described above over service orchestration components 1082 is that they can be advantageously applied and / or used in various fields of technology, other than those related specifically to intelligent automated assistants. Examples of these other technology areas where aspects and / or characteristics of service orchestration procedures include, for example, one or more of the following:
• Network-hybrid applications on network sites and network-based applications and services;
• Optimization of queries to distributed databases;
• Dynamic service-oriented architecture configuration.
Components of 1088 Service Capability Models
In at least one embodiment, the 1088 service capability model components may be operable to perform and / or implement various types of functions, operations, actions, and / or other features such as, for example, one or more of the following (or their combinations):
155
ΙΜΡΙ
MWCICANt INSTITUTE »Ol la INDUSTRIAL PROFIRUAD
<img file="MX338784B_D0157.tif" />
• Provide human-readable information regarding service capabilities to perform certain kinds of computation;
• Provide machine-readable information regarding the capabilities of services to respond to certain kinds of queries;
• Provide machine-readable information regarding what kinds of transactions are provided by various services;
• Provide machine-readable information regarding API parameters exposed to various services;
• Provide machine-readable information regarding parameters that can be used in database queries that are provided by various services.
Processor Components fie Output 1090
In at least one embodiment, the output processor components 1090 may be operable to perform and / or implement various types of functions, operations, actions, and / or other features such as, for example, one or more of the following (or combinations thereof) ):
• Format output data that is represented in a uniform internal data structure, in forms and arrangements that make it appropriately in different modalities. The output data may include, for example, natural language communication between the intelligent automated assistant and the user;
156
IMPI
MEXICAN INSTITUTE OF PROPERTY
INDUSTRIAL
<img file="MX338784B_D0158.tif" />
data regarding domain entities, such düinu {Jltipitídadea of · restaurants, movies, products, and the like; domain-specific data results from information services, such as weather reports, flight status checks, pricing, and the like; and / or interactive links and buttons that allow the user to respond by direct interaction with the output presentation.
Present output data for modalities that may include, for example, any combination of: graphical user interfaces; text messages; email messages; sounds; animations; and / or speech output.
Dynamically present data for different graphical user interface display engines based on the request. For example, use of different output processing formats and distributions depending on which display and / or device is used.
Present output data in different voices dynamically.
• Dynamically present to specified modalities based on user preferences.
• Dynamically present the output using user-specific covers that tailor the look and feel or perception.
• Send a stream of outgoing packets to a modality, showing intermediate status, feedback or
157
<img file="MX338784B_D0159.tif" />
I to ΡI
INSTITUTO MEXICANO DI LA mOFIEDAD industrial results through phases of interaction with 100? - „
In accordance with specific embodiments, multiple instances or threads of the output processor components 1090 may be concurrently implemented and / or initiated by the use of one or more processors 63 and / or other combinations of hardware and / or hardware and software. For example, in at least some embodiments, various aspects, features, and / or functionalities of the output processor components 1090 may be implemented, implemented, and / or initiated by one or more of the following types of systems, components, systems, devices, procedures, processes and the like (or their combinations):
• software modules within the client or server in a form of a smart automated wizard;
• remote request services;
• use a mix of templates and procedure codes.
Now referring to Figure 39, a flow chart is shown illustrating an example of a multi-phase output procedure in accordance with one embodiment. The multi-stage exit procedure includes the automated wizard '1002 that processes stages 702 and multi-stage exit stages 704.
At step 710, a speech feed pronunciation and a speech-to-text component (such as
158
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL MONEDAD
<img file="MX338784B_D0160.tif" />
the component described in connection with the gu-ryi?) int-? rp<sup>,</sup>~^'<sup>l</sup>~<sup>to</sup> pl speaks to produce a set of candidate speech interpretations 712. In one embodiment, the speech-to-text component is implemented using, for example, Nuance Recognizer, available from Nuance Communications, Inc. of Burlington, MA.
Interpretations of candidate speech 712 may be displayed to the user at 730, for example in paraphrased form. For example, the interface can show you said? Alternatives that cite a few alternate text interpretations of the same speech sound sample.
In at least one embodiment, a user interface is provided to allow the user to interrupt and select from candidate speech interpretations.
At step 714, the candidate speech interpretations
712 they are sent to a language interpreter 1070, which can produce user intention representations 716 for at least one candidate speech interpretation 712. At step 732, paraphrased from these user intention representations
716 they are generated and presented to the user. (See related step 132 of procedure 120 in Figure 22).
In at least one embodiment, the user interface allows the user to interrupt and select from the paraphrases of the 732 natural language interpretations.
At step 718, task analysis and dialogues are performed. At step 734, the task interpretations and
159
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0161.tif" />
Domains are presented to the user using an intent paraphrasing algorithm.
Now referring also to Figure 40, a screen shot illustrating an example of output processing according to one embodiment is illustrated. Screen 4001 includes echo 4002 of the user's speech feed generated by step 730. Screen 4001 further includes paraphrasing 4003 of the user's intention generated by step 734. In one embodiment, as illustrated in the example of Figure 40, special formatting / highlighting is used for keywords such as events, which can be used to facilitate user training for interaction with the intelligent automated assistant 1002. For example, by visually observing the formatting of the displayed text, the user can easily identify and interpret back to the intelligent automated assistant that recognizes keywords such as events, next Wednesday, San Francisco, and the like.
Returning to Figure 39, as requests are dispatched at 720 to services and results are dynamically collected, intermediate results may be displayed in the form of 736 real-time feed. For example, a list of restaurants may be returned and then their Reviews can be dynamically populated as the results of the review services arrive. Services may include network enabled services and / or services that have access to
160
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL EROFIEDAD
<img file="MX338784B_D0162.tif" />
information stored locally in the HÍcpngU-iwn and / n hp any other source.
A uniform representation of response 722 is generated and formatted 724 for the appropriate output mode. After the final output format is completed, a different type of paraphrasing may be offered in 738. At this stage, the entire set of results can be analyzed and compared against the initial request. A summary of results or answer to a question can then be offered.
Also referring to Figure 41, another example of output processing according to one embodiment is illustrated. Screen 4101 illustrates paraphrase 4102 of the text interpretation, generated in step 732, real-time feed 4103 generated by step 736, and summary of paraphrase 7104 generated by step 738. Detailed results 4105 are also included.
In one mode, wizard 1002 is capable of generating output in multiple modes. Now referring to Figure 42, a flow chart is shown illustrating an example of multimodal output processing according to one embodiment.
The method starts at 600. Output processor 1090 takes uniform representation of response 490 and formats 612 the response according to the appropriate and applicable device and modality. Step 612 may include device information and mode models 610 and / or data models of
161
<img file="MX338784B_D0163.tif" />
IMPI
MEXICAN INSTITUTE
OF THE PROPERTY
INDUSTRIAL domain 614. ' <sup>one x</sup>
Once the 490 response has been formatted at 612, any one of a number of different exit mechanisms can be employed, in any combination. Examples illustrated in Figure 42 include:
• Generate text message output at 620, which is sent at 630 to a text message channel;
• Generate at 622 the email output, which is sent at 632 as an email message;
• Generate at 624 the GUI output, which is sent at 634 to a device or display on the network for rendering;
• Generate 626 speech output, which is sent in 636 to a speech generation module.
A person skilled in the art will recognize that many other exit mechanisms can be employed.
In one embodiment, the content of the output messages generated by the multi-phase output processor 700 is custom tailored to the multimodal output processing mode 600. For example, if the output mode is 626 speech, the language is used to paraphrase user feed 730, text interpretations 732, task and domain interpretations 734, advance 736, and / or result summaries 738 may be more or Less detailed or use phrases that are easier to understand audibly than in written form. In one modality, the language is tailored in stages of
162
<img file="MX338784B_D0164.tif" />
IMPI
IKTITUTOMSXICANO
SAY THE PROPERTY
INDUSTRIAL multi-stage exit procedure juu; In the various modalities, the multi-phase output procedure 700 produces an intermediate result which is further refined in specific language by multimodal output processing 600.
Components of Short-Term Personal Memory 1052
In at least one embodiment, the short-term personal memory components 1052 may be operable to perform and / or implement various types of functions, operations, actions, and / or other features such as, for example, one or more of the following (or their combinations):
• Maintain a history of the recent dialogue between the wizard mode and the user, including the history of user feeds and their interpretations;
• Maintain a history of recent user selections in the GUI, such as which items were opened or scanned, what phone number was called, what items were mapped, what movie previews were featured, and the like;
• Store dialogue history and user interactions in a database on the client, the server in a specific user session, or in a client session state such as pieces of information stored on the hard drive (cookies) of the display of the network or RAM used by the client;
• Store the list of recent user requests;
• Store the sequence of results of requests
163
IMPI
MIXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0165.tif" />
recent user; —............ ..,. , • Store event click stream history
UI, including button presses, small taps, gestures, voice triggers, and / or any other user feed.
• Store device sensor data (such as location, time, position orientation, movement, light level, sound level, and the like) that can be correlated with interactions with the wizard.
According to specific modalities, multiple instances or threads of short-term personal memory components 1052 may be concurrently implemented and / or initiated by the use of one or more processors 63 and / or other combinations of hardware and / or hardware physical and software.
According to different modalities, one or more different threads or instances of short-term personal memory components 1052 may be initiated in response to detection of one or more conditions or events that satisfy one or more different types of minimum threshold criteria to trigger startup of at least one instance of 1052 short-term personal memory components. For example, short-term personal memory components 1052 may be invoked when there is a user session with the wizard mode 1002, in at least one form or action of feed by the user or response by the system.
In at least one embodiment, a given instance of 1052 short-term personal memory components can have
164
IMPI
MEXICAN INSTITUTE DS INDUSTRIAL PROPERTY
<img file="MX338784B_D0166.tif" />
access and / or use information from one or more · —'teoooo da 'datt partners. In at least one embodiment, at least a portion of the database information can be accessed by communication with one or more local and / or remote memory devices. For example, short-term personal memory components 1052 may access data from long-term personal memory components 1054 (for example, to obtain user identity and personal preferences) and / or local device data regarding time and location, which can be included in short-term memory entries.
Now referring to Figures 43A and 43B, screenshots are shown illustrating an example of using 1052 short-term personal memory components to maintain dialogue context while changing location, according to one embodiment. In this example, the user has asked about local weather conditions, then only says in New York. Screen 4301 shows the initial response, including local weather conditions.
When the user says in New York, the wizard 1002 uses the short-term personal memory components 1052 to access the dialogue context and thus determine that the current domain is weather forecasts. This enables Assistant 1002 to interpret the new mention in New York to mean what is the weather forecast for New York next Tuesday? Screen 4302 shows the answer
165
ΙΜΡΙ
MOOCano INSTITUTE DS LA PHOP (M) AL) INDUSTRIAL
<img file="MX338784B_D0167.tif" />
appropriate, including weather forecasts p'áf'á * NéW lUiJ?: · <sup>1</sup> - In the example of Figures 43A and 43B, what is stored in the short-term memory is not only the words of the power, is it going to rain the day after tomorrow? but the semantic interpretation of the food system as the meteorological domain and the time parameter established the day after tomorrow.
Long Term Personal Memory Components 1054
In at least one embodiment, the long-term personal memory components 1054 may be operable to perform and / or implement various types of functions, operations, actions, and / or other features such as, for example, one or more of the following ( or their combinations):
• To persistently store personal information and data regarding the user, including for example their preferences, identities, authentication credentials, accounts, addresses and the like;
• To store information that the user has collected when using assistant mode 1002, such as the equivalent of bookmarks, favorites, clippings and the like;
• To persistently store saved lists of business entities, including restaurants, hotels, shops, theaters and other meeting places. In one mode, 1054 long-term personal memory components store more than just names or URLs, but they also store enough information
166
IMPI
INSTITUTO MMICANO oe LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0168.tif" />
to retrieve a complete list of enLl'CMUtíS liicluyendw phone numbers, locations on a map, photographs and the like;
• To persistently save saved movies, videos, music, shows, and other entertainment items;
• To persistently store the user's personal calendar (s), to-do lists, reminders and alerts, contact databases, social network lists, and the like;
• To persistently store shopping lists and wish lists or favorites for products and services, coupons and discount codes purchased and the like;
• To persistently store history and transaction receipts or receipts including reservations, purchases, event tickets, and the like.
In accordance with specific embodiments, multiple instances or threads of long-term personal memory components 1054 may be concurrently implemented and / or initiated by the use of one or more processors 63 and / or other combinations of hardware and / or hardware. and software. For example, in at least some embodiments, various aspects, features, and / or functionalities of long-term personal memory components 1054 may be implemented, implemented, and / or started using one or more databases and / or files in (or associated with) ) 1304 clients and / or 1340 servers, and / or residents
167
IMPI
MEXICAN INSTITUTE
OF INDUJTR1AL PROPERTY
<img file="MX338784B_D0169.tif" />
on storage devices. '——
According to different modalities, one or more different threads or instances of long-term personal memory components 1054 may be initiated in response to detection of one or more conditions or events that satisfy one or more different types of minimum threshold criteria to trigger start of at least one instance of 1054 long-term personal memory components. Various examples of conditions or events that may trigger the initiation and / or implementation of one or more threads or different instances of 1054 long-term personal memory components may include, but are not limited to, one or more of the following (or their combinations):
• Long-term personal memory entries can be acquired as a side effect of the user interacting with a mode of the wizard 1002. Any type of interaction with the wizard can lead to additions to long-term personal memory, including viewing, searching, Find, buy, program, acquire, reserve, communicate with other people through an assistant.
Long-term personal memory can also be accumulated as a consequence of users signing up for an account or service, allowing wizard 1002 to access accounts in other services, using wizard service 1002 on a client device with access to other databases of personal information such as calendars, things for
168
IMPI
MflklCANO INSTITUTE OF INDIVIDUAL PHOFISDAD
<img file="MX338784B_D0170.tif" />
make, contact lists, and the like. .....
In at least one embodiment, a given instance of the 1054 long-term memory components can access and / or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information can be accessed by communication with one or more local and / or remote memory devices, which may be located, for example, on clients 1304 and / or or 1340 servers. Examples of different types of data that can be accessed by long-term memory components 1054 may include, but are not limited to data from other personal information databases such as contact or friend lists, calendars, To-do lists, other list managers, personal accounts, and portfolio managers that are provided by external 1360 devices and the like.
Now referring to Figures 44A-44C, prints or screen shots are shown illustrating an example of the use of long-term personal memory components 1054, in accordance with one embodiment. The example provides a feature (called My Stuff), which includes access to saved entities such as restaurants, movies, and businesses that are found by interactive sessions with a wizard mode 1002. In screen 4401 of Figure 44 A, the user has found a restaurant. User tap Save to My
169
ΙΜΡΙ
HSTITUTO MEXICANO OE industrial property
<img file="MX338784B_D0171.tif" />
Things 4402, which stores information regarding restaurants and Icfs * components of long-term personal memory 1054.
Screen 4403 in Figure 44B illustrates user access to My Stuff. In one modality, the user can select between categories to navigate to the desired item.
Screen 4404 in Figure 44C illustrates the category
My Restaurant, including items previously stored in My
Things.
Automated Call and Response Procedure
Now referring to Figure 33, a flowchart illustrating an automatic call and answer procedure is illustrated, according to one embodiment. The procedure of Figure 33 can be implemented in connection with one or more modalities of the intelligent automated assistant
1002. It can be appreciated that the intelligent automated assistant 1002 as illustrated in Figure 1 is only one example of a wide range of intelligent automated assistant system modalities that can be implemented. Other modalities of intelligent automated assistant systems (not shown) may include additional, fewer and / or different components / features than those illustrated, for example in exemplary intelligent automated assistant 1002 illustrated in Figure 1.
In at least one embodiment, the automated call and answer procedure of Figure 33 may be operable to
170
<img file="MX338784B_D0172.tif" />
perform and / or implement various types of functions, opUiaLÍunej¡jT ~~ —— actions and / or other characteristics such as, for example, one or more of the following (or combinations thereof):
• The automated call and answer procedure of Figure 33 can provide a loop for interface control flow of a conversation interface between the user and the intelligent automated assistant 1002. At least one iteration of the automated call and answer procedure can serve as a cape or trip in conversation. A conversation interface is an interface in which the user and the assistant
1002 They communicate by making mentions both ways, in a form of conversation.
• The automated call and answer procedure of Figure 33 can provide the executive control flow for the intelligent automated assistant 1002. That is, the procedure controls the power harvesting, power processing, output generation, and output presentation to the user.
• The automated call and answer procedure of Figure 33 can coordinate communications between components of the intelligent automated assistant 1002. That is, it can be direct when the output of one component feeds another, and where total feeding of the environment can occur and action in the environment.
In at least some modalities portions of
171
IMPI iwmvro mwicano OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0173.tif" />
Automated call and answer procedure can also be implemented on other devices and / or systems on a computer network.
According to specific modalities, multiple instances or threads of the automated call and answer procedure can be implemented and / or started concurrently by the use of one or more processors 63 and / or other combinations of hardware and / or hardware and software.
In at least one embodiment, one or more or select portions of the automated call and answer procedure may be implemented on one or more clients 1304, on one or more servers 1340, and / or combinations thereof.
For example, in at least some modalities, various aspects, features, and / or functionalities of the automated call and answer procedure can be performed, implemented, and / or initiated by software components, network services, databases, and / or the like. , or any combination thereof.
According to different modalities, one or more different threads or instances of the automated call and answer procedure may be initiated in response to detection of one or more conditions or events that satisfy one or more different types of criteria (such as, for example, minimum threshold criteria) to activate an automated call and answer procedure instance from the menu.
172
IMPI
MEXICAN INSTITUTE OF INDUÍTRIAL EMOEISTY
<img file="MX338784B_D0174.tif" />
Examples of various types of conditions or eOThtüij 'that' may trigger initiation and / or implementation of one or more threads or different instances of the automated call and response procedure may include, but are not limited to, one or more of the following (or their combinations):
• a user session with an instance of the intelligent automated wizard 1002, such as, but not limited to, one or more of:
a mobile device application that launches for example a mobile device application that implements an intelligent automated assistant mode 1002;
a computer application that starts, for example an application that implements a modality of the intelligent automated assistant 1002;
a dedicated button on a mobile device that is pressed, such as a speech power button;
a button on a peripheral device connected to a computer or mobile device, such as a headset, micro phone, or base station, a GPS navigation system, consumer device, remote control, or any other device with a button that can be associated with invoking assistance ;
a session on the network that starts from a network viewer to a network site that implements the intelligent automated assistant 1002;
an interaction that starts within a session of
173
<img file="MX338784B_D0175.tif" />
IMPI
MEXICAN INSTITUTE
OF LA FROFIEDAO
INDUSTRIAL display of the existing network to a network site <sup>i</sup>what<sup>,,</sup>ímpí''enieiiLa the intelligent automated assistant 1002, where for example a service of the intelligent automated assistant is requested
1002;
an email message that is sent to a mode 1426 server that is mediating communication with a mode of the intelligent automated wizard 1002;
a text message is sent to a mode server
1426 that media communication with a modality of the intelligent automated assistant 1002;
a phone call is made to a mode 1434 server that mediates a mode of the intelligent automated assistant 1002;
an event such as an alert or notification is sent to an application that provides a modality of the intelligent automated assistant 1002.
• when a device that provides the intelligent automated assistant 1002 is activated and / or started.
According to different modalities, one or more different threads or instances of the automated call and answer procedure can be started and / or implemented manually, automatically, statically, dynamically, concurrently and / or their combinations. Additionally, different instances and / or modalities of the automated call and response procedure may be initiated at one or more different
174
ΙΜΡΙ Mexican institute DI industrial PROPERTY
<img file="MX338784B_D0176.tif" />
different times (for example, during a specific inC ^ l'Value, at regular periodic intervals, at irregular periodic intervals, on demand and the like).
In at least one mode, a given instance of the automated call and answer procedure can use
<td colspan="3">and / or generate various types</td><td>different</td><td colspan="3">data and / or other types</td><td>of</td>
<td>information,</td><td>when</td><td>I know</td><td>perform</td><td>chores</td><td>and / u</td><td colspan="2">operations</td>
<td>specific.</td><td>This</td><td>can</td><td>include,</td><td>by</td><td>example</td><td>data</td><td>of</td>
power / information and / or data / information output. For example, in at least one embodiment, at least one instance of the automated call and response procedure may access, process, and / or otherwise use information from one or more different types of sources, such as, for example, one or more databases. of data. In at least one embodiment, at least a portion of the database information can be accessed by communication with one or more local and / or remote memory devices. Additionally, at least one instance of the automated call and answer procedure can generate one or more different types of output data / information, which for example can be stored in local memory and / or remote memory devices.
In at least one embodiment, initial configuration of a given instance of the automated call and answer procedure can be performed using one or more different types of initialization parameters. In at least one
175
IMPI
MEXICAN INSTITUTE DS INDUSTRIAL PROPERTY
<img file="MX338784B_D0177.tif" />
At least one portion of the initialization parameters can be accessed by communication with one or more local and / or remote memory devices. In at least one embodiment, at least a portion of the initialization parameters allow an instance of the automated call and answer procedure to correspond to and / or be derived from the input data / feed data.
In the particular example of Figure 33, a single user is considered to have access to an instance of intelligent automated assistant 1002 over a network from an application to the client with speech feed capability. The user is interested in finding a good place to dine in a restaurant, and connects with the intelligent automated assistant 1002 in a conversation to help provide this service.
In step 100, the user is asked to provide a request. The client user interface offers various power modes, as described in connection with Figure 26. These may include, for example:
• a written feed interface, which can invoke a procedure to produce active written feed as illustrated in Figure 11;
• a speech feed interface, which can invoke a procedure to produce active speech feed as illustrated in Figure 22.
• an interface to select supplies from a
176
IMPI
INSTITUTO MMICANO Dt LA RUO El IDAD INDUSTRIAL
<img file="MX338784B_D0178.tif" />
menu, which can invoke the produce power L-üil Last<sup>1</sup> in GU4activa as illustrated in Figure 23.
A person skilled in the art will recognize that other modes of feeding can be provided.
In one embodiment, step 100 may include presenting options left over from a previous conversation with the wizard
1002, for example using the techniques described in the procedure to produce feed with suggestion of active dialogue described in connection with Figure 24.
For example, by one of the methods of producing active feed in step 100, the user can tell assistant 1002, where can I get good Italian here? For example, the user may have said this on a speech feed component. An embodiment of the component to produce active feed 1094 requests a speech-to-text service, prompts the user for confirmation, and then represents confirmed user feed as a uniform annotated feed format 2690.
An embodiment of the language interpreter component 1070 is requested in step 200, as described in connection with Figure 29. The language interpreter component 1070 parses the text feed and generates a list of possible interpretations of the user's intention 290 In a syntactic analysis, the word Italian is associated with Italian-style restaurants; good is associated with the
177
<img file="MX338784B_D0179.tif" />
DE LA PROESDAC INDUSTRIAL recommended by restaurants; and act on or in the vicinity is associated with a location parameter that describes a distance from a global sensor reading (for example, the user's location as given by GPS on a mobile device).
At step 300, the user intention representation 290 is passed to the dialogue flow processor 1080, which implements a modality of a dialogue and flow analysis procedure as described in connection with Figure 32. The dialogue flow processor 1080 determines which interpretation of intentions is most likely, maps the interpretation to domain model instances and parameters of a task model, and determines the next flow stage in a dialogue flow. In the present example, a restaurant domain model is represented with a constrained selection task to find a restaurant by constraints (cuisine style, recommendation level, and proximity constraints). The dialogue flow model indicates that the next stage is to obtain some examples of restaurants that comply with these restrictions and present them to the user.
In step 400, a modality of the flow service orchestration procedure 400 is invoked, using the service orchestration component 1082. A set of services 1084 is invoked on behalf of the user's request to find a restaurant. In one modality, these services
1084 some data contribute to a common result. Your data is
178
MEXICAN INSTITUTE
OF THE PROPERTY
INDUSTRIAL
<img file="MX338784B_D0180.tif" />
merge or merge and the resulting list of<sup>1</sup> gestauvanÍas' — jg · represents in a uniform, service-independent way.
At step 500, output processor 1092 generates a dialog summarizing the results, such as I found some recommended Italian restaurants near here. The output processor 1092 combines this summary with the output result data and then sends the join to a module that formats the output for the user's particular mobile device at step 600.
At step 700, this device-specific output packet is sent to the mobile device, and is displayed on the screen (or other output device) of the mobile device by the client software on the device.
The user views this presentation, and decides to explore different options. If the user has terminated 790, the method terminates. If the user has not finished 490, another iteration of the loop begins upon returning to step 100.
The automatic calls and responses procedure can be applied, for example to a user who asks what is Mexican food? This feeding may occur at stage 100. At stage 200, the feeding is interpreted as Mexican-style restaurants, and combined with the other state (which is kept in short-term personal memory 1052) to support its interpretation. Intention that last time, with a change in the restaurant style parameter. In
179
IMPI
MEXICAN INSTITUTE OF FROEIEDAD * 4 MUTUAL
<img file="MX338784B_D0181.tif" />
step 300, this refinement intention (^ - ί ^ -'- οο-Ι-Ιοίί ^ ά, which is given to service orchestration components 1082 in step 400.
At step 400, the updated request is dispatched to multiple services 1084, resulting in a new set of restaurants that summarizes in dialogue at 500, formats for device 600, and sends over the network to display new information on the mobile device of the user at step 700.
In this case, the user finds a restaurant of his liking, shows it on the map and sends instructions to reach a friend.
A person skilled in the art will recognize that different modalities of the automated call and answer procedure (not shown) may include features and / or operations in addition to those illustrated in the specific modality of Figure 33, and / or may omit at least one portion of the automated call and answer procedure features and / or operations illustrated in the specific mode of Figure 33.
Restricted Selection
In one embodiment, intelligent automated assistant 1002 uses restricted selection in its interactions with the user, to more effectively identify and present items that are likely to be of interest to the user.
Constrained selection is a generic type of task.
180
IMPI
MEXICAN INSTITUTE
<img file="MX338784B_D0182.tif" />
OF THE INDUSTRIAL PRON1DAD
Generic tasks are abstractions that characterize domain objects, feeds, outputs, and control flow that are common among a class of tasks. A restricted selection task is performed by selecting items from a selection set of domain objects (such as restaurants) based on selection restrictions (such as a desired type of cuisine or location). In one embodiment, wizard 1002 assists the user in exploring the space of possible selections, producing user restrictions and preferences, presenting selections, and offering actions to make on those selections such as booking, purchasing, remembering, or sharing.
The task is completed when the user chooses one or more items on which to perform the action.
Narrow selection is useful in many contexts:
for example, selecting a movie to watch, a restaurant for dinner, a hotel for the night, a place to buy a book, or the like. In general, restricted selection is useful when the category is known and requires selecting an instance of the category with certain desired properties.
A conventional approach to restricted selection is a directory service. The user chooses a category and the system offers a list of selections. In a local directory, you can restrict the directory to a location, such as a city. For example, in a yellow telephone directory service the user chooses the book for a city and searches
181
IMPI
INSTITUTO MSXICANO DC LA NOMSOAD INDUSTRIAL
<img file="MX338784B_D0183.tif" />
then the category, and the book shows one or more, · ». fams-for this category. The main problem with a directory service is that the number of possible relevant selections is large (for example, restaurants in a given city).
Another conventional approach is a database application, which provides a way to generate an established selection by producing a user query, retrieve matching items, and present the items in a way that highlights outstanding features. The user views the rows and columns of the resulting set, possibly sorting results or changing the query until he finds some suitable candidates. The problem with the database service is that it may require the user to make their human need operational or functional like a formal query and use the abstract sort, filter, and visualize machinery to explore the resulting data. These are difficult for most people to do, even with a graphical user interface.
A third conventional approach is open-end search, such as local search. Search is easy to do, but there are several problems with search services that make it difficult for people to accomplish the task of narrow selection. Specifically:
• As with directory search, the user can not only supply a category and search for one or more selections
182
<img file="MX338784B_D0184.tif" />
IMPI rwjrrruro meucano
Dt THE PROPERTY
INDUSTRIAL possible, but should narrow the list. ........— • If the user can narrow the selection by restrictions, it is not obvious that restrictions can be used (for example, can I search for sites that are within walking distance or are open late?) • It is not clear how stable restrictions are (for example, what type of restaurant or cuisine, and what are the possible values?) • There is a conflict of multiple preferences; Usually objectively there is no better to a certain situation (for example, you want a place that is close by and has cheap gourmet food with excellent service and that is open until midnight).
• Preferences are relative and depend on what is available. For example, if the user can get a table at a highly rated restaurant, they can choose it even if it is expensive. In general, however, the user will prefer less expensive options.
In various embodiments, the wizard 1002 of the present invention helps simplify the restricted selection task. In various modalities, wizard 1002 employs search services and databases, as well as other functionality, to reduce the effort by the user to establish what they are searching for, considering what is available, and deciding on a satisfactory solution.
In various modalities, the assistant 1002 helps to make
183
IMPI
INSTITUTO MKICANO OI LA FROIHSDAC INDUSTRIAL
<img file="MX338784B_D0185.tif" />
Restricted selection for human ™ an nuaigLiíora is made simpler in a number of different ways.
For example, in one mode, wizard 1002 can operate constrained properties. The user sets what he wants in terms of desired result properties. Assistant 1002 operates this feed in formal constraints. For example, instead of saying find one or more restaurants less than 2 miles from downtown Palo Alto whose cuisine includes Italian food, the user can only say Italian restaurants in Palo Alto. Wizard 1002 can also operationalize user-requested attributes that are not parameters to a database. For example, if the user requests romantic restaurants, the system can also operate this as a text search or label or restriction corresponding to label. In this way, Wizard 1002 helps to overcome some of the problems that users may otherwise have with restricted selection. It is easier for a user to imagine and describe a satisfactory solution that describes conditions that conveniently distinguish from undesirable solutions.
In one embodiment, wizard 1002 can suggest useful selection criteria, and the user only needs to say which criteria are important at the moment. For example, wizard 1002 may ask which of these matters: price
184
IMPI
<img file="MX338784B_D0186.tif" />
Muucano βκτπυτο
Dt THE INDUSTRIAL NtOniTY
<img file="MX338784B_D0187.tif" />
(the cheaper the better), location (the closer the better), 'rating (the higher the better). The assistent
1002 it can also suggest criteria that may require specific values; for example, you can say what type of cuisine you would like or a food item you would prefer.
In one embodiment, wizard 1002 can assist the user in making the decision between selections that differ by a number of competing criteria (eg, price, quality, availability, and convenience).
By providing this guide, the wizard 1002 can assist users in making multi-parameter decisions in any of several ways:
• One is to reduce the dimensionality of the space, combining raw data such as ratings from multiple sources into a composite recommendation rating. The composite rating may take into account domain knowledge regarding data sources (for example, the Zagat rating may be more predictive of quality than the Yelp rating).
• Another approach is to focus on a sub-set of criteria, past a problem of which are all the possible criteria to consider and how are they combined? in a selection of criteria, most important in a given situation (for example, which is more important, price or proximity?).
• Another form of decision making is assuming predefined values and preference orders (for example, if everything
185
IMPI
USTITUTO MKICANO Dt LA INDUSTRIAL RROREDAD
<img file="MX338784B_D0188.tif" />
others are the same, with higher qualification and more<sup>1</sup> near<sup>1</sup> y '-raá<sup>1</sup>»Economic are better). The system can also remember previous user responses indicating their predefined values and preferences.
• Fourth, the system can offer outstanding item properties in the selection set that were not mentioned in the original request. For example, the user may have ordered local Italian food. The system can offer a set of restaurant selections, and with them a list of popular labels used by critics or a label line from a guide (for example, a nice spot for a great pasta date). This can allow people to select a specific item and complete the task. Research shows that most people make decisions by evaluating specific instances rather than deciding on criteria and rationally accepting the one that stands out. It also shows that people learn about characteristics of specific cases. For example, when choosing between cars, buyers may not care about navigation systems until they see that some of the cars have them (and then the navigation system may become an important criterion). The assistent
1002 it can present outstanding properties of the cited items that help people choose a winner or suggest a dimension on which to optimize.
Conceptual Data Model
186
IMPI
INSTITUTO MMICANO Dt LA «CRIDAD INDUSTRIAL
<img file="MX338784B_D0189.tif" />
In one embodiment, the wizard 1002 ufttíitu · assist with the restricted selection task by simplifying the conceptual data model. The conceptual data model is the abstraction presented to users in the wizard interface 1002. To overcome the psychological problems described above, in one embodiment, wizard 1002 provides a model that allows users to describe what they want in terms of a few easily recognized and retrieved properties of convenient selections rather than constraint expressions. In this way, properties can be made easy to formulate in natural language requests (for example, adjectives that modify keyword markers) and be recognizable in advertisements (can also favor recommended restaurants ... "). In one embodiment, a data model is employed that enables assistant 1002 to determine the domain of interest (eg, restaurants vs. hotels) and a general approach to guidance that can be represented with domain-specific properties.
In one embodiment, the conceptual data model used by wizard 1002 includes a selection class. This is a representation of the space of things to choose from. For example, in the find-a-restaurant application, the selection class is the restaurant class. The selection class can be abstract and have subclasses, such as things to do while in a destination. In one modality, the conceptual data model considers that, in a situation of
187
ΙΜΡΙ
INSTTTUTO MUICANC) Say LA MIOflEDAt * INDUSTRIAL
<img file="MX338784B_D0190.tif" />
For specific troubleshooting, the user is selected from a single selection class. This consideration simplifies interaction and also allows attendee 1002 to declare their boundaries of competence (Se regarding restaurants, hotels and movies versus Se regarding city life) ·
Given a sort of selection, in one modality the data model presented to the user for the restricted selection task includes for example: items; item characteristics;
Selection criteria; and restrictions.
items are instances of the selection class.
Item characteristics are properties, attributes or calculated values that can be presented and / or associated with at least one item. For example, the name and phone number of a restaurant are item characteristics. Features can be intrinsic (the name or cuisine of a restaurant) or relational (for example, the distance from the person's current location of interest). They can be static (for example, restaurant name) or dynamic (rating). They can be composite values calculated from other data (for example, a value for money rating). Item characteristics are abstractions for the user made by the domain modeler; they do not need to correspond to underlying data from administrative and control services.
Selection criteria are characteristics of items that
188
ΙΜΡΙ
MEXICAN INSTITUTE Dt THE INDUSTRIAL PROPERTY
<img file="MX338784B_D0191.tif" />
They can be used to compare the value or relevance of items. That is, they are ways of saying which items are preferred. Selection criteria are modeled as characteristics of the items themselves, whether they are intrinsic or calculated properties. For example, proximity (defined as the distance from the location of interest) is a selection criterion. Location in space-time is a property, not a selection criteria, and is used in conjunction with the location of interest to calculate the distance from the location of interest.
Selection criteria may have an inherent order of preference. That is, the values of any particular criteria can be used to align items in a better first order. For example, the proximity criterion has an inherent preference that the closer the better. The location, on the other hand, has no inherent preference value. This restriction allows the system to make predefined considerations and if the user only mentions the criteria. For example, the user interface may offer sort by rating and consider that the highest rated is best.
One or more selection criteria are also item characteristics; are those related characteristics to choose between possible items. However, item characteristics are not necessarily related to a preference (for example, restaurant names and phone numbers are usually irrelevant to selecting among them).
189
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL EROEIEDAD
<img file="MX338784B_D0192.tif" />
In at least one modality, constrain — to the desired values of the selection criteria. Formally, constraints can be represented as a given membership (for example, a type of cuisine includes Italian), pattern matches (for example, restaurant review text includes romantic), mismatches of inaccurate match (for example, distance less than a few). how many kilometers), qualitative thresholds (for example, highly rated), or more complex functions (for example, a good value for money). To make things simple enough for normal humans, this data model reduces at least one or more constraints to symbolic values that can match as words. Time and distance can be excluded from this reduction.
In one mode, the operators and threshold values used to implement constraints are hidden from the user. For example, a restriction on the selection criteria called type of cuisine can be represented as a symbolic value such as Italian or Chinese. A qualification constraint is recommended (a binary selection). For time and distance, in one mode wizard 1002 uses property representations that handle a range of feeds and constraint values. For example, distance may be walking distance and the weather may be today at night; in one embodiment, wizard 1002 uses special processing to match that feed with more accurate data.
<img file="MX338784B_D0193.tif" />
<sup>190</sup> IMPI ικπτυτο MEXICAN
OF THE PROPERTY
INDUSTRIAL
In at least one modality, some restrictions may require restrictions. This means that the task simply cannot be completed without this data. For example, it is difficult to select a restaurant without some notion of desired location, even if the name is known.
To summarize, a domain is modeled as selection classes with item characteristics that are important to users. Some of the characteristics are used to select and order items offered to the user - these characteristics are called selection criteria. Constraints are symbolic limits on the selection criteria that narrow the set of items to those that match.
Often multiple criteria can compete and constraints can partially coincide. The data model reduces the selection problem from an optimization (finding the best solution) to a matching problem (finding items that meet a set of specified criteria and match a set of symbolic constraints). The algorithms for selecting criteria and constraints and determining an order are described in the next section.
Restricted Selection Methodology
In one modality, assistant 1002 performs restricted selection by taking an ordered list of criteria as feed, with implicit or explicit restrictions in at least one, and generating a set of candidate items with characteristics
191
IMPI
INSTITI ITO Μ 6X ICA NO Dt LA MONEDAD INDUSTRIAL
<img file="MX338784B_D0194.tif" />
outstanding. Computationally, the der-B'^ íite-ü-ián task can be characterized as a nested search: first, identify a class of selection, then identify the important selection criteria, then specify constraints (the boundaries of acceptable solutions), and Search for instances to make a better fit to find acceptable items.
Now referring to Figure 45, an example of an abstract 4500 model is shown for a narrow selection task in a nested search. In the example, the wizard
1002 it identifies in 4505 a request for selection among all local search types 4501. The identified class is restaurant. Within the sets of all restaurants 4502, the assistant 1002 selects in 4506 the criteria. In the example, the criterion is identified as the distance. Within the set of restaurants in PA 4503, assistant 1002 specifies in 4507 the search restrictions. In the example, the constraint identified is Italian cuisine.) Within the set of Italian restaurants in PA 4504, the assistant 4508 selects items for presentation to the user.
In one embodiment, this embedded search is what the wizard 1002 does once it has the relevant feed data, rather than the flow to produce the data and presents results. In one embodiment, this flow of control is regulated by a dialogue between the wizard 1002 and the user that operates by other procedures, such as task flow models and
192
IMPI
MEXICAN INSTITUTE Say THE INDUSTRIAL PROPERTY
<img file="MX338784B_D0195.tif" />
dialogues. Restricted selection offers an it jfci-'uii.tuga for building task flow models and dialogs at this level of abstraction (that is, suitable for restricted selection of tasks regardless of domain).
Now referring to Figure 46, an example of a 4600 dialog is shown to help guide the user through a search process so that relevant power data can be retrieved.
In the 4600 dialog example, the first step is for the user to set the type of things they are looking for, which is the selection class. For example, the user will be able to do this when saying have dinner in Palo Alto. This allows the wizard 1002-infer at 4601 the task and domain.
Once wizard 1002 has understood the task and domain link (selection class = restaurants), the next step is to understand what selection criteria are important to this user, for example when requesting 4603 criteria and / or restrictions. In the example above, in Palo
High indicates a location of interest. In the restaurant context, the system can interpret a location as a proximity constraint (technically, a constraint on the proximity criterion). Assistant 1002 explains that it requires, receives power. If there is enough information to restrict the set of selections to a reasonable size, then wizard 1002 paraphrases the feed and presents
193
<img file="MX338784B_D0196.tif" />
IMPI
KSTmiTO MEXICANO Dt LA PRWISOAD INDUSTRIAL in 4 605 one or more restaurants that meet the proximity restriction, classified in some useful order. The user can then select 4607 from this list, or refine 4606 criteria and restrictions. Wizard 1002 reasons about already established constraints, and uses domain-specific knowledge to suggest other criteria that may help, requesting restrictions on these criteria equally. For example, assistant 1002 may reason that when recommending restaurants within walking distance of a hotel, the useful criteria for requesting will be table availability and types of cuisine.
The restricted selection task is completed when the user chooses in 4607 an instance of the selection class. In one mode, additional 4602 trace tasks are enabled by wizard 1002. In this way, wizard 1002 can offer services that indicate selection while providing some other value. Examples in 4608 reserve a restaurant, set a reminder on a calendar, and / or share the selection with others when sending an invitation. For example, booking a restaurant surely indicates that you were selected; Other options may be to put the restaurant on a calendar or send an invitation with directions on how to reach friends.
Now referring to Figure 47, a flowchart illustrating a narrow selection method is shown.
194
IMPI
MFIXICAN INSTITUTE OF INDUSTRIAL RORBITY
<img file="MX338784B_D0197.tif" />
according to a modality. In one modality,<sup>1</sup> yi ciyii-iLente 1 (JÜ2 operates in an opportunistic and mixed initiative manner, allowing the user to jump into the inner loop, for example, by setting task, domain, judgment and restrictions, one or more at a time in the feed.
The method starts at 4701. Power is received at
4702 user, according to any of the modes described here. If, based on the power, the unknown task, the wizard 1002 requests in 4705 to clarify the power of the user.
At step 4717, wizard 1002 determines whether the user provides additional power. If so, wizard 1002 returns to step 4702. Otherwise, method 4799 ends.
<td>Yes,</td><td>in</td><td>step 4703</td><td>I know</td><td>known</td><td>the task, the assistant</td>
<td>1002 determines</td><td>in</td><td colspan="2">4704 if homework</td><td>is a</td><td>restricted selection.</td>
<td>If not,</td><td>the</td><td>assistant 1002</td><td colspan="2">proceeds in</td><td>4706 to task flow</td>
<td>specified.</td><td></td><td></td><td></td><td></td><td></td>
If in step 4704, the task is selection restricted, wizard 1002 determines in 4707 whether the selection class can be determined. If not, wizard 1002 offers a selection of known selection classes at 4708 and returns to step 4717.
If, in step 4707, the selection class can be determined, wizard 1002 determines in 4709 whether all
195
IMPI
MWICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0198.tif" />
Required restrictions can be determined D® 1ΓΟ 'stü so, wizard 1002 requests at 4710 the required information, and returns to step 4717.
If, at step 4709, all required constants can be determined, wizard 1002 determines at 4711 whether any result items can be found, given the constraints. If there are no items that meet the constraints, the wizard 1002 offers in 4712 ways to relax the constraints. For example, wizard 1002 can relax constraints from lowest to highest precedence, using a sort / filter algorithm. In one modality, if there are items that meet some of the restrictions, then wizard 1002 can paraphrase the situation (sending output for example, I couldn't find recommended Greek restaurants that stock up on Sundays in San Carlos. However, I found 3 Greek restaurants and 7 recommended restaurants in San Carlos). In one modality, if there are no items matching any restrictions, then wizard 1002 can paraphrase this situation and warn different restrictions (sending out, for example, Sorry, I couldn't find any restaurant in Anytown, Texas. You can select a different location). Wizard 1002 returns to step 4717.
If, at step 4711, the result items cannot be found, wizard 1002 provides a list of items at 4713.
In one embodiment, wizard 1002 paraphrases the criteria and
196
ΙΜΡΙ
TπΟ MVtlCANO OF INDUSTRIAL RRORIRDAD
<img file="MX338784B_D0199.tif" />
currently specified restrictions (shipping from saiiaa, püF example, Here are some recommended Italian restaurants in San José. (Recommended = yes, Type of cuisine = Italian, proximity = <in San Jose>)). In one embodiment, wizard 1002 presents a list of classified, paged items that meet known restrictions. If an item only displays some of the constraints, that condition can be displayed as part of the item display. In one embodiment, wizard 1002 offers the user ways to select an item, for example when starting another task on that item such as booking, remembering, scheduling, or sharing. In one embodiment, on any given item, wizard 1002 displays item characteristics that are outstanding for selecting instances of the selection class. In one embodiment, wizard 1002 shows how the item meets a constraint; for example, the Zagat rating of 5 meets the restriction Recommended = yes, and at 1.6 km approx. (1 mile) away complies with the restriction within a walking distance of one direction. In one embodiment, wizard 1002 allows the user to delve more detail into an item, resulting in the display of more item features.
Wizard 1002 determines at 4714 whether the user has selected an item. If the user selects an item, the task is complete. Any follow-up task is performed at 4715, if there is one, and the method ends at 4799.
If, in step 4714, the user does not choose an item, the
197
IMPI mkicang institute Df LA FF.OFlkUAÍ
ÍNWJSTIIAI
<img file="MX338784B_D0200.tif" />
Wizard 1002 provides the user with 4716 forms for-a-r-Qi.a-agi.-uiiit other criteria and restrictions and returns to step 4717. For example, given the currently specified criteria and restrictions, wizard 1002 may offer criteria that are very It will probably restrict the set selection to a desired size. If the user chooses a constraint value, that constraint value is added to the previously determined constraints when steps 4703 to 4713 are repeated.
Since one or more criteria may have an inherent preference value, selecting the criteria can add information to the request. For example, allowing the user to indicate that positive reviews are valuable allows assistant 1002 to sort this by criteria. This information can be taken into account when repeating steps 4703 to 4713.
In one embodiment, wizard 1002 allows the user to increase the importance of a criterion that is already specified, so that it is superior in the preceding order. For example, if the user requests highly recommended, inexpensive fast food restaurants within a block within one block of their location, wizard 1002 may request that the user choose when these criteria are most important. Such information may be taken into account when steps 4703 to 4713 are repeated.
In one mode, the user can provide additional power at any point while the method
198
IMPI
MEXICAN MSTITUTO Di LA MOPIIUAD INDUSTRIAL
<img file="MX338784B_D0201.tif" />
from Figure 47 is executed. In one embodiment, wizard 1002 periodically or continuously verifies this power, and in response loops back to step 4703 to process it.
In one modality, when an item or item list is sent out, the assistant 1002 indicates in the presence of items, the characteristics that were used to select and order them. For example, if the user requests nearby Italian restaurants, these item characteristics by distance and type of cuisine may be displayed in the item display. This may include highlighting correspondences or coincidences, as well as selection criteria that were involved in the presentation of an item.
Exemplary Domains
Table 1 provides an example of restricted selection domains that can be managed by wizard 1002 according to various modalities.
199
IMPí
<img file="MX338784B_D0202.tif" />
M1XICANO KSTITUTO OF INDUSTRIAL PROPERTY
<td>Select a</td><td colspan="4">Based on these criteria</td>
<td></td><td>Located- tion</td><td>Price</td><td>Available bility</td><td>Kind</td>
<td>Restau- rante</td><td>Proximi- give</td><td>Accessories ble</td><td>Tables available</td><td>Kind of kitchen</td>
<td>hotel</td><td>Proximi- give</td><td>Interval prices</td><td>Inhabits- tions available bles</td><td>Motel, hotel, B&B (accommodation that offers bed and breakfast</td>
<td>Movie s</td><td>Proximity of the living room</td><td></td><td>To show Schedule</td><td>Gender</td>
<td>Deal local</td><td>Proxi- mity</td><td></td><td></td><td>Category of business</td>
<td>Event local</td><td>Proximity of site of meeting</td><td></td><td>By date</td><td></td>
200
<img file="MX338784B_D0203.tif" />
ΙΜΡΙ
IN! TmnOM «ICANO
DI LA FIOF1WAD inomtwal
<td>With-</td><td>Proxi-</td><td></td><td>Program</td><td>Gender</td>
<td>true</td><td>half of</td><td></td><td>per tour</td><td>music</td>
<td></td><td>site of</td><td></td><td></td><td></td>
<td></td><td>meeting</td><td></td><td></td><td></td>
<td>CD,</td><td></td><td>Interval-</td><td>online,</td><td>to download,</td>
<td>books</td><td></td><td>what of</td><td>in shop,</td><td>physical,</td>
<td>DVD for</td><td></td><td>prices</td><td>etc.,</td><td></td>
<td>to buy</td><td></td><td></td><td></td><td></td>
Second part of Table 1
<td>Select ciona a</td><td>Based on</td><td colspan="4">n these criteria</td>
<td></td><td>Quality</td><td>Name</td><td>Servi-</td><td>Search</td><td>Search</td>
<td></td><td></td><td></td><td>cios</td><td>Special</td><td>in</td>
<td></td><td></td><td></td><td></td><td></td><td>general</td>
<td>Restau</td><td>Rate-</td><td>Name</td><td>Shipping</td><td>items of</td><td>Words</td>
<td>-rant</td><td>tion by</td><td>of the</td><td></td><td>menu</td><td>key</td>
<td></td><td>guide,</td><td>restau-</td><td></td><td></td><td></td>
<td></td><td>review</td><td>rante</td><td></td><td></td><td></td>
<td>hotel</td><td>Rate-</td><td>Name</td><td>Ameni-</td><td></td><td>Words</td>
<td></td><td>tion</td><td>of</td><td>dades</td><td></td><td></td>
201
<img file="MX338784B_D0204.tif" />
IMPI
Mexican Institute of Industrial Property
<td></td><td>by guide, review</td><td>hotel</td><td></td><td></td><td>key</td>
<td>Movie</td><td>Rate-</td><td>Title</td><td></td><td>Actors,</td><td></td>
<td>the</td><td>tion by</td><td>of</td><td></td><td>etc.</td><td></td>
<td></td><td>reviews</td><td>movie</td><td></td><td></td><td></td>
<td>I negotiated</td><td>Rate-</td><td>Name</td><td></td><td></td><td>Words</td>
<td>or</td><td>tion by</td><td>of</td><td></td><td></td><td>key</td>
<td>local</td><td>Review</td><td>deal</td><td></td><td></td><td></td>
<td>Event</td><td></td><td>Title</td><td></td><td></td><td>Words</td>
<td>local</td><td></td><td>of</td><td></td><td></td><td>key</td>
<td></td><td></td><td>event</td><td></td><td></td><td></td>
<td>Count</td><td></td><td>Name</td><td></td><td>Members</td><td>Words</td>
<td>rto</td><td></td><td>band</td><td></td><td>of the</td><td>key</td>
<td></td><td></td><td></td><td></td><td>band</td><td></td>
<td>CD,</td><td>populari-</td><td>Album or</td><td></td><td>Artist,</td><td>Words</td>
<td>books</td><td>give</td><td>Name</td><td></td><td>Title,</td><td>key</td>
<td>DVD</td><td></td><td>of</td><td></td><td>etc.,</td><td></td>
<td>for</td><td></td><td>song</td><td></td><td></td><td></td>
<td>purchase</td><td></td><td></td><td></td><td></td><td></td>
<td>r</td><td></td><td></td><td></td><td></td><td></td>
202
IMPI
INSTITUTO NRXICAN-.) OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0205.tif" />
Table 1 ...... .
Filtering and Classification Results
In one modality, when items that meet currently specified criteria and restrictions are presented, a classification / filter methodology can be used. In one modality, selection constraints can serve as both filter parameters and classification to the underlying services. In this way, you can use any selection criteria to determine which items are in the list, and to calculate the order to be paginated or adjusted and displayed. The ranking order for this task is related to the search relevance range. For example, proximity is a criterion with symbolic restriction values such as within a driving distance and a general notion of distance ranking. The driving distance constraint can be used to select a candidate item group. Within that group, closer items will have a higher ranking in the list.
In one modality, associated selection and filtering and selection constraints are at discrete levels, which are functions of both the underlying data and the user feed. For example, proximity is grouped into levels such as walking distance, taxi distance, car distance. When sorting, one or more items within walking distance, are treated as if they were the same distance. User power can come into play in the way that
203
IMPI ustitutomkicano
OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0206.tif" />
you specify a restriction. If the user piupuitiunu '' a
Palo Alto, for example then one or more items within the city limits of Palo Alto are perfect matches and are equivalent. If the user provides near the University Avenue train station then the match will depend on a distance from that direction, with the degree of match depending on the kind of selection (for example, close for restaurants is different than close for hotels ). Even within a constraint that can be specified with a continuous value, you can apply discretization. This can be important for sorting, so that multiple criteria can be involved in determining the best first order.
In one modality, the item list - those items that were considered matching or good enough - may be shorter or longer than the number of items illustrated on an output page. In general, items on the first page are given the most attention, but conceptually there is a longer list, and pagination is simply a function of the output medium form factor. This means, for example, that if the user is offered a way to classify or display the items by some criteria, then it is the complete set of items (more than one page in value) that is classified or displayed.
In one modality, there is an ordering by precedence between selection criteria. That is, some criteria may
204
IMPI
INSTITUTO MMICANO DI LA FWOneDAf) INDUSTRIAL
<img file="MX338784B_D0207.tif" />
import more than others in filtering and classification'Í In a modality, those criteria selected by the user are given higher precedence than others, and there is a predefined ordering on one or more criteria. This allows a general lexicographical classification. The consideration is that there is a significant precedence a priori. For example, unless the user states otherwise, it may be more important that a restaurant is closer, that it is inexpensive. In one embodiment, the a priori order of precedence is domain specific. The model allows user-specific preferences to exceed the domain's predefined values, if desired.
Since constraint values can represent various internal data types, there are different ways for constraints to match, and they can be constraint-specific. For example, in a modality:
• Binary constraints correspond to one or more or none. For example, whether a restaurant is fast may or may not be true.
• Membership restrictions setting matches one or more or none based on a property value. For example, cuisine type = Greek means that the set of cuisine types for a restaurant includes Greek.
• Numbering constraints match a threshold. For example, a rating criterion can have values of
205
IMPI ίΝτητυτο mwicano oe la enoenDAu INDUSTRIAL
<img file="MX338784B_D0208.tif" />
constraint rated, highly rated, U niujiíiitei · rating. Restricting to high rating will also coincide with the highest rating.
• Numerical restrictions coincide with a threshold that may be criterion specific. For example, open late can be a criterion, and the user can ask for open sites after 10:00 pm. This type of constraint may be slightly out of scope for the constrained selection task, as it is not a symbolic constraint value. However, in one modality, wizard 1002 recognizes some cases of numerical restrictions like this, and maps them to threshold values with symbolic restrictions (for example, restaurants in Palo Alto open now -> here are 2 restaurants in Palo Alto that are open See you later).
• Location and time are handled in a special way. A proximity constraint can be a specified location of interest at a certain level of granularity, and that determines the match. If the user specifies a city, then the city-level mapping is appropriate; a zip code can allow a radius. Wizard 1002 can also comprise locations that are close to other locations of interest, also based on special processing. Time is relevant as a constraint value for criteria that have a threshold value based on a service call, such as table availability or flights within a time interval
206
ΙΜΡΙ
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0209.tif" />
determined. - In one modality, constraints can be modeled such that there is a single threshold value for selection and a small set of discrete values for classification. For example, the scope criterion can be modeled as a roughly binary constraint, where the scope restaurants are under some threshold price range. When the data justify multiple discrete levels for selection, constraints can be modeled using a correspondence gradient. In one embodiment, two levels of match (such as strong and weak match) can be provided; however, a person skilled in the art will recognize that in other modalities, any number of levels of matching may be provided. For example, proximity can match an inaccurate match boundary, such that things that are close to the location of interest may weakly match. The operational consequence of a strong or weak match is in the sort / filter algorithm as described below.
For at least one criterion, a matching approach and predefined thresholds can be established, if relevant. The user may be able to say only the name of the constraint, a symbolic constraint value, or a precise constraint expression if it is handled in a special way (such as time and location).
207
<img file="MX338784B_D0210.tif" />
<img file="MX338784B_D0211.tif" />
An ideal situation for selection i ^ stiiuyidu<sup>11</sup> ucuirti when the user sets restrictions that result in a shortlist of candidates, one or more of whom meets the restrictions. The user then selects among winners based on item characteristics. In many cases, however, the problem is over- or under-restricted. When overly restricted, there are few or no restrictions. When it is loosely restricted, there are too many candidates, such that browsing the list is not quick. In one embodiment, the general constrained selection model of the present invention is capable of handling multiple constraints with robust matching and usually producing something to select from. Then the user can choose to refine their criteria and restrictions or just complete the task with a good enough solution.
Method
In one modality, the following method is used to filter and classify results:
one. Given an ordered list of selection criteria chosen by the user, determine to build the constraints on at least one.
to. If the user specified a restriction value, use it. For example, if the user says Greek food the restriction is type of cuisine = Greek. If the user says San
Francisco restriction is in the City of San Francisco. If the user says south of the market then the restriction is on the
208
IMPI
INSTITUTO MBKICANO Dt LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0212.tif" />
proximity of SoMa. '
b. Otherwise it uses a domain and criteria specific predefined value. For example, if the user says a table somewhere thai is indicating that the availability criterion is relevant, but did not specify a restriction value. The constraint values defined for availability can be a certain time range of the date such as today at night and a predefined group size of 2.
2. Select a minimum of N results for specified constraints.
to. Trying to get N results with strong coincidence.
b. If that fails, try to relax the constraints, in reverse order of precedence. That is, correspondence at a strong level by one or more of the criteria except the last one, which may coincide at a weak level. If there is no weak match for that constraint, then test weak matches to the line from lowest to highest precedence.
3. After obtaining a minimum selection set, it classifies one or more criteria (which may include user-specified criteria as well as other criteria) in lexicographic form in order of precedence.
to. Consider the set of user-specified criteria as the highest precedence, then one or more of the remaining criteria in their a priori precedence. For example, if the
209
<img file="MX338784B_D0213.tif" />
IMPI
MEXICAN INSTITUTE
OE LA MONEDAD
INDUSTRIAL a priori precedence is (availability, proximity, rating), and the user gives restrictions on proximity and type of cuisine, so the classification precedence is (type of cuisine, proximity, availability, rating).
b. Classify in criteria using discrete levels of correspondence (strong, weak, none), using the same approach as in relaxation constraints, this time applying the entire list of criteria.
i. if a set of selections is obtained without relaxing constraints, then one or more of the set of selections may be tied in the ranking because one or more coincide at strong levels. Then, the following criteria in the precedence list can be entered to classify them. For example, if the user says type of cuisine = Italian, proximity = in San Francisco, and the ranking precedence is (type of cuisine, proximity, availability, rating), then one or more of the sites on the list have equal values. match for type of cuisine and proximity. In such a way that the list is classified in availability (places with available tables rise like bubbles to the top). Among the available sites, those of the highest at the top.
ii. If the selection set is obtained by relaxing constraints, then one or more of the items
210
Γ Α Λ ”Τ> χ
<img file="MX338784B_D0214.tif" />
totally matching are in the part <sup>c, ii</sup>p<sup>g</sup>‘<sup>T</sup>ÍLl Jl the iota after the partial coincidence items. Within the match or match group, they are classified by the remaining criteria, and the same for the partial match group. For example, if there are only two Italian restaurants in San Francisco, then the one available will be displayed first, then the one not available. Later, the rest of the restaurants in San
Francisco will be shown classified by availability and rating.
Order of Precedence
The techniques described here allow Wizard 1002 to be extremely robust against partially specified constraints and incomplete data. In one modality, the wizard
1002 uses these techniques to generate a list of user items in the best first order, that is, according to relevance.
In one modality, this relevance classification is based on an a priori order of precedence. These are, from the things that matter regarding a domain, a set of criteria is chosen and ranked in order of importance. One or more things being equal, higher criteria in the order of precedence may be more relevant to a restricted selection among items than those of lower order. Wizard 1002 can operate on any number of criteria. Furthermore, the criteria can be modified over time without breaking existing behaviors.
211
IMPI
INSTITUTO mkicano DR LA NOPBM) INDUSTRIAL
<img file="MX338784B_D0215.tif" />
In one modality, the order of pryeedeiHJld <sup>1</sup> yTIL'i'tí criteria can be adjusted with domain-specific parameters, as the way the criteria interact may depend on the kind of selection. For example, when selecting between hotels, availability and price may be dominant restrictions, while for restaurants, type of cuisine and proximity may be more important.
In one mode, the user can exceed or override predefined criteria in the dialog. This allows the system to guide the user when searches are excessively restricted, by using the ordering to determine what restrictions should be relaxed. For example, if the user gave restrictions regarding the type of cuisine, proximity, recommendation and food items and there were no items of total correspondence, the user can say that the food item is more important than the level of recommendation and change the mixture. such that the desired food item matches will be ranked to the top.
In one mode, when determining the order of precedence, user-specified constraints take precedence over others. For example, in one modality, proximity is a required constraint and is thus always specified, and they also take precedence over other unselected constraints. Therefore, it does not have to be the highest precedence constraint in order to be substantially
212
IPI
MEXICAN INSTITUTE OF IX INDUSTRIAL PROPERTY
<img file="MX338784B_D0216.tif" />
dominant. Also, many criteria can n <r i-oinr.ididir in one or more unless a restriction is given by the user, and thus the precedence within the criteria selected by user. For example, when the user specifies a kitchen it is important to them, and is otherwise not relevant to classify items.
For example, the following is a candidate precedence ranking paradigm for the restaurant domain:
one. type of cuisine * (not classified unless a restriction value is given)
2. availability * (is classified using a predefined restriction value, for example time)
3. recommended
Four. proximity * (a constraint value is always given)
5. within reach
6. can do shipping
7. food item (not classified unless a constraint value, for example, a keyword, is given)
8. keywords (not classified unless a constraint value, for example, a keyword, is de)
9. restaurant name
The following is an example of a design rationale for the above classification paradigm:
If a user specifies a type of cuisine, he wants
213
IMPI
INSTITUTO MEXICANO Di LA ΡΚΟΡ, ΕΠΑΟ INDUSTRIAL
<img file="MX338784B_D0217.tif" />
Please remain.
• One or more things being equal, classified by level of qualification (it is the highest precedence among criteria that can be used to classify without restriction).
• In at least one modality, proximity can be more important than most things. However, since it coincides at discrete levels (in a city, within a walking radius, and the like), and is always specified, then most of the time most of the matching items may be moored nearby. Availability (as determined by a search on a website such as open-table.com, for example) is a valuable ranking criterion, and can be based on a predefined value to rank when not specified. If the user indicates a time to book, then only available sites can be listed and ranking can be based on recommendation.
• If the user says they want highly recommended sites, then they can rank on proximity and availability, and these criteria can be relaxed before recommendation. The consideration that someone is looking for a nice place, can drive a little more or drive a little more and is more important than availability of a predefined table. If a specific time for availability is set, and the user requests recommended sites, then sites that are both recommended and available can enter first, and the
214
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0218.tif" />
recommendation can relax to a weak match diiLau that ---- availability fails to match in qno or more.
• The remaining restrictions except for name and one or more based on incomplete data or correspondence. So they are weak default classification heuristics and when one or more-or-none correspondence is specified.
• Name can be used as a constraint to handle the case where someone mentions the restaurant by name, for example, one or more Ho-bee's restaurants near Palo Alto. In this case, one or more items can match the name, and can be classified by proximity (the other restriction specified in this example).
Domain Modeling: Mapping Selection Criteria to
Underlying Data
It may be desirable to distinguish between the data that is available to calculate by wizard 1002 and the data used to make selections. In one embodiment, wizard 1002 uses a data model that reduces complexity for the user by folding one or more data types used to distinguish between items in a simple selection criteria model. Internally, this data can take various forms. Instances of the selection class can have intrinsic properties and attributes (such as a restaurant's kitchen type), can be compared over dimensions (such as the distance from some location), and can be discovered by
215
IMPI
MEXICAN INSTITUTE Di LA TROHÍDAD INDUSTRIAL
<img file="MX338784B_D0219.tif" />
any queries (such as if they correspond to a '”T)?<sup>l</sup>oh ílé 'LtíXLU ir are available at a certain time). It can also be calculated from other data that is exposed to the user as selection criteria (for example, weighted combinations of ratings from multiple sources). This data is one or more relevant to the task, but the distinctions between these three types of data are not relevant to the user. Since the user thinks in terms of characteristics of the desired selection rather than properties and dimensions, wizard 1002 operates these various criteria on item characteristics. Wizard 1002 provides a data model of *
Domain in front of the user and maps it in data found in network services.
One type of mapping is an isomorphism from data underlying criteria that confront the user. For example, the availability of tables for reservations as seen by the user may be exactly what an online reservation site such as opentable.com offers, using the same granularity for time and group size.
Another type of mapping is a normalization of data from one or more services to a common value set, possibly with a unification of equivalent values. For example, types of cuisine from one or more restaurants can be represented as a single ontology in assistant 1002, and mapped to various vocabularies used in different services. That
216
IMPI
MEXICAN INSTITUTE I> E INDUSTRIAL PROPERTY
<img file="MX338784B_D0220.tif" />
Ontology can be hierarchical, and have nodes dg ~ liujn quo — d- ·· τ-i πρπ at specific values of at least one service. For example, one service may have a cuisine type value for Chinese, another for Szechuan, and a third for Asian. The ontology employed by wizard 1002 will cause references to Chinese or szechuan food to correspond semantically to one or more of these nodes, with confidence levels that reflect the degree of coincidence.
Normalization may also be involved when differences in precision are resolved. For example, the location of a restaurant may be at street level in one service but only at the city level in another. In one embodiment, wizard 1002 uses a structural representation of locations and times that can be mapped to different surface data values.
In one embodiment, wizard 1002 uses a special type of mapping for open-end qualifiers (eg, romantic, quiet, or calm) that can be mapped for full-text search matches, labels, or other 'open' texture features. The name of the select constraint in this case would be something as described as.
In at least one modality, restrictions to operational preference arrangements can be mapped. That is, given the name of a selection criterion and its value of
217
<img file="MX338784B_D0221.tif" />
constraint, wizard 1002 is capable of Ho — τ <~> «criteria as an ordering of possible items. There are several technical aspects to focus on this mapping. For example:
• Preference orders may conflict. The order given by one constraint may be inconsistent or even inversely correlated with the order given by another. For example, price and quality tend to be in opposition. In one embodiment, wizard 1002 interprets user-selected constraints in a weighted or otherwise combined order that reflects the user's wishes but is true for the data. For example, the user can order cheap, fast-food French restaurants within walking distance, with high ratings. In many places, you may not have said such a restaurant. However, in one embodiment, wizard 1002 can display a list of items that attempts to optimize at least one constraint and explain why at least one is cited. For example, item one may be high-ranking French cuisine and other inexpensive fast food within walking distance.
• Data can be used either as hard or soft constraints. For example, the price range of a restaurant may be important in selecting one, but it may be difficult to set a threshold value for entry price. Even seemingly harsh restrictions like type of cuisine can be • in practice, soft restrictions due to correspondence
218
IMPI
ΤΟυΤΟ MKICANO t> e la ruoneoAD INOIJCT * 1AL
<img file="MX338784B_D0222.tif" />
partial. Since, in one modality, the assistant ΓϋΟζΓ '^ ΊΙΰ UlTll' ^ a · the data modeling strategy that seeks to simplify one or more criteria in symbolic values (such as economic or close), these constraints can be mapped into a function that obtains the criteria and order, without being strict about matching specific threshold values. For symbolic criteria with clear objective real values, assistant 1002 can weight the objective criteria higher than other criteria, and make clear the explanation that he knows that some of the items do not strictly correspond to the requested criteria.
• items may match some but not one or more constraints, and better fit items may be displayed.
• In general, wizard 1002 determines which item characteristics are outstanding for a domain, and which can serve as selection criteria, and for at least one criterion, possible constraint values. This information can be provided, for example, by operational data and API calls.
Notice and Paraphrased Text
As described above, in one embodiment of the assistant 1002 provides feedback to show that you understand the user's intention and work toward the user's goal by producing paraphrases of your current understanding. In the conversation dialogue model of the present invention, the
218
ΙΜΡΙ
INSTTTUTO MIXICANO Dt LA INDUSTRIAL RMORIEDAD
<img file="MX338784B_D0223.tif" />
partial. Since, in one modality, the agigi-oni-a-tnn? gn<sub>P</sub> uses the data modeling strategy that seeks to simplify one or more criteria in symbolic values (such as economic or close), these restrictions can be mapped into a function that obtains the criteria and order, without being strict about matching specific threshold values. For symbolic criteria with clear objective real values, assistant 1002 can weight the objective criteria higher than other criteria, and make clear the explanation that he knows that some of the items do not strictly correspond to the requested criteria.
• items may match some but not one or more constraints, and better fit items may be displayed.
• In general, wizard 1002 determines which item characteristics are outstanding for a domain, and which can serve as selection criteria, and for at least one criterion, possible constraint values. This information can be provided, for example, by operational data and API calls.
Notice and Paraphrased Text
As described above, in one embodiment of the assistant 1002 provides feedback to show that you understand the user's intention and work toward the user's goal by producing paraphrases of your current understanding. In the conversation dialogue model of the present invention, the
219
<img file="MX338784B_D0224.tif" />
ΙΜΡΙ
MST1 MEXICAN TUTO
OF IA PROPERTY
INDUSTRIAL paraphrase is what the wizard 1002 outputs after 2TS the user feed, as a preface (for example, paraphrase 4003 in Figure 40) or summary of the results to follow (for example, list 3502 in Figure 35) . The notice is a suggestion to the user about what else they can do to refine their request to explore the selection space on some dimensions.
In one embodiment, the purposes of notice and paraphrase text include, for example:
•. show that wizard 1002 understands the concepts in the user's feed, not just the text;
• to indicate the boundaries of the understanding of the assistant 1002;
• to guide the user in providing the text that is required for the estimated task;
• to assist the user in exploring the space of possibilities in restricted selection;
• to explain the current results that are obtained from services, in terms of the established criteria of the user and considerations of the assistant 1002 (for example, to explain the results of requests for restriction in excess and in shortage).
For example, the following paraphrase and notice illustrates several of these goals:
User Food: Indonesian Food at Menlo Park
Interpretation of system:
220
<img file="MX338784B_D0225.tif" />
ΙΜΡΙ
MEXICAN INSTITUTE
OF THE PROPERTY
INDUSTRIAL
Task = restricted selection ...........
Selection Class = restaurant
Restrictions:
Location = Menlo Park, CA
Type of cuisine = Indonesian (known in ontology)
Service Results: no strong matches
Paraphrased: Sorry, I couldn't find any Indonesian restaurants near Menlo Park.
Notice: You could try other types of cuisine or locations.
Notice under hypertext links:
indonesia: You can try other food categories such as china, or favorite food items such as meat.
Menlo Park: Provides in a location such as city, neighborhood, street address or near followed by a reference.
Cuisine Type: Provides a food category such as china or pizza.
Locations: Provides a location: a city, zip code, or nearby followed by the site-name.
In one embodiment, wizard 1002 responds to user power relatively quickly with paraphrasing. The paraphrase is then updated after the results are known. For example, an initial answer may be Searching for Indonesian restaurants near Menlo Park .... Once results are obtained, wizard 1002 will update the text to read, Sorry I can't find
221
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL BROMITY
<img file="MX338784B_D0226.tif" />
No Indonesian restaurants near Menlo Park. Pódl'lcf you try other types of cuisine or locations. Note that certain items are highlighted (indicated here by underlining), showing that the items represent restrictions that can be relaxed or changed.
In one modality, special formatting / highlighting is used for keywords in the paraphrase. This may assist in facilitating user training for interaction with the intelligent automated assistant 1002, by indicating to the user which words are most important and most likely to be recognized by the assistant 1002. The user may then most likely use these words in the future.
In one embodiment, paraphrases and prompts to generate use any relevant context data. For example, any of the following data items can be used, alone or in combination:
• Syntactic parsing - a tree of ontology nodes linked to their corresponding power signals, with annotations and exceptions. For each node in the parsing, this can include the node metadata and / or any signals in the feed that provide evidence for the node value.
• The task, if known. • The selection class.
The location constraint, independent of
222
<img file="MX338784B_D0227.tif" />
IMPI
MEXICAN INSTITUTE
Dt THE PROPERTY
INDUSTRIAL selection. - • What required parameters are known for the given selection class (for example, location is a required restriction in restaurants).
• The name of an entity mentioned in parsing that is an instance of the selection class, if there is one (for example, a specific restaurant or movie name).
• Is this a follow-up refinement or the start of a conversation? (Restart starts a new conversation).
• What constraints in parsing are tied to values in the feed that change their values? In other words, what restrictions were only changed by the most recent feeding?
• Is the class of selection inferred or directly established?
• Classified by quality, relevance or proximity?
• For each specified constraint, how closely did it coincide?
• What refinement entered as text or click?
In one modality, the paraphrase algorithm takes into account the query, domain model 1056, and the service results. The 1056 domain model contains classes and features including metadata that is used to decide how to generate text. Examples of these metadata for paraphrase generation include:
• IsConstraint = {true | false)
223
<img file="MX338784B_D0228.tif" />
IMPI
MStlCANO INSTITUTE
O »THE PROPERTY
INDUSTRIAL • IsMultiValued = {true | false} • ConstraintType = {EntityName, Location, Time, Category
Constraint, AvailabilityConstraint, BinaryConstraint,
SearchQualifier, Guess- edQualifier} • DisplayName = string • Display TemplateSingular = string • Display TemplatePlural = string • Grammatical- Role = {AdjectiveBeforeNoun, Noun, ThatClauseModifer}
For example, a parsing can contain these elements:
Class: Restaurant
IsConstraint = false
Display TemplateSingular = restaurant
Display TemplatePlural = restaurants
GrammaticalRole = Noun
Characteristic: Restaurant Name (RestaurantName) (example:
II Fornaio)
IsConstraint = true
IsMultiValued = false
ConstraintType = EntityName Display TemplateSingular = named $ 1
Display TemplatePlural = named $ 1
GrammaticalRole = Noun
Feature: Restaurant Cuisine Type (RestaurantCuisine)
224
IMPI
MEXICAN INSTITUTE OF PROPERTY
INDUSTRIAL
<img file="MX338784B_D0229.tif" />
(example: Chinese)
IsConstraint = true
IsMultiValued = false
ConstraintType = CategoryConstraint GrammaticalRole = Adj ectiveBeforeNoun
Characteristic: Restaurant Subtype (RestaurantSubtype) (example: cafe)
IsConstraint = true
IsMultiValued = false
ConstraintType = CategoryConstraint Display TemplateSingular = $ l
Display TemplatePlural = $ ls
GrammaticalRole = Noun
Feature: RestaurantQualifiers (example: romantic) IsConstraint = true
IsMultiValued = true
ConstraintType = SearchQualifier
Display TemplateSingular = is described as $ 1
Display TemplatePlural = are described as $ 1
Display TemplateCompact = matching $ 1
GrammaticalRole = Noun
Characteristic: Type of Food (FoodType) (example: burritos)
IsConstraint = true
IsMultiValued = false
ConstraintType = SearchQualifier
225
<img file="MX338784B_D0230.tif" />
INSTITUTO MEXICANO Di LA FAOPIEDAI »INDUSTRIAL
Display TemplateSingular = serves $ 1
Display TemplatePlural = serve $ 1
Display TemplateCompact = serving $ 1
GrammaticalRole = ThatClauseModifer
Feature: Is Recommended (IsRecommended) (example:
true)
IsConstraint = true
IsMultiValued = false
ConstraintType = BinaryConstraint Display TemplateSingular = recommended
Display TemplatePlural = recommended
GrammaticalRole = Adj ectiveBeforeNoun
Feature: RestaurantGuessedQualifiers (example: spectacular)
IsConstraint = true
IsMultiValued = false
ConstraintType = GuessedQualifier
Display TemplateSingular = matches $ 1 in reviews
Display TemplatePlural = match $ 1 in reviews
Display TemplateCompact = matching $ 1
GrammaticalRole = ThatClauseModifer
In one embodiment, the wizard 1002 is capable of handling power without matching. To handle this feed, the 1056 domain model can provide GuessedQualifier type nodes for each selection class, and rules that
226
MBKICANO INSTITUTE • AND PROPERTY
INDUSTRIAL
<img file="MX338784B_D0231.tif" />
correspond to otherwise mismatched words s'l OS Láii in · the correct grammatical context. That is, GuessedQualifiers are treated as miscellaneous nodes in parsing that match when there are words that are not found in the ontology but are in the correct context to indicate that they are probably qualifiers of the selection class. The difference between GuessedQualifiers and SearchQualifiers is that the latter match vocabulary in the ontology. This distinction allows us to paraphrase the 1002 wizard to identify identity solidly in ios SearchQualifiers and can be more hesitant when it echoes back to the
GuessedQualifiers.
In one mode, wizard 1002 performs the following steps when generating paraphrase text:
one. If the task is unknown, explain what wizard 1002 can do and ask for more power.
2. If the task is a restricted selection task and the location is known, then it explains the domains that the wizard 1002 knows about and asks for the selection class.
3. If the selection class is known, but a required restriction is missing, then ask for that restriction, (for example, location is required for restricted selection in restaurants)
Four. If the feed contains an Entity Name (EntityName) of the selection class, then the output is searching
227
IMPI
MWICANO INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0232.tif" />
<name> in <location>. * '<sup>1</sup>
5. If this is the initial request in a conversation, then the output searching followed by the complex name phrase that describes the constraints.
6. If this is a follow-up refinement stage in the dialogue,
to. If the user requires power, 'then the output is thanks and then paraphrase normally. (This happens when there is a required restriction that is mapped to the user's feed.)
b. If the user is changing a constraint, they acknowledge this and then paraphrase normally.
c. If the user entered the proper name of an instance of the selection class, it handles this in a special way.
d. If the user just added an unrecognized phrase, then it indicates as duplicate as search. If appropriate, the food can be dispatched to a search service.
and. If the user is just adding a normal constraint, then it sends OUTPUT CORRECT (OK), and the paraphrase is done normally.
7. To explain results, use the same approach as for paraphrasing. However, when the results are surprising or unexpected, then explain the results using knowledge about the data and service. Also, when the query is too restricted or too little, ask for more
228
IMPI
INSTITUTO MWICANO M LA FROFItOAD mXJLTXIAL
<img file="MX338784B_D0233.tif" />
feeding.
Grammar for Building Complex Name Phrases
In one embodiment, when paraphrase 7 34 of a constrained selection task query, the base is a complex naming phrase around the selection class that refers to the current constraints. Each constraint has a grammatical position, based on its type. For example, in one modality, wizard 1002 can construct a paraphrase such as:
Recommended romantic Italian restaurants near
Menlo Park with tables available for 2 serving Osso Buco and described as quiet
A grammar for building this is <paraphaseNounClause>: == <binaryConstraint>
<searchQualifier> <cate- goryConstraint> <itemNoun>
<locationConstraint> <availabiltyContraint> <adjectivalClauses>
<binaryConstraint>: == a simple adjective indicating the presence or absence of a Binary Constraint (BinaryConstraint) (for example, recommended (best), within reach (economic))
It is possible to cite more than one in the same query.
<searchQualifier>: == a word or words that match the ontology for a selection class qualifier, which will be passed to a search engine service, (for example, romantic restaurants, funny movies).
It is used when ConstraintType = SearchQualifier.
229
IMPI
INSTITUTO MHUCANO Di LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0234.tif" />
<category Constraint: == an adjective that identifies 6Γ genre, type of cuisine, or category of selection class (for example, Chinese restaurant or R-rated file). It is the last adjective prefix because it is the most intrinsic. Use for characteristics of type Category Constraint and GrammaticalRole = Adj ectiveBeforeNoun.
<itemNoun>: == <namedEntityPhrase> | <selection> | <selection- ClassSubType>
Find the most specific way to display the name. NamedEntity <SubType <Class <selection>: == a name that is the generic name for the selection class (for example, restaurant, movie, site) <selection>: == a noun phrase that is subtype 15 of the class Selection, if known (eg dinner, museum, shop, bar for local selection class business). Use for features where ConstraintType = Categor (Constraint and
GrammaticalRole = Adj ectiveBeforeNoun.
<namedEntityPhrase>: == <entityName> | the (<selectionClass> | <selectionClassSubType>) <entityName>: == the proper name of an instance of the selection class (eg, II Fornaio, Animal House, Harry's
Bar) <locationConstraint>: == <locationPreposition>
230
IMPI
MBCICANO DB INSTITUTE INDUSTRIAL PROPERTY
<img file="MX338784B_D0235.tif" />
<locationName> '<locationPreposition>: == in, near, to, and the like <locationName>: == city, street address, reference, or something GPS your current location
<td><availabilityConstraint></td><td> : ==</td><td>the</td><td>restriction</td><td>of</td>
<td colspan="2">availability expressed as a phrase</td><td colspan="2">of preposition that</td><td>comes</td>
<td>after the noun (for example,</td><td>with</td><td>tables</td><td>available,</td><td>with</td>
<td>seats available, available</td><td>in</td><td>line</td><td>). Comes</td><td>fair</td>
after the noun to give relevance.
<adjectivalClauses>: == <modiferVerbPhrase> | that <modiferVerbPhrase> and <modiferVerbPhrase>
<modiferVerbPhrase>: = a verbal phrase expressing a search keyword style constraint in the selection class (for example, restaurants that are described as quiet, serve meat after 11, matches tragically hip (tragically hip) in reviews; movies containing violence, star Billy Bob Thornton). Includes all constraints in parsing where
GrammaticalRole = ThatClauseModifer, using the variant that ... and if there is more than one. Use Display- TemplatePlural to generate the clauses, putting the GuessedQualifier at the end. If there is only one such restriction, use the Display- TemplateCompact variant.
Table 2 provides some examples of paraphrases
231
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0236.tif" />
that can be provided in response to the first feeding for a task, according to one modality.
<td>Feeding</td><td>Initial Paraphrase</td>
<td>tie house</td><td>Sorry, I don't understand tie house. I know</td>
<td></td><td>regarding restaurants, hotels, and others local businesses, flights, report Weather and buy things online. What are you looking for and where?</td>
<td>Palo Alto</td><td>I can help you find restaurants, business, events and things to do in Palo Alto. What are you looking for?</td>
<td>food</td><td>Looking for a place to eat? Where? (examples)</td>
<td>food</td><td>I can give you Italian restaurants in your</td>
<td>Italian</td><td>area / zone. Where would you like to eat?</td>
<td>burritos</td><td>I can help you find sites that They serve burritos in your area. Where would you like to eat</td>
<td>the best food</td><td>I can help you find good ones</td>
232
<img file="MX338784B_D0237.tif" />
IMPI
INSTITUTO MEJICANO E »THE INDUSTRIAL PROPERTY
<td>mexican</td><td>Mexican restaurants in your</td><td>area.</td>
<td></td><td>Where are you?</td><td></td>
<td>Palo restaurants</td><td>Looking for restaurants near</td><td>Stick</td>
<td>High</td><td>High.</td><td></td>
<td>Italian Palo</td><td>Looking for Italian restaurants</td><td>near</td>
<td>High</td><td>from Palo Alto.</td><td></td>
<td>thai food in</td><td colspan="2">Looking for a Thai restaurant near</td>
<td>Menlo Park</td><td>Menlo Park</td><td></td>
<td>Indonesian food</td><td colspan="2">Looking for Indonesian restaurants</td>
<td>in Menlo Park</td><td>close to Menlo Park</td><td></td>
<td>Best food</td><td colspan="2">Looking for Mexican restaurants</td>
<td>Mexican in Palo</td><td>Recommended near Palo Alto</td><td></td>
<td>High</td><td></td><td></td>
<td>fast food in</td><td colspan="2">Looking for food restaurants</td>
<td>Menlo Park</td><td>quick near Menlo Park</td><td></td>
<td>healthy food in</td><td>Looking for places near Menlo</td><td>Park</td>
<td>Menlo Park</td><td>they serve healthy food.</td><td></td>
<td>small local bar</td><td colspan="2">Looking for small local bars</td>
<td>cheap and dirty</td><td>cheap and dirty with atmosphere</td><td>only</td>
<td>but with atmosphere</td><td>near Menlo Park.</td><td></td>
<td>unique and what is</td><td></td><td></td>
<td>like the secret</td><td></td><td></td>
<td>from customers</td><td></td><td></td>
<td>(say see</td><td></td><td></td>
233
<img file="MX338784B_D0238.tif" />
IMPI
MEXICAN INSTITUTE BE LA INDUSTRIAL EROF1SDAD
<td rowspan="2">bar) in Menlo Park</td><td></td>
<td></td>
<td>billiards in Menlo Park</td><td>Looking for billiards near Menlo Park</td>
<td>pipe room</td><td>Looking for places near Menlo Park</td>
<td>water (hooka) in Menlo Park</td><td>described as hookah room</td>
<td>food</td><td>Looking for places near Menlo Park</td>
<td>spiced in sf</td><td>they serve spicy food</td>
<td>Burgers in</td><td>Looking for places near Menlo Park</td>
<td>Menlo Park</td><td>They serve burgers.</td>
<td>Site with dance</td><td>Looking for places near San</td>
<td>belly in sf</td><td>Francisco described as dance of belly.</td>
<td>Dance site of</td><td>Looking for places near Menlo Park</td>
<td>belly in Menlo Park</td><td>described as belly dancing.</td>
<td>establishments</td><td>Looking for small establishments of</td>
<td>guy guys</td><td>South American guy near San</td>
<td>Southern USA (juke joint) in sf</td><td>Francisco described as juke.</td>
<td>tie house Menlo</td><td>Looking for a place near Menlo Park</td>
<td>Park</td><td>described as a tie house.</td>
234
<img file="MX338784B_D0239.tif" />
ΙΜΡΙ
MIXICAN INSTITUTE DS INDUSTRIAL PROPERTY
<td>thai house</td><td>Stick</td><td></td>
<td>High</td><td></td><td></td>
<td colspan="2">il fornaio Menlo</td><td>Looking for II Fornaio in Palo Alto</td>
<td>Park</td><td></td><td></td>
<td>il fornao</td><td>in</td><td>Looking for places near Palo Alto</td>
<td>Menlo Park</td><td></td><td>described as il fornao.</td>
<td>cliff house</td><td>in</td><td>Looking at Cliff House in Palo Alto</td>
<td>Menlo Park</td><td></td><td></td>
<td>restaurant</td><td></td><td>Looking for romantic restaurants in</td>
<td>romantic in</td><td>Stick</td><td>Palo Alto</td>
<td>High</td><td></td><td></td>
<td>restaurant</td><td></td><td>Looking for romantic restaurants in</td>
<td>romantic in</td><td>Stick</td><td>Palo Alto</td>
<td>High</td><td></td><td></td>
<td>restaurant</td><td></td><td>Looking for romantic restaurants in</td>
<td>Romantic</td><td></td><td>Palo Alto described as quiet</td>
<td>quiet in</td><td>Stick</td><td></td>
<td>High</td><td></td><td></td>
<td>restaurant</td><td></td><td>Looking for romantic restaurants in</td>
<td>Romantic</td><td></td><td>Palo Alto described as quiet</td>
<td>quiet in</td><td>Stick</td><td></td>
<td>High</td><td></td><td></td>
<td>restaurant</td><td>type</td><td>Looking for restaurants in Palo Alto</td>
<td>tragically</td><td>of</td><td>described as tragicallv hip</td>
<td colspan="2">fashion (tragically</td><td></td>
235
<img file="MX338784B_D0240.tif" />
ΙΜΡΙ
INSTITUTO MMICANO • E rRoruD<sub>AD </sub>INDUSTRIAL
<td>hip) High</td><td>in Palo</td><td></td>
<td colspan="2">type restaurant</td><td>Looking for restaurants in Palo Alto</td>
<td colspan="2">tragically of</td><td>described as tragicallv hip</td>
<td>fashion</td><td>(tragically</td><td></td>
<td>hip) </td><td>in Palo</td><td></td>
<td>High</td><td></td><td></td>
<td>table</td><td>for two</td><td>Looking for availability for this</td>
<td>this</td><td>Night in</td><td>night in restaurants near Palo</td>
<td>Stick</td><td>High</td><td>High.</td>
<td>Table</td><td>for two of</td><td>Looking for availability for this</td>
<td colspan="2">Italian food</td><td>night in italian restaurants nearby</td>
<td>this</td><td>Night in</td><td>from Palo Alto.</td>
<td>Stick</td><td>High</td><td></td>
<td colspan="2">il fornaio Palo</td><td>Checking availability for tables</td>
<td>high</td><td>6 pm to 2</td><td>with tonight at II Fornaio in Palo</td>
<td></td><td></td><td>High.</td>
<td>Table</td><td>for 2 in</td><td>Checking availability for tables</td>
<td>thai</td><td>house in PA</td><td>tonight at Thai House in Palo</td>
<td></td><td></td><td>High.</td>
<td>Shipping</td><td>pizza in</td><td></td>
<td>Menlo</td><td>Park</td><td></td>
<td>wish</td><td>the best</td><td>CORRECT, looking for restaurants</td>
<td colspan="2">restaurant</td><td>romantic Italians recommended</td>
<td colspan="2">Italian</td><td>near Menlo Park they serve</td>
236
IMPI
MEXICAN INSTITUTE
M PROPERTY
INDUSTRIAL
<td>Romantic</td><td></td>
<td>quiet</td><td>for</td>
<td>Osso Buco</td><td>in or</td>
<td>near</td><td>Menlo</td>
<td>Park</td><td></td>
<td>Reserve me</td><td>a</td>
<td>table for 2</td><td>at</td>
<td colspan="2">best restaurant</td>
<td>Italian</td><td></td>
<td>Romantic</td><td></td>
<td>quiet</td><td>for</td>
<td>Osso Buco</td><td>in or</td>
<td>near</td><td>Menlo</td>
<td>Park</td><td></td>
<img file="MX338784B_D0241.tif" />
I am Buco and you ”ISÍtUllLit! cuati »calm.
RIGHT, Italians looking for recommended romantic restaurants near Menlo Park serving Osso Buco and described as quiet.
Table 2: Paraphrases in response to the first feeding
Refine a query regarding places to eat
Table 3 provides some examples of paraphrases in response to a situation where the user's intention to find a place to eat is known, but they have not selected a particular place to eat. They are in front of a list of restaurants, and provide refinements.
237
<img file="MX338784B_D0242.tif" />
IMPIf
MEXICAN INSTITUTE V Di LA PROPIEDAD C INDUSTRIAL
<td colspan="2">Refinements</td><td></td>
<td>Menlo Park</td><td>(was</td><td>Thank you, I am now looking for restaurants in</td>
<td>unknown)</td><td></td><td>Menlo Park</td>
<td>More economical</td><td></td><td>Right, I'm looking for restaurants at</td>
<td></td><td></td><td>reach near Menlo Park.</td>
<td>Chinese</td><td>(was</td><td>Right, changing type of cuisine to</td>
<td>Italian)</td><td></td><td>China. Looking for Chinese restaurants</td>
<td></td><td></td><td>near Menlo Park.</td>
<td colspan="2">pizza (add</td><td>Right, looking for restaurants</td>
<td colspan="2">food item)</td><td>Italians near Menlo Park who</td>
<td></td><td></td><td>serve pizza.</td>
<td colspan="2">San Francisco</td><td>Right, changing location to San</td>
<td colspan="2">(it was Menlo Park)</td><td>Francisco. Looking for restaurants</td>
<td></td><td></td><td>Italians near San Francisco)</td>
<td></td><td></td><td>serve pizza.</td>
<td>tragically</td><td>of</td><td>Right, I'll look for reviews by</td>
<td colspan="2">fashion (tragically</td><td>tragically fashionable (tragically</td>
<td>hip)</td><td></td><td>hip).</td>
<td>Romantic</td><td></td><td>Right, looking for restaurants</td>
<td>quiet</td><td></td><td>romantics in Palo Alto described as</td>
<td></td><td></td><td>quiet</td>
<td>cliff house</td><td></td><td>Cliff House sounds familiar. The</td>
<td></td><td></td><td>I will look in Palo Alto.</td>
Table 3: Paraphrases in response to refinements
238
IMPI
INSTITUTO MSKICANO Df THE INDUSTRIAL PROPERTY
<img file="MX338784B_D0243.tif" />
Table 4 provides some exemplary Hn summary results that are provided once results are obtained.
<td>Feeding</td><td># of</td><td>Summary of Results</td>
<td></td><td>Resul- all</td><td></td>
<td>tie house</td><td>N / A</td><td>N / A (no call to services)</td>
<td>Palo Alto</td><td>N / A</td><td>N / A (no call to services)</td>
<td>food</td><td>N / A</td><td>N / A (no call to services)</td>
<td>food Italian</td><td>N / A</td><td>N / A (no call to services)</td>
<td>Burrito</td><td>N / A</td><td>N / A (no call to services)</td>
<td>Best food mexican</td><td>N / A</td><td>N / A (no call to services)</td>
<td>restaurants</td><td>Many</td><td>Here are some restaurants</td>
<td>Palo Alto</td><td></td><td>recommended near Palo High. Do you prefer some kind of particular food? Would you like one with reservation?</td>
239
<img file="MX338784B_D0244.tif" />
IMPI
MEXICAN INSTITUTE DB INDUSTRIAL PROPERTY
<td>Italian Stick</td><td>Some</td><td>Here are some restaurants ”'</td>
<td>High</td><td></td><td>Italians near Palo Alto. To reserve a table do click times available</td>
<td>Thai food in</td><td>Some</td><td>Here are some restaurants</td>
<td>Menlo Park</td><td></td><td>Thai near Menlo Park. To reserve a table do click times available</td>
<td>Food</td><td> 0</td><td>Sorry i can't find</td>
<td>indonesia in</td><td></td><td>no indonesian restaurant</td>
<td>Menlo Park</td><td></td><td>near Menlo Park. Could you try other types of cuisines or locations</td>
<td>Best food</td><td>Some</td><td>Here are some restaurants</td>
<td>Mexican in</td><td></td><td>Recommended Mexicans Nearby</td>
<td>Palo Alto</td><td></td><td>from Palo Alto ·</td>
<td>Fast food</td><td>Some</td><td>Right, here are some</td>
<td>in Menlo Park</td><td></td><td>Fast-food restaurants close to Menlo Park</td>
<td>Healthy food in</td><td>Some</td><td>Here are some sites nearby</td>
<td>Menlo Park</td><td></td><td>from Menlo Park that</td>
240
<img file="MX338784B_D0245.tif" />
IMPI
HST1 MEXICAN INDUSTRIAL PROPERTY TUTE
<td colspan="3" rowspan="2"></td><td rowspan="2"></td><td></td><td></td>
<td>can s'tü'Vll cuurida healthy, based on reviews</td><td></td>
<td>Pub</td><td></td><td>local</td><td>Some</td><td>Here are some bars</td><td></td>
<td>little</td><td colspan="2">cheap and</td><td></td><td>near Menlo Park. No</td><td></td>
<td>dirty</td><td>but</td><td>with</td><td></td><td>I see nothing specific for</td><td></td>
<td colspan="2">environment</td><td>and</td><td></td><td>dive bar. There are others</td><td></td>
<td>secret</td><td></td><td>between</td><td></td><td>ways to say it?</td><td></td>
<td>the</td><td colspan="2">customers</td><td></td><td></td><td></td>
<td>(dive</td><td>Pub)</td><td>in</td><td></td><td></td><td></td>
<td colspan="2">Menlo Park</td><td></td><td></td><td></td><td></td>
<td>Bowling</td><td>in</td><td>Menlo</td><td>Some</td><td>Here are some sites on</td><td></td>
<td>Park</td><td></td><td></td><td></td><td>Menlo Park described</td><td></td>
<td></td><td></td><td></td><td></td><td>like bowling alleys. You can</td><td></td>
<td></td><td></td><td></td><td></td><td>add to other words</td><td></td>
<td></td><td></td><td></td><td></td><td>to describe what</td><td></td>
<td></td><td></td><td></td><td></td><td>you want</td><td></td>
<td colspan="2">Room for</td><td>pipe</td><td> 0</td><td>Can't find sites</td><td></td>
<td>of water</td><td>in</td><td>Menlo</td><td></td><td>at Menlo Park they match</td><td></td>
<td>Park</td><td></td><td></td><td></td><td>with hookah room.</td><td></td>
<td></td><td></td><td></td><td></td><td>Interested in restaurants,</td><td></td>
<td></td><td></td><td></td><td></td><td>hotels or local businesses?</td><td></td>
241
<img file="MX338784B_D0246.tif" />
IMPI
MEXICAN INSTITUTE DS LA MONEDAD INDUSTRIAL
<td>Food</td><td>weak</td><td>Here are some sites</td>
<td>spiced in</td><td>match</td><td>to eat in San</td>
<td>sf</td><td>Inc</td><td>Francisco. I don't know which</td>
<td></td><td></td><td>They serve spicy food.</td>
<td></td><td></td><td>Could I go with another name?</td>
<td>burgers in</td><td>Some</td><td>You can find</td>
<td>Menlo Park</td><td></td><td>burgers in these</td>
<td></td><td></td><td>places to eat near</td>
<td></td><td></td><td>Menlo Park.</td>
<td></td><td></td><td></td>
<td>Dance site</td><td>Some</td><td>Here are some sites on</td>
<td>belly in</td><td></td><td>Menlo Park described</td>
<td>sf</td><td></td><td>like belly dancing.</td>
<td></td><td></td><td>You can add others</td>
<td></td><td></td><td>words to describe what</td>
<td></td><td></td><td>what do you want.</td>
<td>Site with dance</td><td> 0</td><td>I can help you find</td>
<td>belly in</td><td></td><td>restaurants, businesses,</td>
<td>Menlo Park</td><td></td><td>events and things to do</td>
<td></td><td></td><td>in San Francisco. Which</td>
<td></td><td></td><td>are some other words</td>
<td></td><td></td><td>for site with dance of</td>
<td></td><td></td><td>belly?</td>
242
<img file="MX338784B_D0247.tif" />
IMPI iNSTmrro MtxicANo de la rrofiedad industrial
<td rowspan="2">Establishment</td><td rowspan="2"> 0</td><td> —</td>
<td>It can help you to éiifidllLiar-</td>
<td>boy where I know</td><td></td><td>restaurants, businesses,</td>
<td>they serve drinks</td><td></td><td>events and things to do</td>
<td>to dance,</td><td></td><td>in San Francisco. Which</td>
<td>bet and serve</td><td></td><td>are some other words</td>
<td>food (juke</td><td></td><td>for small establishment</td>
<td>joint) at sf</td><td></td><td>where drinks are served</td>
<td></td><td></td><td>to dance, gamble and</td>
<td></td><td></td><td>serve food?</td>
<td>tie house Menlo</td><td> 0</td><td>Sorry, I don't know about</td>
<td>Park</td><td></td><td>tie house in Menlo Park.</td>
<td></td><td></td><td>Any other name for</td>
<td></td><td></td><td>that?</td>
<td>thai house in</td><td> 1</td><td>I get it: Thai Restaurant</td>
<td>Palo Alto</td><td></td><td>House in Palo Alto, CA.</td>
<td></td><td></td><td>Do you want to reserve a table?</td>
<td></td><td></td><td>When and for how many?</td>
<td>il fornaio Menlo</td><td>weak</td><td>There is a restaurant il</td>
<td>Park</td><td>match</td><td>Fornaio in Palo Alto, which</td>
<td></td><td>Inc</td><td>It is close to Menlo Park.</td>
<td>il fornao in</td><td>match</td><td>You mean il Fornaio in</td>
<td>Menlo Park</td><td>company in</td><td>Palo Alto? If not,</td>
243
IMPI
MEXICAN INSTITUTE
K LA MONEDAD
INDUSTKLAL spelling words tries different.
cliff house
Menlo Park in
<img file="MX338784B_D0248.tif" />
romantic restaurant
Palo Alto romantic restaurant
Palo Alto quiet romantic restaurant
Palo Alto some in some in
I couldn't find Cliff
House in Palo Alto. Could I go with a different name or be in a different city?
Here are some restaurants near Palo
High that are described in reviews as romantic.
Here are restaurants near Palo Alto. Looking for reviews I did not find any described as romantic.
Any other preferences?
Here are some restaurants near Palo
High matching romantic, quiet in the reviews.
244
<img file="MX338784B_D0249.tif" />
IMPI
INJTITUTOMWICANO OS LA MONiDAO INDUSTRIAL
<td>restaurant</td><td></td><td> 0</td><td>Here are some</td>
<td>Romantic</td><td></td><td></td><td>restaurants near Palo</td>
<td>quiet</td><td>in</td><td></td><td>High. Looking for reviews no</td>
<td>Palo Alto</td><td></td><td></td><td>I found none that</td>
<td></td><td></td><td></td><td>match romantic</td>
<td></td><td></td><td></td><td>quiet. Any</td>
<td></td><td></td><td></td><td>other preferences?</td>
<td>restaurant</td><td></td><td>Some</td><td>Here are some</td>
<td>tragically</td><td>of</td><td></td><td>restaurants near Palo</td>
<td colspan="2">fashion (tragically</td><td></td><td>High matching on the</td>
<td>hip) in</td><td>Stick</td><td></td><td>reviews with tragically</td>
<td>High</td><td></td><td></td><td>fashionable (tragically hip).</td>
<td>restaurant</td><td></td><td> 0</td><td>Here are some</td>
<td>tragically</td><td>of</td><td></td><td>restaurants near Palo</td>
<td colspan="2">fashion (tragically</td><td></td><td>High. Looking for reviews</td>
<td>hip) in</td><td>Stick</td><td></td><td>i didn't find any that</td>
<td>High</td><td></td><td></td><td>correspond tragically</td>
<td></td><td></td><td></td><td>fashionable (tragically hip).</td>
<td></td><td></td><td></td><td>Any others</td>
<td></td><td></td><td></td><td>preferences?</td>
<td></td><td></td><td>Some</td><td>Here are some</td>
<td></td><td></td><td></td><td>restaurants with tables</td>
245
<img file="MX338784B_D0250.tif" />
IMPI
IKTITUTO MEXICANO D «INDUSTRIAL PROPERTY
<td rowspan="2"></td><td rowspan="2"></td><td></td>
<td>available from 'b £> ΪΓΙ' to 0 pm tonight for group two in Palo Alto, CA.</td>
<td>Table for two</td><td>Some</td><td>Here are some</td>
<td>Italian food</td><td></td><td>restaurants with tables available from 6 pm to 8 pm tonight for group two in Palo Alto, CA.</td>
<td>il fornaio Palo</td><td> 1</td><td>Here are table options</td>
<td>Stop 6 pm for two</td><td></td><td>at restaurant 11 Fornaio in Palo Alto, CA a 6 pm today for a group of 2.</td>
<td>Table for 2 in</td><td> 1</td><td>Here is information for</td>
<td>Thai House in PA</td><td></td><td>Thai House in Palo Alto, AC. You can call them at quoted phone number.</td>
<td>Pizza delivery</td><td></td><td>Here are sites that serve</td>
<td>in Menlo Park</td><td></td><td>pizza in Menlo Park. No I have shipping options at this moment. Click on the numbers</td>
246
<img file="MX338784B_D0251.tif" />
ΙΜΡΙ
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<td colspan="2" rowspan="2"></td><td rowspan="2"></td><td colspan="2"></td>
<td>from phone to call delivery.</td><td>restaurant and request</td>
<td colspan="2"></td><td></td><td colspan="2"></td>
<td>I want the</td><td>best</td><td>weak</td><td colspan="2">Here are the best</td>
<td>restaurant</td><td></td><td>match</td><td>coincidences</td><td>for</td>
<td>Italian</td><td></td><td>Inc,</td><td>restaurant</td><td>Italian</td>
<td>Romantic</td><td></td><td>classifies</td><td colspan="2">recommended romantic close</td>
<td>quiet</td><td>for</td><td>do for</td><td>from Menlo Park</td><td>they serve</td>
<td>Osso Buco</td><td>in or</td><td>available</td><td colspan="2">Osso Buco and are described</td>
<td>near</td><td>Menlo</td><td>quality</td><td>as calm</td><td>. Click on</td>
<td>Park</td><td></td><td></td><td>the times</td><td>available</td>
<td></td><td></td><td></td><td colspan="2">to reserve a table.</td>
<td>Reserve me</td><td>a</td><td>weak</td><td colspan="2">Here are the best</td>
<td>table for</td><td>2 in</td><td>match</td><td>coincidences</td><td>for</td>
<td>the</td><td>best</td><td>Inc,</td><td>restaurant</td><td>Italian</td>
<td>restaurant</td><td></td><td>classifies</td><td colspan="2">recommended romantic close</td>
<td>Italian</td><td></td><td>do for</td><td>from Menlo Park</td><td>with tables</td>
<td>Romantic</td><td></td><td>available</td><td colspan="2">available for two who</td>
<td>quiet</td><td>for</td><td>quality</td><td>they serve Osso</td><td>Buco and be</td>
<td>Osso Buco</td><td>in or</td><td></td><td>describe</td><td>how</td>
<td>near</td><td>Menlo</td><td></td><td>quiet.</td><td>Click on</td>
<td>Park</td><td></td><td></td><td>the times</td><td>available</td>
<td></td><td></td><td></td><td>to reserve</td><td></td>
247
<img file="MX338784B_D0252.tif" />
IMPI
MEXICAN INSTITUTE Dt THE INDUSTRIAL PROPERTY
<td></td><td></td><td colspan="2">a table. ” - ......</td>
<td></td><td></td><td colspan="2"></td>
<td>Ref inamien tos</td><td></td><td colspan="2"></td>
<td>Menlo Park (it</td><td>Some</td><td>Here</td><td>there are some</td>
<td>unknown)</td><td></td><td colspan="2">recommended restaurants</td>
<td></td><td></td><td colspan="2">near Menlo Park)</td>
<td></td><td></td><td>Do you prefer</td><td>some type of</td>
<td></td><td></td><td colspan="2">particular food?</td>
<td>More economical</td><td>Some</td><td>I found</td><td>4 restaurants at</td>
<td></td><td></td><td>scope</td><td>close to Menlo</td>
<td></td><td></td><td>Park.</td><td></td>
<td>Chinese (was</td><td></td><td>I found</td><td>4 restaurants</td>
<td>Italian)</td><td></td><td colspan="2">Chinese near Menlo Park.</td>
<td>pizza (add</td><td>Some</td><td>I found</td><td>4 restaurants</td>
<td>food item)</td><td></td><td>Italians</td><td>close to Menlo</td>
<td></td><td></td><td>Park that</td><td>They serve pizza.</td>
<td>San Francisco</td><td>Some</td><td>I found</td><td>4 restaurants</td>
<td>(it was Menlo Park)</td><td></td><td>Italians</td><td>near San</td>
<td></td><td></td><td>Francisco</td><td></td>
<td>tragically of</td><td>Some</td><td>I found</td><td>4 restaurants</td>
<td>fashion (tragically</td><td></td><td>near</td><td>Palo Alto that</td>
<td>hip)</td><td></td><td>match</td><td>with tragic-</td>
<td></td><td></td><td>mind of</td><td>fashion (tragically</td>
<td></td><td></td><td>hip) ”in</td><td>reviews.</td>
248
<img file="MX338784B_D0253.tif" />
IMPI
MÍXICANO INSTITUTE OF THE INDUSTRIAL PROPBDAU
<td rowspan="2"></td><td rowspan="2">Romantic</td><td rowspan="2">Some</td><td rowspan="2">Here</td><td></td>
<td></td>
<td></td><td>quiet</td><td></td><td colspan="2">restaurants near Palo</td>
<td></td><td></td><td></td><td>Right now</td><td>match with</td>
<td></td><td></td><td></td><td>Romantic</td><td>quiet in</td>
<td> 5</td><td></td><td></td><td>reviews.</td><td></td>
<td></td><td>cliff house</td><td> 0</td><td>I could not</td><td>find cliff</td>
<td></td><td></td><td></td><td colspan="2">House in Palo Alto. Could</td>
<td></td><td></td><td></td><td colspan="2">go with another name or is</td>
<td></td><td></td><td></td><td colspan="2">in a different city?</td>
Table 4: Results Summaries
Table 5 provides some examples of notices that are provided when users click hot links.
Alerts when the user clicks on active links
<td></td><td>Text,</td><td colspan="2">Notice Text</td><td colspan="2">Notes</td>
<td></td><td>Anchor</td><td></td><td></td><td></td><td></td>
<td></td><td>Location,</td><td>Provide</td><td>a</td><td colspan="2">This notice may</td>
<td> 20</td><td>where</td><td colspan="2">location, city,</td><td>be used when</td><td>the</td>
<td></td><td></td><td>Postal Code</td><td>or</td><td>user no</td><td>he has</td>
<td></td><td></td><td>close followed</td><td>by</td><td>still specified</td><td>a</td>
<td></td><td></td><td>the name of a</td><td>site</td><td>Location</td><td></td>
249
<img file="MX338784B_D0254.tif" />
IMPI
MIXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<td rowspan="2">Palo Alto</td><td rowspan="2">Provides location such city, cer ce address close followed reference</td><td rowspan="2">a how cania, lile o of a</td><td></td>
<td>This notice may be used when the user is Changing locations</td>
<td>Kind of</td><td>Provides</td><td>a</td><td>Merges kind of</td>
<td>food</td><td>category of</td><td>food</td><td>food and kind of</td>
<td></td><td>such as china or</td><td>Pizza</td><td>kitchen can</td>
<td></td><td></td><td></td><td>merge</td>
<td>Italian</td><td>You can try</td><td>others</td><td>The user already</td>
<td></td><td>categories</td><td>of</td><td>Italian mentioned.</td>
<td></td><td>such foods</td><td>how</td><td>Wizard 1002</td>
<td></td><td colspan="2">Chinese, or an item from</td><td>helps the user to</td>
<td></td><td>favorite food</td><td>such</td><td>to explore</td>
<td></td><td>I eat meat</td><td></td><td>alternatives, if it is</td>
<td></td><td></td><td></td><td>a food item</td>
<td></td><td></td><td></td><td>dominates over</td>
<td></td><td></td><td></td><td>type of cuisine</td>
<td>Reservation</td><td>Provide day</td><td>and hour</td><td>Warns or indicates a</td>
<td></td><td>to reserve</td><td>a</td><td>reservation</td>
<td></td><td>table, such as'</td><td>morning</td><td></td>
<td></td><td>at 8</td><td></td><td></td>
<td></td><td> 250</td><td></td><td>IMPL • «TITUTO MKICANO \ Ot LA MONEDAD industrial</td><td></td>
<td>Healthy food</td><td colspan="2">You can provide</td><td>Kind of</td><td>food</td>
<td></td><td colspan="2">menu items or type</td><td>known</td><td></td>
<td></td><td>cooking</td><td></td><td></td><td></td>
<td>Food</td><td>Too</td><td>you can</td><td>Kind of</td><td>food</td>
<td>condiment-</td><td>provide</td><td>items of</td><td>unknown</td><td></td>
<td>tada</td><td>menu or types</td><td>cooking</td><td></td><td></td>
<td>Restauran-</td><td colspan="2">What kind of</td><td>Clicking</td><td>at</td>
<td>teas</td><td>restaurants?</td><td>(by</td><td>link</td><td>of</td>
<td></td><td>chinese example</td><td>, Pizza)</td><td>restaurant</td><td>shall</td>
<td></td><td></td><td></td><td colspan="2">insert the word</td>
<td></td><td></td><td></td><td>restaurant</td><td>to the</td>
<td></td><td></td><td></td><td>end of</td><td>the</td>
<td></td><td></td><td></td><td>feeding</td><td>of</td>
<td></td><td></td><td></td><td>text</td><td></td>
<td>Business</td><td>You can</td><td>find</td><td>Clicking</td><td>at</td>
<td></td><td>florists</td><td>local,</td><td colspan="2">business link</td>
<td></td><td>ATMs,</td><td>doctors,</td><td colspan="2">you should add to the</td>
<td></td><td>pharmacy</td><td>and</td><td colspan="2">readable label</td>
<td></td><td>alike.</td><td>What type</td><td>per machine</td><td>than</td>
<td></td><td colspan="2">Are you looking for business?</td><td>This is</td><td>a</td>
<td></td><td></td><td></td><td colspan="2">local search</td>
<td></td><td> 251</td><td></td><td>IMPI xmTVTo mexican of the individual FMOPTSDAD</td><td></td>
<td>Events</td><td>You can</td><td>discover</td><td></td><td></td>
<td></td><td>next</td><td>concerts,</td><td></td><td></td>
<td></td><td colspan="2">shows and</td><td></td><td></td>
<td></td><td>alike,</td><td>what do you</td><td></td><td></td>
<td></td><td>interested?</td><td></td><td></td><td></td>
<td>Things for</td><td colspan="2">Music, art, theater,</td><td></td><td></td>
<td>do</td><td>sports,</td><td>and</td><td></td><td></td>
<td></td><td>seme j before.</td><td>What type</td><td></td><td></td>
<td></td><td>of things</td><td>would you like</td><td></td><td></td>
<td></td><td colspan="2">to do in this area?</td><td></td><td></td>
<td>Hotels</td><td colspan="2">It can help you</td><td></td><td></td>
<td></td><td>find</td><td>a</td><td></td><td></td>
<td></td><td>room</td><td>hotel</td><td></td><td></td>
<td></td><td>available.</td><td>Any</td><td></td><td></td>
<td></td><td>preference</td><td>for</td><td></td><td></td>
<td></td><td>amenities</td><td> 0</td><td></td><td></td>
<td></td><td>Location?</td><td></td><td></td><td></td>
<td>Time o</td><td>Provides</td><td>a city</td><td colspan="2">If the location is</td>
<td>terms</td><td colspan="2">and I'll tell you how is the</td><td>known,</td><td>alone</td>
<td>meteorol-</td><td>time there.</td><td></td><td>shows the</td><td>data</td>
<td>gicas</td><td></td><td></td><td>weather.</td><td></td>
<td>To buy</td><td colspan="2">I can help you</td><td></td><td></td>
<td>things</td><td>find</td><td>music,</td><td></td><td></td>
<td></td><td>films,</td><td>books,</td><td></td><td></td>
252
<img file="MX338784B_D0255.tif" />
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<td></td><td>electronic toys and more and buy them from Amazon. Than you are looking for?</td><td></td>
Table 5: Warnings when the user clicks on active links
Suggesting Possible Answers in a Dialogue
In one embodiment, wizard 1002 provides contextual suggestions.
Suggestions is a way for the wizard 1002 to offer the user options to move from their current location in the dialog. The set of suggestions offered by wizard 1002 is context dependent, and the number of suggestions offered may depend on the medium and the form factor. For example, in one modality, the most salient hints can be offered online in the dialog, a list of extended hints (more) can be offered in a scrolling menu, and even more hints are achieved by typing a few characters and selecting from options. automatic termination. A person skilled in the art will recognize that other mechanisms can be used to provide suggestions.
In various modalities, different types of suggestions can be provided. Examples of suggestion types include:
options to refine a query, including adding or
253
IMPI • MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0256.tif" />
remove or change constraint values;
• options to repair or recover from bad situations, such as not what you intended or start again or search the network;
• options to clarify or reduce ambiguity;
• speech interpretations;
• text interpretations, including speech correction and semantic ambiguity;
• context-specific commands, such as showing these on a map or sending directions to my partner or explaining these results;
• offers suggested cross-selling, such as next steps in event or meal planning scenarios;
• options to reuse previous commands, or their parts.
In various modalities, the context that determines the most relevant suggestions, for example, can be derived from:
• dialog status • user status, including for example:
o static properties (name, home address, etc.) o dynamic properties (location, time, network speed) • interaction history, including, for example:
query history results history the text that has been fed so far in
254
<img file="MX338784B_D0257.tif" />
IMPI
MEXICAN INSTITUTE
Dt THE PROPERTY
INDUSTRIAL automatic termination. - .......
In various modalities, suggestions can be generated by any mechanism, such as:
• paraphrase of a domain, task or restriction based on the ontology model;
• notice on automatic termination based on current domain and restrictions;
• paraphrase of ambiguous alternate interpretations;
• alternate speech-to-text interpretations;
• Authorship by hand, based on special dialogue conditions.
According to a modality, suggestions are generated as operations on commands in some state of completion. Commands are explicit canonical representations of requests, including assumptions and inferences, based on attempted interpretations in the user's feed. In situations where the user feed is incomplete or ambiguous, the suggestions are an attempt to assist the user in adjusting the feed to clarify the command.
In a modality, each command is an imperative phrase that has some combination of:
• command verb (imperative such as find or where it is);
• domain (type of selection such as restaurants);
one or more restrictions such as location
Stick
255
<img file="MX338784B_D0258.tif" />
IMPI
MBKICANO INSTITUTE
OF THE PROPERTY
INDUSTRIAL
High and type of cuisine = Italian. ·
These parts of a command (verb, domain, constraints) correspond to nodes in the ontology.
A suggestion can then be considered as oper ations on a command, such as setting, changing, or declaring it to be relevant or not relevant. Examples include:
• set a command or domain verb (find restaurants) • change a command verb (book, map, save) • change a domain (looking for a restaurant, not a local business) • set a constraint to be relevant (try refining by type of cuisine) • select a value for a restriction (Italian, French and the like) • select a restriction and value together (near here, table for 2) • set that a constraint value is wrong (not that
Boston) • set a constraint not relevant (ignore the expense) • set the intention to change a constraint value (try a different location) • change a constraint value (Italian, not Chinese)
256 add to a constraint value ·· (y aan - alboroa? IMPI
MEXICAN INSTITUTE PE INDUSTRIAL PROPERTY
<img file="MX338784B_D0259.tif" />
too) • snap a value into the plot (Los Angeles, not los angeles) • start a new command, reuse context ([after the movie] fi nd nearby restaurants, send directions to my friend) • start a command that is jump into context (explain these results) • start a new command, reset or ignore context (start again, help with speech)
A suggestion may also involve some combination of the above. For example:
• the movie Milk not [restaurants serving] the food item milk • restaurants serving pizza, not just pizzerias • The place called Costco in Mountain View, it doesn't matter if you think it is a restaurant or a business local • Chinese in Mountain View [recent consultation]
In one embodiment, wizard 1002 includes a general mechanism for maintaining a list of suggestions, ordered by relevance. The format in which a suggestion is offered may differ depending on the current context, mode and form factor of the device.
In one mode, wizard 1002 determines that
257
<img file="MX338784B_D0260.tif" />
IMPI
M axiCANO INSTITUTE
I heard THE OWNERSHIP industrial restrictions modify by considering any or all of the following factors:
• Consider whether the constraint has a value;
• Consider whether the restriction was explicitly inferred or established;
• Consider its relevance (suggestion index).
In one mode, wizard 1002 determines an output format for the suggestion. Examples of output formats include:
• change domain:
• if there is the option to automatically find restaurants, then try something different • other [inferred] I am not looking for restaurants • change name restriction:
• if the name was inferred, offers alternate ambiguous interpretation • inserts current result entity names automatically • different name • considers it was not a name search (remove restriction) - may be offer category instead • not named • not in Berkeley • some other day • not that sense of (uses ambiguity alternatives)
258
IMPI
MÍKICANO INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX338784B_D0261.tif" />
• inferred date: any day, “1Γ0” léqultílü una<sup>1</sup> reservation
In one embodiment, wizard 1002 attempts to resolve ambiguities by suggestions. For example, if the current set of user intention interpretations is too ambiguous 310, then hints are a way to ask for more information 322. In one mode, for restricted selection tasks, wizard 1002 factorizes common constraints between ambiguous interpretations of intention 290 and presents the difference between them to the user. For example, if the user feed includes the word café and this word can correspond to the name of a restaurant or the restaurant type, then the assistant 102 can ask does it mean restaurants with the name café or café restaurant?
In one embodiment, assistant 1002 infers restrictions under certain situations. That is, for restricted selection tasks, not all restrictions need to be explicitly mentioned in the user's feed; some may be inferred from other information available in active ontoiogy 1050, short-term memory 1052 and / or other sources of information available to assistant 1002. For example:
• infer domain or location • predefined consideration, such as location • constraint with weak coincidence (inaccurate coincidence, location of low importance or relevance, etc.)
259
Jl
MEXICAN INSTITUTE
Dt LA nOPSDAD
INDUSTRIAL
<img file="MX338784B_D0262.tif" />
• ambiguous criteria (match to constraint value without prefix) name against category, often ambiguous)
In cases where the wizard 1002 infers constraint values, it can also offer these considerations as suggestions for the user to override. For example, you can tell the user I thought I meant around here. Would you like us to search a different place?
The present invention has been described in particular detail with respect to possible embodiments. Those skilled in the art will appreciate that the invention can be practiced in other embodiments. First, the particular naming of components, capitalization of terms, attributes, data structures, or any other programming or structural aspect is not mandatory or significant, and the mechanisms implementing the invention or its features may have different format names or protocols. In addition, the system can be implemented by a combination of hardware and software as described, either fully in hardware, or fully in hardware. Also, the particular dilution of functionality between the various system components described herein is exemplary only, and not mandatory; functions performed by a single system component can instead be performed by multiple components, and functions performed by multiple components instead can be performed by a single component.
60
IMPI
INSTITUTO MEXICANO DK LA PROPIEDAD INDUSTRIAL
<img file="MX338784B_D0263.tif" />
In various modalities, the present iHVéftól'dñ'pUétttí Sóf 'implemented as a system or a method to perform the techniques described above, either individually or in any combination. In another embodiment, the present invention may be implemented as a computer program product comprising a non-transient computer readable storage medium and computer program code, encoded in the medium, to cause a processor in a computing device or another electronic device perform the techniques described above.
Reference in the specification to an embodiment or an embodiment, means that a particular structure or aspect feature of writing in connection with the embodiments is included in at least one embodiment of the invention. The occurrences of the phrase in a modality at various places in the specification do not necessarily all refer to the same modality.
Some portions of the above are presented in
<td>terms of</td><td>algorithms</td><td>and</td><td>representations</td><td>symbolic</td><td>of</td>
<td>operations in</td><td>bits of</td><td>data</td><td>within a</td><td>memory of</td><td>a</td>
<td>device</td><td>calculation.</td><td>These</td><td>descriptions and</td><td colspan="2">representations</td>
Algorithmic are the means employed by those skilled in data processing techniques to more effectively convey the substance of their work to others skilled in the art. An algorithm is here and generally conceived as a
261
ΙΜΡΙ ικπτυτο MEXICAN Say INDUSTRIAL PROPERTY
<img file="MX338784B_D0264.tif" />
self consistent sequence of steps (instructions')<sup>1</sup> 'ÍJÜS' Lluvarr-e · a desired result. The stages are those that require physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of magnetic or optical electrical signals capable of being stored, transferred, combined, compared, and otherwise manipulated.
It is sometimes convenient, mainly for reasons of common use, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. In addition, it is also convenient at times to refer to certain stage arrangements that require physical manipulations of physical quantities such as modules or code devices, without loss of generality.
It should be kept in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are only convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that through the description, discussions using terms such as processing or computation or calculation or display or determining or the like, they refer to the action and processes of a computer system or module and / or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the memories or records of the computer system or other storage devices, transmission or display of information.
262
MEXICAN INSTITUTE
OE THE PROPERTY
INDUSTRIAL
Certain aspects of this ii<sup>i</sup>iv<sup>i</sup>ciieiürr-<sup>i</sup>± ntluyeii— process steps and instructions described here in the form of an algorithm. It should be noted that the process steps and instructions of the present invention can be incorporated into software, firmware and / or hardware, and when software is incorporated, they can be downloaded to receive on and operate from different platforms used by a variety of operating systems.
The present invention also relates to an apparatus for performing the present operations. This apparatus may be specifically built for the required purposes, or may comprise a general-purpose computing device that is selectively activated or reconfigured by a computer program stored in the computing device. This computer program may be stored on a computer-readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CDROMS, magneto optical disks, read-only memories (ROMS), electronic access (RAMs), EPROMs, EEPROMs, magnetic or optical cards, Application Specific Integrated Circuits (ASICs) or any type of means suitable for storing electronic instructions and each coupled to a computer system busbar. Further. The computing devices referred to here may include a single processor or may be architectures that employ multiple
263
IMPI
MBKICAN INSTITUTE OF LA MONEDAD industrial
<img file="MX338784B_D0265.tif" />
processor designs for increased computing power. ”
The algorithms and displays presented here are not inherently related to any particular computing device, virtualized system, or other apparatus. Various general tank systems can also be used with programs in accordance with the present teachings, or it may prove desirable to build more specialized apparatus to perform the required method steps. The structure required for a variety of these systems will be apparent from the description provided here. Furthermore, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the present invention as described herein, and any previous references to specific languages are provided for the enabling description and best mode of the present invention.
Accordingly, in various embodiments, the present invention can be implemented as software, hardware, and / or other elements to control a computer system, computing device, or other electronic device, or any combination or plurality thereof. . This electronic device may include, for example, a processor, a power supply device (such as a keyboard, mouse, touch pad, ball track, touch panel, remote control).
264
<img file="MX338784B_D0266.tif" />
ΙΜΡΙ
MKICANO INSTITUTE • E INDUSTRIAL PROPERTY games, microphone, and / or any combination of 1st vmc'c), an output device (such as display, speaker, and / or the like), memory, long-term storage (such as magnetic storage, optical storage, and / or the like), and / or network connectivity, according to techniques that are well known in the art. This electronic device can be portable or non-portable. Examples of electronic devices that can be employed to implement the invention include: a mobile phone, personal digital assistant, smartphone, kiosk, desktop computer, laptop, tablet computer, consumer electronic device, device consumer entertainment;
music player; camera; television; decoder; electronic game unit; or the like. An electronic device to implement the present invention can use any operating system such as, for example, iOS or MacOS, available from Apple Inc. of Cupertino, California, or any other system.
Operative that is adapted to use on the device.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, with the benefit of the foregoing description, will appreciate that other embodiments may be designed that do not depart from the scope of the present invention as described herein. . In addition, it should be noted that the wording used in the specification has been selected primarily for readability and
265
IMPI
INSTITUTO MÍXICANO Dt LA MOreOAD INDUSTRIAL
<img file="MX338784B_D0267.tif" />
instructions, and may not have been splprrinnadn to delineate or circumscribe the subject matter of the invention. Accordingly, the description of the present invention is intended to be illustrative, but not limiting of the scope of the invention, which is established in the claims.
266
IMPIOS
MEXICAN INSTITUTE
OF THE EROEIEDAD ΐΛ »-
INDUSTRIAL
Contents347
314 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23 Sheet 24 Sheet 25 Sheet 26 Sheet 27 Sheet 28 Sheet 29 Sheet 30 Sheet 31 Sheet 32 Sheet 33 Sheet 34 Sheet 35 Sheet 36 Sheet 37 Sheet 38 Sheet 39 Sheet 40 Sheet 41 Sheet 42 Sheet 43 Sheet 44 Sheet 45 Sheet 46 Sheet 47 Sheet 48 Sheet 49 Sheet 50 Sheet 51 Sheet 52 Sheet 53 Sheet 54 Sheet 55 Sheet 56 Sheet 57 Sheet 58 Sheet 59 Sheet 60 Sheet 61 Sheet 62 Sheet 63 Sheet 64 Sheet 65 Sheet 66 Sheet 67 Sheet 68 Sheet 69 Sheet 70 Sheet 71 Sheet 72 Sheet 73 Sheet 74 Sheet 75 Sheet 76 Sheet 77 Sheet 78 Sheet 79 Sheet 80 Sheet 81 Sheet 82 Sheet 83 Sheet 84 Sheet 85 Sheet 86 Sheet 87 Sheet 88 Sheet 89 Sheet 90 Sheet 91 Sheet 92 Sheet 93 Sheet 94 Sheet 95 Sheet 96 Sheet 97 Sheet 98 Sheet 99 Sheet 100 Sheet 101 Sheet 102 Sheet 103 Sheet 104 Sheet 105 Sheet 106 Sheet 107 Sheet 108 Sheet 109 Sheet 110 Sheet 111 Sheet 112 Sheet 113 Sheet 114 Sheet 115 Sheet 116 Sheet 117 Sheet 118 Sheet 119 Sheet 120 Sheet 121 Sheet 122 Sheet 123 Sheet 124 Sheet 125 Sheet 126 Sheet 127 Sheet 128 Sheet 129 Sheet 130 Sheet 131 Sheet 132 Sheet 133 Sheet 134 Sheet 135 Sheet 136 Sheet 137 Sheet 138 Sheet 139 Sheet 140 Sheet 141 Sheet 142 Sheet 143 Sheet 144 Sheet 145 Sheet 146 Sheet 147 Sheet 148 Sheet 149 Sheet 150 Sheet 151 Sheet 152 Sheet 153 Sheet 154 Sheet 155 Sheet 156 Sheet 157 Sheet 158 Sheet 159 Sheet 160 Sheet 161 Sheet 162 Sheet 163 Sheet 164 Sheet 165 Sheet 166 Sheet 167 Sheet 168 Sheet 169 Sheet 170 Sheet 171 Sheet 172 Sheet 173 Sheet 174 Sheet 175 Sheet 176 Sheet 177 Sheet 178 Sheet 179 Sheet 180 Sheet 181 Sheet 182 Sheet 183 Sheet 184 Sheet 185 Sheet 186 Sheet 187 Sheet 188 Sheet 189 Sheet 190 Sheet 191 Sheet 192 Sheet 193 Sheet 194 Sheet 195 Sheet 196 Sheet 197 Sheet 198 Sheet 199 Sheet 200 Sheet 201 Sheet 202 Sheet 203 Sheet 204 Sheet 205 Sheet 206 Sheet 207 Sheet 208 Sheet 209 Sheet 210 Sheet 211 Sheet 212 Sheet 213 Sheet 214 Sheet 215 Sheet 216 Sheet 217 Sheet 218 Sheet 219 Sheet 220 Sheet 221 Sheet 222 Sheet 223 Sheet 224 Sheet 225 Sheet 226 Sheet 227 Sheet 228 Sheet 229 Sheet 230 Sheet 231 Sheet 232 Sheet 233 Sheet 234 Sheet 235 Sheet 236 Sheet 237 Sheet 238 Sheet 239 Sheet 240 Sheet 241 Sheet 242 Sheet 243 Sheet 244 Sheet 245 Sheet 246 Sheet 247 Sheet 248 Sheet 249 Sheet 250 Sheet 251 Sheet 252 Sheet 253 Sheet 254 Sheet 255 Sheet 256 Sheet 257 Sheet 258 Sheet 259 Sheet 260 Sheet 261 Sheet 262 Sheet 263 Sheet 264 Sheet 265 Sheet 266 Sheet 267 Sheet 268 Sheet 269 Sheet 270 Sheet 271 Sheet 272 Sheet 273 Sheet 274 Sheet 275 Sheet 276 Sheet 277 Sheet 278 Sheet 279 Sheet 280 Sheet 281 Sheet 282 Sheet 283 Sheet 284 Sheet 285 Sheet 286 Sheet 287 Sheet 288 Sheet 289 Sheet 290 Sheet 291 Sheet 292 Sheet 293 Sheet 294 Sheet 295 Sheet 296 Sheet 297 Sheet 298 Sheet 299 Sheet 300 Sheet 301 Sheet 302 Sheet 303 Sheet 304 Sheet 305 Sheet 306 Sheet 307 Sheet 308 Sheet 309 Sheet 310 Sheet 311 Sheet 312 Sheet 313 Sheet 314
1,637 members in 17 offices
Priority claims5
| Document | Office | Kind | Date |
|---|---|---|---|
| 61295774 | United States of America | – | |
| 29577410 | United States of America | P | |
| 12987982 | United States of America | – | |
| 98798211 | United States of America | A | |
| 2011020861 | United States of America | W |
Members1,637
| Document | Office | Kind | |
|---|---|---|---|
| US2007100790A1 | United States of America | A1 | |
| GB0907592D0 | United Kingdom | D0 | |
| CN101582053A | China | A | |
| GB2459956A | United Kingdom | A | |
| AU2009246654A1 | Australia | A1 | |
| US2009284476A1 | United States of America | A1 | |
| WO2009140095A2 | World Intellectual Property Organization (WIPO) | A2 | |
| JP2010033548A | Japan | A | |
| WO2009140095A3 | World Intellectual Property Organization (WIPO) | A3 | |
| US2010064053A1 | United States of America | A1 | |
| GB201009318D0 | United Kingdom | D0 | |
| HK1137831A | Hong Kong, China | A | |
| HK1137831A1 | Hong Kong, China | A1 | |
| GB2459956B | United Kingdom | B | |
| US2010293462A1 | United States of America | A1 | |
| US2010312547A1 | United States of America | A1 | |
| WO2010141802A1 | World Intellectual Property Organization (WIPO) | A1 | |
| MX2010012494A | Mexico | A | |
| GB2472482A | United Kingdom | A | |
| KR20110014194A | Republic of Korea | A | |
| EP2283424A2 | European Patent Office (EPO) | A2 | |
| TW201112228A | Taiwan Province of China | A | |
| US2011145863A1 | United States of America | A1 | |
| CA2787351A1 | Canada | A1 | |
| CA2791791A1 | Canada | A1 | |
| CA2792412A1 | Canada | A1 | |
| CA2792442A1 | Canada | A1 | |
| CA2792570A1 | Canada | A1 | |
| CA2793002A1 | Canada | A1 | |
| CA2793118A1 | Canada | A1 | |
| CA2793248A1 | Canada | A1 | |
| CA2793741A1 | Canada | A1 | |
| CA2793743A1 | Canada | A1 | |
| CA2954559A1 | Canada | A1 | |
| CA3000109A1 | Canada | A1 | |
| CA3077914A1 | Canada | A1 | |
| CA3203167A1 | Canada | A1 | |
| WO2011088053A2 | World Intellectual Property Organization (WIPO) | A2 | |
| GB2472482B | United Kingdom | B | |
| US2011246891A1 | United States of America | A1 | |
| US2011265003A1 | United States of America | A1 | |
| AU2010254812A1 | Australia | A1 | |
| US2012016678A1 | United States of America | A1 | |
| WO2011088053A3 | World Intellectual Property Organization (WIPO) | A3 | |
| US2012022872A1 | United States of America | A1 | |
| AU2011205426A1 | Australia | A1 | |
| GB201213633D0 | United Kingdom | D0 | |
| MX2012008369A | Mexico | A | |
| US2012245944A1 | United States of America | A1 | |
| AU2009246654B2 | Australia | B2 | |
| US2012265528A1 | United States of America | A1 | |
| AU2012101191A4 | Australia | A4 | |
| GB2490444A | United Kingdom | A | |
| KR20120120316A | Republic of Korea | A | |
| GB201217449D0 | United Kingdom | D0 | |
| CN102792320A | China | A | |
| EP2526511A2 | European Patent Office (EPO) | A2 | |
| US2012309363A1 | United States of America | A1 | |
| US2012311583A1 | United States of America | A1 | |
| US2012311584A1 | United States of America | A1 | |
| US2012311585A1 | United States of America | A1 | |
| WO2012167168A2 | World Intellectual Property Organization (WIPO) | A2 | |
| KR20120136417A | Republic of Korea | A | |
| KR20120137424A | Republic of Korea | A | |
| KR20120137425A | Republic of Korea | A | |
| KR20120137434A | Republic of Korea | A | |
| KR20120137435A | Republic of Korea | A | |
| KR20120137440A | Republic of Korea | A | |
| KR20120138826A | Republic of Korea | A | |
| KR20120138827A | Republic of Korea | A | |
| KR20130000423A | Republic of Korea | A | |
| KR20130005310A | Republic of Korea | A | |
| AU2013200021A1 | Australia | A1 | |
| JP5137899B2 | Japan | B2 | |
| JP2013047954A | Japan | A | |
| WO2012167168A3 | World Intellectual Property Organization (WIPO) | A3 | |
| CA2791277A1 | Canada | A1 | |
| CA3023918A1 | Canada | A1 | |
| MX2012011426A | Mexico | A | |
| EP2575128A2 | European Patent Office (EPO) | A2 | |
| GB2495222A | United Kingdom | A | |
| NL2009544A | Netherlands (Kingdom of the) | A | |
| DE102012019178A1 | Germany | A1 | |
| WO2013048880A1 | World Intellectual Property Organization (WIPO) | A1 | |
| KR20130035983A | Republic of Korea | A | |
| AU2012232977A1 | Australia | A1 | |
| JP2013080476A | Japan | A | |
| US2013110505A1 | United States of America | A1 | |
| US2013110515A1 | United States of America | A1 | |
| US2013110518A1 | United States of America | A1 | |
| US2013110519A1 | United States of America | A1 | |
| US2013110520A1 | United States of America | A1 | |
| US2013111348A1 | United States of America | A1 | |
| US2013111487A1 | United States of America | A1 | |
| AU2012101191B4 | Australia | B4 | |
| US2013115927A1 | United States of America | A1 | |
| US2013117022A1 | United States of America | A1 | |
| JP2013517566A | Japan | A | |
| KR101275466B1 | Republic of Korea | B1 | |
| US2013185074A1 | United States of America | A1 |
Numbers
- Publication
- 338784
- Application
- 11800
Titles2
- Spanish
- PARAFRASIS DE SOLICITUDES DE USUARIO Y RESULTADOS POR ASISTENTE AUTOMATIZADO DIGITAL.
- English
- INTELLIGENT AUTOMATED ASSISTANT.
Classification
- CPC, 37
- G06F3/16
- G06F16/9537
- G06F3/167
- G10L15/22
- G10L15/1815
- G06F16/00
- G10L15/18
- B60K35/10
- B60K35/22
- B60K35/80
- G06F16/3329
- G06F16/3344
- G10L15/1822
- G06N5/022
- G06N3/006
- G06N5/041
- G06F40/35
- G06F40/30
- G10L15/26
- G10L21/06
- G10L2015/223
- G06Q50/12
- G06F9/44
- G06Q10/00
- G06F40/40
- G06F40/279
- H04M1/7243
- H04M1/72448
- H04M1/72484
- G10L15/30
- B60K35/00
- G06F9/54
- G10L13/00
- G10L13/02
- G10L13/08
- G10L15/183
- H04M1/6091
- IPC, 10
- G06F17 28
- G06F17 30
- G06F3 16
- G06F9 54
- G10L15 00
- G10L15 18
- G10L15 22
- G10L15 26
- G10L21 06
- G06F40 00