Distinguishable open sounds
Claim Score by NHIP
Abstract
Systems for speech enabling devices perform methods of configuring distinct open sounds for different devices to indicate to users when each device is recognizing speech. Open sounds are stored both on computer-readable media within a device and on server systems to which devices interface over networks. Open sounds are a parameter of device personalities, and can be configured by system designers, users, or service providers. Devices detect the presence of others by spotting known open phrases, and provide distinctiveness by changing their selected open phrase. Server system providers analyze non-verbal and spoken phrase open sounds from developers using audio fingerprinting and speech recognition.

Term
Projected expiry 31 October 2036.
- Priority and filed
- Published
- Today
- Projected expiry
14 claims: 3 independent, 11 dependent
- 1A non-transitory computer readable medium storing code that, when executed by one or more processors would cause the one or more processors to:receive input indicative of a selection of one of a plurality of distinguishable open sounds to be used for indicating that a system is receptive to a user query;capture audio through a microphone;digitize the audio into audio samples;perform sound spotting using a neural network algorithm on the audio samples, the neural network trained for a specific wake-up phrase;in response to the neural network spotting the specific wake-up phrase, receive speech input through the microphone, the speech input including an audible user query;further in response to spotting the specific wake-up phrase, read an open sound audio segment, corresponding to the selection, from a storage device;and output, through a speaker, the open sound audio segment indicating that a system is receptive to capturing the user's speech, wherein the user is able to distinguish between at least two speech enabled devices within a shared audible environment.
- 8Broadest claimClaim Score 49, average(NHIP)A non-transitory computer readable medium storing code that, when executed by one or more processors would cause the one or more processors to:receive a client request for an open sound selected from a plurality of distinguishable open sounds, the open sound to be used as an indication that the client is receptive to a user's query;according to an indication of which of the plurality of open sounds, read a corresponding open sound audio segment;and transmit the open sound audio segment to the client;capture audio through a microphone;digitize the audio into audio samples;perform sound spotting on the audio samples to detect a specific wake-up phrase;in response to detecting the specific wake-up phrase, output the open sound audio segment, through a speaker, indicating that the client is receptive to capturing the user's speech, wherein the user is able to distinguish between at least two speech enabled devices within a shared audible environment.
- 14A natural language virtual assistant server system enabled to:receive and store at least one domain-specific natural language grammar from a first developer;receive and store at least one open sound selected from a plurality of distinguishable open sounds from the first developer;receive and store at least one domain-specific natural language grammar from a second developer;receive and store at least one open sound selected from the plurality of distinguishable open sounds from the second developer, the at least one open sound of the first developer being distinguishably different from the at least one open sound of the second developer;read and transmit the first open sound to a first device, the first device having a first wake-up phrase;and read and transmit the second open sound to a second device;capture audio through a first microphone of the first device and through a second microphone of the second device;digitize the audio into an audio sample;perform sound spotting on the audio sample at the first device and the second device, to determine if there is a match between the audio sample and at least one of the first wake-up phrase and the second wake-up phrase;and in response to determining a match between the audio sample and at least one of the first wake-up phrase and the second wake-up phrase, activate one of the first device and the second device to output through that device's speaker the corresponding open sound indicating that the corresponding device is receptive to capturing speech, wherein a user is able to distinguish between the first device and the second device within a shared audible environment based on the device's corresponding open sound.
Independent claims3
64 paragraphs in 5 sections, as filed
FIELD OF THE INVENTION
0001The invention is related to computer systems and, more specifically, to embedded systems enhanced with speech recognition.
BACKGROUND
0002Ever increasing numbers of consumer devices are responsive to speech. Some examples of devices are mobile phones, tablets, watches, video gaming systems, televisions, appliances such as refrigerators, home automation and personal assistant devices, robots, and automobiles. Many such devices are “always listening”. That means, they continuously capture audio, such as through microphones, process it, and attempt to spot a specific wake-up phrase. Upon spotting the wake-up phrase, they capture a following speech utterance, and behave in a programmed responsive manner. Many such devices additionally or alternatively accept manual user input such as a tap on a touch screen, a button press, or a gesture. Either by spotting a wake-up phrase or by receiving appropriate manual input such devices detect that a user is addressing them. When such a device receives an indication that a user is addressing it, the device outputs an open sound, such as from a speaker, to indicate to users when the device is receptive to capturing the users' speech.
0003Many such devices use the same models of speech recognition and natural language processing subsystems for reasons such as that their speech recognition software is from the same vendor or source code repository or service is from the same back-end cloud service provider. Each speech recognition system has one or more open sounds such as a beep, boop, blip, or spoken phrases. Since multiple devices, enabled by the same speech recognition system, have the same open sound, users of multiple speech-enabled devices do not sense a distinction between them. Therefore, what is needed is a system and method that provides distinct, distinguishable, or distinguishing open sounds for speech-enabled devices.
SUMMARY OF THE INVENTION
0004The present disclosure describes systems and methods for providing distinct, distinguishable, or distinguishing open sounds for speech-enabled devices. Speech-enabled devices are ones that respond in useful ways to human speech. The methods, systems, and device disclosed herein are beneficial to users by conditioning them to devices so that the users are less likely to issue commands to the wrong devices. It has a further benefit of providing a clue to help users catch their mistakes in addressing the wrong device. It has a further benefit of improving safety by helping users to avoid giving unintended, potentially dangerous commands to the wrong devices.
0005In accordance to the various aspects of the invention, some embodiments are devices that include a stored collection of open sounds. In accordance to the various aspects of the invention, some embodiments are servers that store a library collection of open sounds. Some servers send the open sound audio to client devices in response to a selection for each utterance request. Some servers send open sounds to devices to store on the device. In accordance to the various aspects of the invention, some embodiments are servers that provide for software developers to store multiplicities of open sounds for use in different devices.
0006In accordance to the various aspects of the invention, some embodiments have a close sound that is output when the system detects that a user has stopped speaking at the end of an utterance, such as after a certain period of silence. In some embodiments and devices, the close sound reverses the pattern of tones in a corresponding open sound. For example, an open sound that has musical notes of increasing pitch would correspond with a closing sound that has the same notes but in order of decreasing pitch.
0007In accordance to the various aspects of the invention, some embodiments allow a user to select an open sound for a device. In accordance to the various aspects of the invention, some embodiments have various parameters that give a device a perceived personality. Some or all parameters can be changed together. Some examples of personality parameters are patterns of colors or changing lights, avatars, text-to-speech (TTS) voices, wake-up phrases, natural language grammar rules, open sounds, and close sounds.
0008Such selection can be done through a graphical user interface menu, and can effect change either locally on the device, remotely on a server, or both. In accordance to the various aspects of the invention, some embodiments provide for a software developer, or developer of components of a speech-enabled system, to select from an array, or define custom open sounds and close sounds.
0009In accordance to the various aspects of the invention, some embodiments use one open sound in response to a phrase spotter, but another open sound in response to a tap on a microphone button to indicate the beginning of a user command. In accordance to the various aspects of the invention, some embodiments use one open sound for an initial address after a long period without interaction, but another open sound in response to a follow-on address during a period of recent activity. In accordance to the various aspects of the invention, some embodiments vary open spoken phrase sounds to model anthropomorphic behavior.
0010In accordance to the various aspects of the invention, some embodiments include a plurality of devices that are not responsive to speech, and a speech-enabled controlling device to which the plurality is responsive. Some such embodiments use open sounds stored in each of the plurality of non-responsive devices. In accordance to the various aspects of the invention, some embodiments use open sounds stored in the controlling device, but with distinct open sounds for each of the plurality of non-responsive devices.
0011In accordance to the various aspects of the invention, some embodiments store open sounds on non-transitory computer readable storage media such as hard disk drives, solid state drives, or embedded flash RAM. In accordance to the various aspects of the invention, some embodiments store open sounds as digital audio files in formats such as .wav, .mp3, .flacc, or other comparable formats.
0012In accordance to the various aspects of the invention, some embodiments use a phrase spotter, not just for detecting user addresses, but also for spotting open sounds of other devices. By doing so, a device can configure itself to ensure that it has an open sound that is distinct from other nearby devices. In accordance to the various aspects of the invention, some embodiments monitor the level of ambient noise, such as by sampling, digitizing, and computing a loudness value. Then, adjust the volume used to output the open sound such that the open sound is louder in a noisy environment and quieter in a quiet environment.
0013In accordance to the various aspects of the invention, some embodiments that include open sounds from different developers ensure that different developers have open sounds that are distinctive from each other and from all others. Such embodiments, when receiving a new open sound from a developer, compute a fingerprint of the sound, maintain a database of fingerprints of other open sounds, and compares for a match. If a match is found, then the system rejects the developer's new open sound and informs the developer that it is too close to another.
0014In accordance to the various aspects of the invention, some embodiments use open sounds that are spoken words. Some such embodiments, to ensure distinctive open sounds, perform speech recognition on the spoken words of developers' open sounds and compare the words to a database of text of speech from other open sounds.
0015In accordance to the various aspects of the invention, some embodiments are used for music detection, capture, or analysis. In accordance to the various aspects of the invention, some embodiments are used with speech recognition. In accordance to the various aspects of the invention, some embodiments are used with natural language processing and understanding.
BRIEF DESCRIPTION OF THE DRAWINGS
0016The specification disclosed includes the drawings or figures, wherein like numbers in the figures represent like numbers in the description and the figures are represented as follows:
0017<figref idref="DRAWINGS">FIG. 1</figref> illustrates a user speaking a wake-up phrase in the presence of three speech-enabled devices, according to an embodiment of the invention.
0018<figref idref="DRAWINGS">FIG. 2</figref> illustrates a mobile device enabled with multiple personalities, including distinct open sounds and close sounds, according to an embodiment of the invention.
0019<figref idref="DRAWINGS">FIG. 3</figref> illustrates a speech-enabled device connected to a server that sources open sounds, according to an embodiment of the invention.
0020<figref idref="DRAWINGS">FIG. 4</figref> illustrates a process of selecting and outputting an open sound from a collection, according to an embodiment of the invention.
0021<figref idref="DRAWINGS">FIG. 5</figref> illustrates a process of configuring a device with an open sound from a collection on a server, according to an embodiment of the invention.
0022<figref idref="DRAWINGS">FIG. 6</figref> illustrates a menu for selecting an open sound from a collection of open sounds, according to an embodiment of the invention.
0023<figref idref="DRAWINGS">FIG. 7</figref> illustrates a menu for selecting a personality from a menu, each personality having a distinct open sound, according to an embodiment of the invention.
0024<figref idref="DRAWINGS">FIG. 8</figref> illustrates elements of a system particular to a selected personality, according to an embodiment of the invention.
0025<figref idref="DRAWINGS">FIG. 9</figref> illustrates a process of detecting open known sounds, according to an embodiment of the invention.
0026<figref idref="DRAWINGS">FIG. 10</figref> illustrates a process of fingerprinting new open sounds and comparing the fingerprints to a database, according to an embodiment of the invention.
0027<figref idref="DRAWINGS">FIG. 11</figref> illustrates a process of recognizing speech in new open sounds and comparing the speech to a database, according to an embodiment of the invention.
0028<figref idref="DRAWINGS">FIG. 12</figref> illustrates components of a computer system according to the various aspects of the invention and appropriate for implementing any embodiment of the invention.
DETAILED DESCRIPTION
0029Sometimes users have multiple speech enabled devices. For example, some families have multiple mobile phones and one or more tablets, each of which can respond to the phrase, “Ok Google”. Many such devices easily detect that phrase from across a room. As a result, if a family member attempts to interact with a device using speech by waking it up with the phrase, “Ok Google”, it is possible that multiple devices will respond. They respond with a characteristic bleep sound, which indicates to users the beginning of a speech session. However, they all make the same open sound, and make it simultaneously. As a result, the user who woke up the devices with the phrase does not know which ones are listening, and might not recognize that more than one device is listening.
0030A large number and wide variety of devices, including many other than mobile phones and tablets, are speech enabled. A small number of system providers enable the speech recognition and natural language processing for the wide variety of devices. Households, workplaces, and places of retail and entertainment that have multiple devices, when the devices use default open sounds of common providers, leave users confused as to which device they control.
0031When devices have different open sounds, they give a subconscious clue as to which device is listening. By providing for different devices to have different open sounds, according to the invention, it is clear to users which device is listening when multiple might be awake.
0032People with multiple speech-enabled devices also sometimes invoke the wrong one. Having distinct open sounds, provides a subconscious reminder by notifying the user as to the identity of the device. This helps users notice their mistakes before issuing meaningless, incorrect, or dangerous commands. In effect, the open sound of a device is part of its personality. The system and devices described herein are computer based systems and method. As recognized by those skilled in the art, the conversion of spoken words to digital data, that is then analyzed, and the conversion of digital data into speech in order to provide information to a user is not abstract in concept, but rather significant improvement in technology according to the various aspects and embodiment of the invention as set forth herein.
0033<figref idref="DRAWINGS">FIG. 1</figref> shows usage of an embodiment of the invention. Within a room <b>10</b>, a user <b>12</b> is able to interact with three speech-enabled devices with distinct open sounds, according to embodiments of the invention. One embodiment is a Sibsung brand refrigerator <b>14</b> that spots the wake-up phrase “Hey, Sammy” and has a microphone button for users to indicate an intended address. Refrigerator <b>14</b> outputs an open sound that is speech audio saying, “How may I serve you?” followed by a beep. One embodiment is a Panasoney brand TV set <b>16</b> that spots the wake-up phrase, “Okay, Penny” and recognizes a hand waving gesture to indicate a user address. TV set <b>16</b> outputs an open sound that is speech audio saying, “What's up?” followed by a boop. One embodiment is an Alibazon brand virtual shopper cylinder <b>18</b> that spot the wake-up phrase, “Hey, Ali”. Virtual shopper <b>18</b> outputs an open sound that is speech audio saying, “Good morning.” if the local time is morning, “Good afternoon.” if the local time is afternoon, and “Good evening.” if the local time is evening, followed by a blip.
0034Each of refrigerator <b>14</b>, TV set <b>16</b>, and virtual shopper <b>18</b> including a computer processor and non-volatile memory. The memory in each device stores digital audio segments that, when output through a speaker, makes its characteristic open sound.
0035<figref idref="DRAWINGS">FIG. 2</figref> shows an embodiment in the form of a tablet computer <b>21</b>. It has a display <b>22</b> that shows a microphone button <b>23</b>. Tablet <b>21</b> also spots for wake-up phrases. When a user taps the microphone button or speaks a wake-up phrase, tablet <b>21</b> begins a session in which the user can provide speech utterances. At the beginning of a session, tablet <b>21</b> sends a message to a server; the server responds with an open sound audio segment; and the tablet outputs the audio segment to signal to the user that the session is open.
0036Tablet <b>21</b> also has storage that stores a collection of open sounds. In accordance with the various aspects and embodiments of the invention, a default collection is part of the device operating system. Various distinct speech-enabled apps invoke different open sounds from the collection. App developers may choose which open sound from the collection to use for their apps, may add their own open sound to the tablet and use their own, or provide a menu that allows tablet users to choose an open sound.
0037<figref idref="DRAWINGS">FIG. 3</figref> shows a system of a device that gets open sounds from a server. System <b>30</b> includes a device <b>31</b>, which captures user speech using a microphone <b>32</b>, and outputs audio, including open sounds, to the user through speaker <b>33</b>. Device <b>31</b> corresponds through a network <b>34</b>, such as the Internet. The network <b>34</b> couples device <b>31</b> with a server <b>35</b>. The device <b>31</b> stores a local cache of open sounds. For each user session, the device <b>31</b> sends a request to the server <b>35</b>. The server responds with an open sound. The device <b>31</b> stores the open sound in its cache. For all user speech interactions with the device <b>31</b>, the device <b>31</b> outputs the cached open sound. After a user session ends, such as after a period of five minutes with no speech interaction, the device <b>31</b> marks the cached open sound as stale.
0038In various systems, it is possible for numerous devices to interact with a server. It is also possible for a device to interact with different servers, to provide its own local speech-enablement, or provide a combination of local or server-based speech enablement.
0039<figref idref="DRAWINGS">FIG. 4</figref> shows a process <b>40</b> of speech enablement in accordance with various aspects and an embodiment of the invention. The process begins at step <b>41</b> when a system spots a wake-up phrase. At step <b>42</b>, the system proceeds to select an open sound from a plurality of open sounds stored in memory or database and includes a collection of open sounds <b>43</b>. In some embodiments, the selection is by the design of the device that incorporates the speech enablement system. In some embodiments, the selection is a device configuration choice made by a user. In some embodiments, a third party content provider makes the selection. Some embodiments store open sound collection <b>43</b> in storage or memory located on a user device. Some embodiments store open sounds on a cloud computing server. Any storage location is appropriate and in accordance with the various aspects and embodiments of the invention.
0040After step <b>42</b> of selecting an open sound, process <b>40</b> proceeds to step <b>44</b> and outputs the open sound and step <b>45</b> to begin capturing audio for a user query. In some embodiments, the steps of outputting an open sound and capturing query audio are sequential.
0041<figref idref="DRAWINGS">FIG. 5</figref> shows a process <b>50</b> of speech enablement according to various aspects and an embodiment of the invention. The process begins by a step <b>51</b> that includes configuring a device to use one sound selected from a collection of sounds <b>52</b>. Process <b>50</b> proceeds to take the sound selected during step <b>51</b> for device configuration and stores it as open sound <b>53</b>. In some embodiments, the configuration step <b>51</b> happens during the design of a system. In some embodiments, the configuration step <b>51</b> happens during manufacturing of a system. In some embodiments, the configuration step <b>51</b> happens as part of a retail sales process. Some such retail sales processes are those of online retailers, ringtone sales, app stores, and speech-based purchasing systems. In some embodiments, the configuration step <b>51</b> happens as part of a user set. In some embodiments, the configuration step <b>51</b> happens through in-field firmware updates.
0042The process <b>50</b> proceeds to step <b>54</b>, for every user session, to spot a wake-up phrase; to step <b>55</b> to output the open sound; and to step <b>56</b> for capture user query audio <b>56</b>.
0043<figref idref="DRAWINGS">FIG. 6</figref> shows a system menu according to an embodiment. Open Sound menu <b>61</b> is part of a graphical user interface (GUI). It allows a user to select one of five open sounds in a collection. The number of open sounds in the collection can be varied to more or fewer open sounds. The open sounds have vaguely descriptive names with pleasing connotations. They are the audio equivalent of the names of house paint colors.
0044Some embodiments are devices that have personalities. <figref idref="DRAWINGS">FIG. 7</figref> shows a personality selection menu for such an embodiment. Personality menu <b>71</b> offers five choices of personalities. The number of personalities can vary to more or fewer open sounds. Each has an anthropomorphic name that is very vaguely descriptive of a personality. Various elements of a system contribute to its anthropomorphic personality.
0045<figref idref="DRAWINGS">FIG. 8</figref> shows a set of elements <b>80</b> that are stored as part of a device personality. Wake-up phrase <b>81</b> defines how a user invokes a session with the device, and is typically a phrase beginning with “Okay” or “Hey”, followed by a two or three syllable name that is anthropomorphic, but uncommon. Text-to-speech (TTS) voice <b>82</b> defines a voice that the system uses to output verbal communication to users. Most TTS voices are distinctly male or female, and have distinct accents and patterns of intonation. Open sound <b>83</b> and close sound <b>84</b> are the audio used to indicate the beginning and ending of a speech session between the user and system. Open sounds and close sounds can have short non-verbal audio segments such as beeps, boops, blips, dings, whooshes, whistles, snaps, cracks, pops, or other appropriate sounds. Open and close sounds can, alternatively or additionally, have spoken phrase audio. Grammar rules <b>85</b> are the vocabulary, word patterns, rules for interpretation, and domains of knowledge that the system may use to understand user speech.
0046Some embodiments use multiple open sounds for the same device. This is particularly useful if the open sounds are spoken phrases. Humans tend to vary their responses, when addressed, based on the situation, conditions, or mood. By a device varying the spoken phrase open sound, users perceive it as more anthropomorphic. Some embodiments of systems that provide different open sounds from the same device provide for customizing the set of open sounds from which the system can choose. For example, a refrigerator might randomly switch between spoken phrase open sounds saying, “How may I serve you?”, and “What would you like?”, whereas a television, when its display is off, uses the spoken phrase open sound, “What would you like to see?”, and, when the display is on, tersely say, “Yes?”.
0047For some types of devices, it is not convenient or practical for users to configure the device personality or open sound, such as from a menu. Some embodiments, such as devices that might be placed within speaking distance of others of the same model, need to avoid the problem of having the same open sound.
0048<figref idref="DRAWINGS">FIG. 9</figref> shows a process for such embodiments to do so. Process <b>90</b> includes step <b>91</b> for continuously capturing ambient audio. Next the system performs sound spotting at step <b>92</b>. This is performed using the same neural network, trained on audio segments for small vocabulary speech recognition, that the system uses for wake-up phrase spotting. The training for sound spotter step <b>92</b> is done a priori from a collection of sounds <b>93</b> used to create acoustic model <b>94</b>. When, at sound spotter step <b>92</b>, the device spots captured audio, from step <b>91</b>, that corresponds to a sound from the sound collection <b>93</b>, if the sound matched in sounds collection <b>93</b> is the same as the system's currently selected open sound then the system proceeds to select a new open sound at step <b>95</b>.
0049Some embodiments select a new open sound by choosing the next on a list of open sounds. Some embodiments select an open sound randomly from the sounds collection. Some embodiments select not just an open sound, but an entire personality. By doing so, similar model devices automatically become distinct from each other within a shared audible environment.
0050Some embodiments are shared systems, such as ones based on cloud servers, which support many types of devices. Device and interface designers using such systems create their own open and close sounds and upload them to the shared system. It is desirable to ensure that different designers have distinct open sounds, or at least similar types of devices, such as ones from competitors serving the same end-user markets, have distinct open sounds.
0051<figref idref="DRAWINGS">FIG. 10</figref> shows an embodiment that provides for such distinctiveness. The system performs process <b>100</b>, which begins when the system receives a new open sound <b>101</b>. The system, at step <b>102</b>, computes a fingerprint of the open sound <b>101</b>. The system also stores a database of all known device open sounds database <b>103</b>. In step <b>104</b>, the system proceeds to compare the fingerprint from step <b>102</b> to fingerprints from database <b>103</b> using a known method of fingerprint comparison, for example, as used for music recognition. If the system detects a match between the fingerprint of new open sound <b>101</b> and a fingerprint stored in database <b>103</b>, then the process proceeds, at step <b>105</b>, to notify the user and the system operator of the overlap between the open sound <b>101</b> and the fingerprint in the database <b>103</b>. Some systems automatically reject a new open sound and refuse to provide it to supported devices.
0052Some embodiments of shared systems, additionally or alternatively, enforce distinctiveness between spoken phrase open sounds. <figref idref="DRAWINGS">FIG. 11</figref> shows one such embodiment. Process <b>110</b> begins by receiving open sound <b>111</b>. It performs a speech recognition, at step <b>112</b>, using a known method of speech recognition. Process <b>110</b> proceeds to search a sound phrase database <b>113</b>, which includes textual representations of speech recognized from each stored open sound. At step <b>114</b>, the system compares the speech recognized, at step <b>112</b>, to the phrases in the sound phrase database <b>113</b>. If a phrase in the database is sufficiently similar to speech recognized from open sound <b>111</b>, then process <b>110</b> proceeds to step <b>115</b> and refuses to accept the open sound <b>111</b> and notifies the developer and system operator.
0053Some embodiments perform simple text string matching. Some embodiments perform fuzzy matching between the recognized speech and speech in the phrase database. Some embodiments include word synonyms in the search. Some embodiments perform natural language understanding algorithms on the speech and compare speech intents. Some embodiments, if they detect no spoken words in speech recognition step <b>112</b> exit the process without comparison step <b>114</b>. Some embodiments check recognized speech text for trademarked names and profane language.
0054Some embodiments are implemented in software that runs on computer processors. One such embodiment is shown in <figref idref="DRAWINGS">FIG. 12</figref>. Computer system <b>120</b> includes parallel processors <b>121</b> and <b>122</b>, which connect through caches <b>123</b> and <b>124</b>, respectively, to interconnect <b>125</b>, through which the processors can execute software from instructions and operate on data stored in random access memory (RAM) <b>126</b> and non-volatile memory <b>127</b>. Software running on computer system <b>120</b> accesses the Internet through network interface <b>128</b>, provides a GUI through display controller <b>129</b>, and accepts user input through I/O controller <b>1210</b>, all of which are also connected through interconnect <b>125</b>.
0055In some embodiments, the processors are ARM instruction set processors. In some embodiments they are x86 processors. In some embodiments, memories, controllers, and interfaces are all on the same system-on-chip. In some embodiments, some elements are in different chips. In some embodiments, the non-volatile memory is a hard disk drive. In some embodiments, it is a solid-state drive. In some embodiments, the display controller connects to a local device display panel through a mobile industry processor interface (MIPI) display serial interface (DSI). In some embodiments, the display controller connects to a HDMI connector. In various embodiments, the I/O controller interfaces to touch screens, keyboards, mice, microphones, speakers, and USB connectors. In various embodiments, the network interface is an Ethernet cable interface, WiFi interface, Bluetooth interface, and 5G LTE interface.
0056In some embodiments, receiving and transmitting between clients and servers is through direct connections. In some embodiments, clients and servers are coupled through intermediate media, such as busses or computer networks, and receiving and transmitting are indirect.
0057Embodiments of the invention described herein are merely exemplary, and should not be construed as limiting of the scope or spirit of the invention as it could be appreciated by those of ordinary skill in the art. The disclosed invention is effectively made or used in any embodiment that comprises any novel aspect described herein. All statements herein reciting principles, aspects, and embodiments of the invention are intended to encompass both structural and functional equivalents thereof. It is intended that such equivalents include both currently known equivalents and equivalents developed in the future.
0058The behavior of either or a combination of humans and machines (instructions that, when executed by one or more computers, would cause the one or more computers to perform methods according to the invention described and claimed and one or more non-transitory computer readable media arranged to store such instructions) embody methods described and claimed herein. Each of more than one non-transitory computer readable medium needed to practice the invention described and claimed herein alone embodies the invention.
0059Some embodiments of physical machines described and claimed herein are programmable in numerous variables, combinations of which provide essentially an infinite variety of operating behaviors. Some embodiments of hardware description language representations described and claimed herein are configured by software tools that provide numerous parameters, combinations of which provide for essentially an infinite variety of physical machine embodiments of the invention described and claimed. Methods of using such software tools to configure hardware description language representations embody the invention described and claimed. Physical machines, such as semiconductor chips; hardware description language representations of the logical or functional behavior of machines according to the invention described and claimed; and one or more non-transitory computer readable media arranged to store such hardware description language representations all can embody machines described and claimed herein.
0060In accordance with the teachings of the invention, a computer and a computing device are articles of manufacture. Other examples of an article of manufacture include: an electronic component residing on a mother board, a server, a mainframe computer, or other special purpose computer each having one or more processors (e.g., a Central Processing Unit, a Graphical Processing Unit, or a microprocessor) that is configured to execute a computer readable program code (e.g., an algorithm, hardware, firmware, and/or software) to receive data, transmit data, store data, or perform methods.
0061The article of manufacture (e.g., computer or computing device) includes a non-transitory computer readable medium or storage that may include a series of instructions, such as computer readable program steps or code encoded therein. In certain aspects of the invention, the non-transitory computer readable medium includes one or more data repositories. Thus, in certain embodiments that are in accordance with any aspect of the invention, computer readable program code (or code) is encoded in a non-transitory computer readable medium of the computing device. The processor or a module, in turn, executes the computer readable program code to create or amend an existing computer-aided design using a tool. The term “module” as used herein may refer to one or more circuits, components, registers, processors, software subroutines, or any combination thereof. In other aspects of the embodiments, the creation or amendment of the computer-aided design is implemented as a web-based software application in which portions of the data related to the computer-aided design or the tool or the computer readable program code are received or transmitted to a computing device of a host.
0062An article of manufacture or system, in accordance with various aspects of the invention, is implemented in a variety of ways: with one or more distinct processors or microprocessors, volatile and/or non-volatile memory and peripherals or peripheral controllers; with an integrated microcontroller, which has a processor, local volatile and non-volatile memory, peripherals and input/output pins; discrete logic which implements a fixed version of the article of manufacture or system; and programmable logic which implements a version of the article of manufacture or system which can be reprogrammed either through a local or remote interface. Such logic could implement a control system either in logic or via a set of commands executed by a processor.
0063Furthermore, all examples and conditional language recited herein are principally intended to aid the reader in understanding the principles of the invention and the concepts contributed by the inventors to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and embodiments of the invention, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents and equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.
0064The scope of the invention, therefore, is not intended to be limited to the exemplary embodiments shown and described herein. Rather, the scope and spirit of present invention is embodied by the appended claims.
Contents5
13 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13
Every citation, both ways
| Document | Relation | Office | Cited during |
|---|---|---|---|
| US11792590B2 | Cited by | United States of America | Applicant |
| US11790911B2 | Cited by | United States of America | Applicant |
| US11380322B2 | Cited by | United States of America | Applicant |
| US11556307B2 | Cited by | United States of America | Applicant |
| US12217748B2 | Cited by | United States of America | Applicant |
| US11726742B2 | Cited by | United States of America | Applicant |
| US11641559B2 | Cited by | United States of America | Applicant |
| US12308026B2 | Cited by | United States of America | Search report |
| US11869503B2 | Cited by | United States of America | Applicant |
| US11714600B2 | Cited by | United States of America | Applicant |
| US11538460B2 | Cited by | United States of America | Applicant |
| US11451908B2 | Cited by | United States of America | Applicant |
| US11132989B2 | Cited by | United States of America | Applicant |
| US11212612B2 | Cited by | United States of America | Applicant |
| US12505832B2 | Cited by | United States of America | Applicant |
| US11563842B2 | Cited by | United States of America | Applicant |
| US11797263B2 | Cited by | United States of America | Applicant |
| US2023118412A1 | Cited by | United States of America | Search report |
| US11694689B2 | Cited by | United States of America | Applicant |
| US11741948B2 | Cited by | United States of America | Applicant |
| US11900937B2 | Cited by | United States of America | Applicant |
| US11727936B2 | Cited by | United States of America | Applicant |
| US11832068B2 | Cited by | United States of America | Applicant |
| US11646045B2 | Cited by | United States of America | Applicant |
| US11562740B2 | Cited by | United States of America | Applicant |
| US11769505B2 | Cited by | United States of America | Applicant |
| US11727933B2 | Cited by | United States of America | Applicant |
| US12424220B2 | Cited by | United States of America | Applicant |
| US11715489B2 | Cited by | United States of America | Applicant |
| US12165644B2 | Cited by | United States of America | Applicant |
| US11175888B2 | Cited by | United States of America | Applicant |
| US11893308B2 | Cited by | United States of America | Applicant |
| US11513763B2 | Cited by | United States of America | Applicant |
| US11646023B2 | Cited by | United States of America | Applicant |
| US12360734B2 | Cited by | United States of America | Applicant |
| US12236932B2 | Cited by | United States of America | Applicant |
| US11516610B2 | Cited by | United States of America | Applicant |
| US11696074B2 | Cited by | United States of America | Applicant |
| US12165651B2 | Cited by | United States of America | Applicant |
| US12211490B2 | Cited by | United States of America | Applicant |
| US11501773B2 | Cited by | United States of America | Applicant |
| US11302326B2 | Cited by | United States of America | Applicant |
| US10970035B2 | Cited by | United States of America | Applicant |
| US11432030B2 | Cited by | United States of America | Applicant |
| US11676590B2 | Cited by | United States of America | Applicant |
| US11531520B2 | Cited by | United States of America | Applicant |
| US11200900B2 | Cited by | United States of America | Applicant |
| US11405430B2 | Cited by | United States of America | Applicant |
| US11922095B2 | Cited by | United States of America | Applicant |
| US11482224B2 | Cited by | United States of America | Applicant |
| US11024331B2 | Cited by | United States of America | Applicant |
| US12230291B2 | Cited by | United States of America | Applicant |
| US11983463B2 | Cited by | United States of America | Applicant |
| US11540047B2 | Cited by | United States of America | Applicant |
| US12327556B2 | Cited by | United States of America | Applicant |
| US11175880B2 | Cited by | United States of America | Applicant |
| US11961519B2 | Cited by | United States of America | Applicant |
| US10971139B2 | Cited by | United States of America | Applicant |
| US11862161B2 | Cited by | United States of America | Applicant |
| US10880378B2 | Cited by | United States of America | Search report |
| CN108847232A | Cited by | China | Search report |
| US11315556B2 | Cited by | United States of America | Applicant |
| US11006214B2 | Cited by | United States of America | Applicant |
| US12387716B2 | Cited by | United States of America | Applicant |
| US11895471B2 | Cited by | United States of America | Search report |
| US2018322865A1 | Cited by | United States of America | Search report |
| US11500611B2 | Cited by | United States of America | Applicant |
| US11557294B2 | Cited by | United States of America | Applicant |
| US11790937B2 | Cited by | United States of America | Applicant |
| US11343614B2 | Cited by | United States of America | Applicant |
| US11875820B1 | Cited by | United States of America | Applicant |
| US11914588B1 | Cited by | United States of America | Search report |
| US11200894B2 | Cited by | United States of America | Applicant |
| US11482978B2 | Cited by | United States of America | Applicant |
| US11551700B2 | Cited by | United States of America | Applicant |
| US10522136B2 | Cited by | United States of America | Search report |
| US10878811B2 | Cited by | United States of America | Applicant |
| US11899519B2 | Cited by | United States of America | Search report |
| US11727919B2 | Cited by | United States of America | Applicant |
| US11080005B2 | Cited by | United States of America | Applicant |
| US11183183B2 | Cited by | United States of America | Applicant |
| US11736860B2 | Cited by | United States of America | Applicant |
| US11538451B2 | Cited by | United States of America | Applicant |
| US2018366107A1 | Cited by | United States of America | Search report |
| US11514898B2 | Cited by | United States of America | Applicant |
| US11430442B2 | Cited by | United States of America | Search report |
| US11354092B2 | Cited by | United States of America | Applicant |
| US12062383B2 | Cited by | United States of America | Applicant |
| US11189286B2 | Cited by | United States of America | Applicant |
| US11698771B2 | Cited by | United States of America | Applicant |
| US11488592B2 | Cited by | United States of America | Search report |
| US12518756B2 | Cited by | United States of America | Applicant |
| US12567435B1 | Cited by | United States of America | Applicant |
| US11308958B2 | Cited by | United States of America | Applicant |
| US11308961B2 | Cited by | United States of America | Applicant |
| US11288039B2 | Cited by | United States of America | Applicant |
| US11984123B2 | Cited by | United States of America | Applicant |
| US11308962B2 | Cited by | United States of America | Applicant |
| US12047753B1 | Cited by | United States of America | Applicant |
| US11545169B2 | Cited by | United States of America | Applicant |
1 member in 1 office
Members1
| Document | Office | Kind | |
|---|---|---|---|
| US2018122372A1 | United States of America | A1 |
48 transactions on the USPTO file
Abandoned after 1 non-final rejection and 1 final rejection.
- Non-final rejections
- 1
- Final rejections
- 1
- RCEs
- 0
- Appeals
- 0
Over time
Point at a mark for the transactionTransactions
| Event | Code | |
|---|---|---|
| Email NotificationEML_NTR | EML_NTR | |
| Mail Abandonment for Failure to Respond to Office ActionAbandonedMABN2 | MABN2 | |
| Aband. for Failure to Respond to O. A.AbandonedABN2 | ABN2 | |
| Email NotificationEML_NTR | EML_NTR | |
| Mail Advisory Action (PTOL - 303)MCTAV | MCTAV | |
| After Final Consideration Program Additional Consideration and/or updated searchAFAC | AFAC | |
| Interview Summary - Examiner Initiated - TelephonicEXET | EXET | |
| Advisory Action (PTOL-303)CTAV | CTAV | |
| Date Forwarded to ExaminerFWDX | FWDX | |
| Reference capture on IDSRCAP | RCAP | |
| Response after Final ActionA.NE | A.NE | |
| Email NotificationEML_NTR | EML_NTR | |
| Application ready for PDX access by participating foreign officesCCRDY | CCRDY | |
| PG-Pub Issue NotificationPG-ISSUE | PG-ISSUE | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Email NotificationEML_NTF | EML_NTF | |
| Mail Final Rejection (PTOL - 326)Final rejectionMCTFR | MCTFR | |
| Final RejectionFinal rejectionCTFR | CTFR | |
| Date Forwarded to ExaminerFWDX | FWDX | |
| Response after Non-Final ActionA... | A... | |
| Request for Extension of Time - GrantedXT/G | XT/G | |
| Mail Interview Summary - Applicant Initiated - TelephonicMEXAT | MEXAT | |
| Interview Summary - Applicant Initiated - TelephonicEXAT | EXAT | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Email NotificationEML_NTF | EML_NTF | |
| Mail Non-Final RejectionNon-final rejectionMCTNF | MCTNF | |
| Non-Final RejectionNon-final rejectionCTNF | CTNF | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Application Dispatched from OIPEOIPE | OIPE | |
| Email NotificationEML_NTR | EML_NTR | |
| Application Is Now CompleteCOMP | COMP | |
| Filing Receipt - UpdatedFLRCPT.U | FLRCPT.U | |
| Sent to Classification ContractorPGPC | PGPC | |
| FITF set to YES - revise initial settingFTFS | FTFS | |
| Patent Term Adjustment - Ready for ExaminationPTA.RFE | PTA.RFE | |
| Additional Application Filing FeesADDFLFEE | ADDFLFEE | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Email NotificationEML_NTF | EML_NTF | |
| Email NotificationEML_NTR | EML_NTR | |
| Notice Mailed--Application Incomplete--Filing Date AssignedINCD | INCD | |
| Filing ReceiptFLRCPT.O | FLRCPT.O | |
| Cleared by OIPE CSRL194 | L194 | |
| PTO/SB/69-Authorize EPO Access to Search ResultsSREXR141 | SREXR141 | |
| Applicants have given acceptable permission for participating foreignAPPERMS | APPERMS | |
| IFW Scan & PACR Auto Security ReviewSCAN | SCAN | |
| Entity Status Set To Undiscounted (Initial Default Setting or Status Change)BIG. | BIG. | |
| Initial Exam Team nnIEXX | IEXX |
20 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| Information on status: application discontinuationABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTIONSTCB | STCB | |
| AssignmentAS | AS |
Numbers
- Publication
- 20180122372
- Application
- 15339291
Titles
- English
- DISTINGUISHABLE OPEN SOUNDS
Classification
- CPC, 3
- G10L15/22
- G06F3/167
- G10L21/0324
- IPC, 2
- G10L15 22
- G06F3 16