Systems, methods, and apparatus for estimating direction of arrival
15 claims: 5 independent, 10 dependent
- 1電子デバイス上にユーザインターフェースを表示するための方法であって、前記方法は下記を備える、 ユーザインターフェースを提示すること、ここにおいて、前記ユーザインターフェースが座標系を備え 、前 記座標系が、センサデータに基づく物理座標に対応する、と、 前 記座標系の少なくとも1つのセクタの選択を可能にするセクタ選択機能を提供すること、 ここにおいて、前記少なくとも1つのセクタが、複数のマイクロフォンからの捕捉されたオーディオに対応し、セクタ選択が、オーディオ信号インジケータを備える、 と、 前記座標系の前記少なくとも1つのセクタの少なくとも1つの境界への調整を提供する受信された入力に基づいて、 前記少なくとも1つのセクタの サイズを調整すること を可能にするセクタ編集機能を提供することと、 前記セクタ選択に基づいて、前記オーディオ信号インジケータに対応する前記捕捉されたオーディオに対してオーディオ動作を実行すること 。
- 2前記電子デバイスは、少なくとも1つのマイクロフォンを備え、前記方法は、前記少なくとも1つのマイクロフォンによって捕捉された少なくとも1つのオーディオ信号の方向を表示することを備え、 好ましくは、前記少なくとも1つのオーディオ信号が音声信号を備え、 好ましくは、前記方法が前記少なくとも1つのオーディオ信号に対応するアイコンを表示することをさらに備え、 さらに好ましくは、前記方法がターゲットオーディオ信号に関するアイコンおよび干渉オーディオ信号に関するアイコンのうちの少なくとも1つを表示することをさらに備える、請求項1に記載の方法。
- 3前記少なくとも1つのセクタ内に示されたオーディオ信号を再生のために渡すこと、および/または前記少なくとも1つのセクタ内に示されないオーディオ信号を再生のために減衰すること、をさらに備える、請求項1に記載の方法。
- 4前記電子デバイスは、1つまたは複数の画像センサを備え、前記方法は、前記1つまたは複数の画像センサからの画像データを示すこと、および/または前記1つまたは複数のセクタに基づいて画像データを渡すこと、をさらに備える、請求項1に記載の方法。
- 5前記セクタ選択機能および前記セクタ編集機能のうちの少なくとも1つが、シングルタッチ入力とマルチタッチ入力とからなるグループのうちの少なくとも1つに基づいて動作し、好ましくは、前記方法は、 前記少なくとも1つのセクタに対応する少なくとも1つのタッチポイントを表示することと、 前記少なくとも1つのタッチポイントに対応するタッチ入力を受信することと、 前記タッチ入力に基づいて、前記少なくとも1つのセクタを編集することと、 をさらに備える、請求項1に記載の方法。
- 6前記ユーザインターフェースの少なくとも一部を基準面と整合させることをさらに備え、好ましくは、前記基準面が水平である、および/または、好ましくは、前記ユーザインターフェースの少なくとも一部を整合させる、請求項1に記載の方法。
- 7前記物理座標が、地球座標であるか、または、地球座標とは無関係な物理空間を表すか、のいずれかである、請求項1に記載の方法。
- 8前記座標系が、電子デバイスの方向とは無関係な方向を維持する、請求項1に記載の方法。
- 9オーディオ署名を認識することと、 データベース内の前記オーディオ署名を調べることと、 前記オーディオ署名に対応する識別情報を取得することと、 前記ユーザインターフェース上に前記識別情報を表示することと をさらに備え、 好ましくは、前記識別情報が、前記オーディオ署名に対応する人物の画像である、請求項1に記載の方法。
- 10前記ユーザインターフェース が固 定モード と編 集可能なモードを切り替えるための制御をさらに備える、請求項1に記載の方法。
- 11前記セクタ選択機能が複数のセクタの同時の選択を可能にするおよび/または1つまたは複数のスワイプ入力に基づき、好ましくは、前記1つまたは複数のスワイプ入力が円領域を示す、または、前記1つまたは複数のスワイプ入力が単一のスワイプである、請求項1に記載の方法。
- 12前記セクタ編集機能が、シングルタッチ入力またはマルチタッチ入力に基づいて前記セクタの調整を可能にする、請求項1に記載の方法。
- 13ユーザインターフェースを表示するための装置であって、前記装置は下記を備える、 ユーザインターフェースを提示するための手段、ここにおいて、前記ユーザインターフェースが座標系を備え、前記座標系が、センサデータに基づく物理座標に対応する、と、 前 記座標系の少なくとも1つのセクタの選択を可能にするセクタ選択機能を提供するための手段 、ここにおいて、前記少なくとも1つのセクタが、複数のマイクロフォンからの捕捉されたオーディオに対応し、セクタ選択が、オーディオ信号インジケータを備える、 と、 前記座標系の前記少なくとも1つのセクタの少なくとも1つの境界への調整を提供する受信された入力に基づいて、 前記少なくとも1つのセクタの サイズを調整すること を可能にするセクタ編集機能を提供するための手段と、 前記セクタ選択に基づいて、前記オーディオ信号インジケータに対応する前記捕捉されたオーディオに対してオーディオ動作を実行する ための手段。
- 14請求項2乃至12のうちのいずれか一項に従う方法を実行するための手段を備える、請求項13に記載の装置。
- 15請求項1乃至12のいずれか一項に従う方法を電子デバイスに実行させるためのコードを備える命令を 記憶した コンピュータ可読 記憶 媒体。
Independent claims15
447 paragraphs, as filed
Cross-reference of related applications [0001] This application is for US Provisional Patent Application No. 61 / 713,447 filed on October 12, 2012 for "SYSTEMS AND METHODS FOR MAPPING COORDINATES" and October 2012 for "SYSTEMS AND METHODS FOR MAPPING COORDINATES". US provisional patent application No. 61 / 714,212 filed on the 15th, US provisional application No. 61 / 624,181 filed on April 13, 2012, "SYSTEMS" with respect to "SYSTEMS, METHODS, AND APPARATUS FOR ESTIMATING DIRECTION OF ARRIVAL" , METHODS, AND APPARATUS FOR ESTIMATING DIRECTION OF ARRIVAL, US provisional application No. 61 / 642,954 filed on May 4, 2012, and "SYSTEMS, METHODS, AND APPARATUS FOR ESTIMATING DIRECTION OF ARRIVAL". With respect to "ARRIVAL", these priorities are claimed with respect to US Provisional Application No. 61 / 726,336 filed on November 14, 2012.
[0002] The present disclosure relates generally to electronic devices. More specifically, the present disclosure relates to systems and methods for displaying user interfaces.
[0003] In recent decades, the use of electronic devices has become commonplace. In particular, advances in electronic technology have reduced the cost of increasingly complex and useful electronic devices. Cost reductions and consumer demand have dramatically increased the use of electronic devices to the extent that they have become virtually ubiquitous in modern society. As the use of electronic devices has expanded, so has the demand for new and improved features of electronic devices. More specifically, electronic devices that perform functions faster, more efficiently, or of higher quality are often sought after.
[0004] Some electronic devices (eg, mobile phones, smartphones, computers, etc.) use audio or audio signals. These electronic devices may encode audio signals for storage or transmission. For example, a mobile phone uses a microphone to capture a user's voice or voice. Microphones convert acoustic signals into electronic signals. The electronic signal can then be formatted (eg, encoded) for transmission to another device (eg, a mobile phone, smartphone, computer, etc.), for playback, or for storage.
[0005] Noisy audio signals can pose certain challenges. For example, competing audio signals can reduce the quality of the desired audio signal. As can be seen from this description, systems and methods that improve the audio signal quality of electronic devices can be beneficial.
[0006] A method for displaying a user interface on an electronic device is described. This method involves presenting a user interface. The user interface includes a coordinate system. The coordinate system corresponds to physical coordinates based on sensor data. The method also includes providing a sector selection feature that allows the selection of at least one sector in the coordinate system. The method further includes providing a sector editing function that allows editing of at least one sector.
[0007] This method may include displaying the direction of at least one audio signal captured by at least one microphone. At least one audio signal may include an audio signal. This method may include displaying an icon corresponding to at least one audio signal. Displaying an icon may include displaying an icon for a target audio signal and / or an icon for an interfering audio signal.
[0008] This method may include passing the audio signal shown in at least one sector. This method may include attenuating an audio signal that is not shown in at least one sector. This method may include showing image data from one or more image sensors. This method may include passing an image based on one or more sectors.
[0009] This method may include displaying at least one touchpoint corresponding to at least one sector. The method may also include receiving a touch input corresponding to at least one touch point. This method may further include editing at least one sector based on touch input.
[0010] This method may include aligning at least a portion of the user interface with a reference plane. The reference plane can be horizontal. Matching at least part of the user interface can include mapping a 2D polar plot to a 3D display space.
[0011] Physical coordinates can be earth coordinates. Physical coordinates can represent a physical space that is independent of earth coordinates. The coordinate system can maintain a direction independent of the electronic device direction.
[0012] This method may include recognizing an audio signature. This method may also include examining the audio signature in the database. This method may further include obtaining the identification information corresponding to the audio signature. This method may additionally include displaying the identification information on the user interface. The identification information can be an image of the person corresponding to the audio signature. This method may include providing a fixed mode and an editable mode. This method may include padding selected sectors.
[0013] At least one of the sector selection and sector editing functions may operate on the basis of single-touch and / or multi-touch inputs. The sector selection function may allow simultaneous selection of multiple sectors. The sector editing function may allow the sector to be adjusted based on single-touch or multi-touch input. The sector selection function can be based on one or more swipe inputs. One or more swipe inputs can indicate a circular area. One or more swipe inputs can be a single swipe.
[0014] Electronic devices are also described. Electronic devices include displays. The display presents a user interface. The user interface includes a coordinate system. The coordinate system corresponds to physical coordinates based on sensor data. The display provides a sector selection feature that allows the selection of at least one sector in the coordinate system. The display also provides sector editing capabilities that allow editing of at least one sector.
[0015] A computer program product for displaying the user interface is also described. Computer program products include non-transitory tangible computer-readable media with instructions. The instruction includes a code for causing the electronic device to present the user interface. The user interface includes a coordinate system. The coordinate system corresponds to physical coordinates based on sensor data. The instructions also allow the electronic device to select at least one sector of the coordinate system. Includes code to provide the sector selection function. The instructions further include code to allow the electronic device to provide a sector editing function that allows editing of at least one sector.
[0016] A device for displaying the user interface is also described. The device includes means for presenting a user interface. The user interface includes a coordinate system. The coordinate system corresponds to physical coordinates based on sensor data. The device also includes means for providing a sector selection function that allows selection of at least one sector in the coordinate system. The device additionally includes means for providing a sector editing function that allows editing of at least one sector.
<figref num="1">[0017] Multiple diagrams of one multi-microphone handset.</figref><figref num="2A">[0018] A diagram showing a long-distance model of plane wave propagation for a microphone pair.</figref><figref num="2B">[0019] A diagram showing a plurality of microphone pairs in a linear array.</figref><figref num="3A">[0020] A diagram showing a plot of frequencies for an unwrapped phase delay band for four different directions of arrival (DOA).</figref><figref num="3B">[0021] A diagram showing a plot of frequencies for a wrapped phase delay band for the same four different directions of arrival (DOA), shown in Figure 3A.</figref><figref num="4A">[0022] The figure which shows an example of the phase delay value measured and the calculated value about two DOA candidates.</figref><figref num="4B">[0023] A diagram showing a linear array of microphones arranged along the top edge of a television screen.</figref><figref num="5A">[0024] The figure which shows an example which calculates the DOA difference with respect to a frame.</figref><figref num="5B">[0025] A diagram showing an example of calculating a DOA estimate.</figref><figref num="5C">[0026] A diagram showing an example of identifying DOA estimates for each frequency.</figref><figref num="6A">[0027] A diagram illustrating an example of using a calculated likelihood to identify the best microphone pair and the best DOA candidate for a given frequency.</figref><figref num="6B">[0028] The figure which shows an example of the likelihood calculation.</figref><figref num="7">[0029] The figure which shows an example of bias removal.</figref><figref num="8">[0030] A diagram showing another example of bias removal.</figref><figref num="9">[0031] A diagram showing an example of an anglogram plotting the source activity likelihood in DOA estimated for frames and frequencies.</figref><figref num="10A">[0032] The figure which shows an example of the speakerphone application.</figref><figref num="10B">[0033] A diagram showing the mapping of pairwise DOA estimates to a 360 ° range in the plane of a microphone array.</figref><figref num="11A">[0034] A diagram showing ambiguity within a DOA estimate.</figref><figref num="11B">The figure which shows the ambiguity in the DOA estimate.</figref><figref num="11C">[0035] A diagram showing the relationship between the observed DOA symbol and the quadrant of the xy plane.</figref><figref num="12A">[0036] A diagram showing an example in which the source is placed above the surface of the microphone.</figref><figref num="12B">The figure which shows an example which the source was placed on the upper part of the surface of a microphone.</figref><figref num="12C">The figure which shows an example which the source was placed on the upper part of the surface of a microphone.</figref><figref num="12D">The figure which shows an example which the source was placed on the upper part of the surface of a microphone.</figref><figref num="13A">[0037] The figure which shows an example of the microphone pair along the non-orthogonal axis.</figref><figref num="13B">[0038] A diagram illustrating an example of using the array of FIG. 13A to obtain DOA estimates for orthogonal x-axis and y-axis.</figref><figref num="13C">[0039] A diagram showing the relationship between the arrivals of parallel wave planes in different array microphones for two different DOA examples.</figref><figref num="14A">[0040] A diagram illustrating an example of pairwise normalized beam formation / null beam formation (BFNF) for two pairs of microphone arrays.</figref><figref num="14B">The figure which shows the example of the pairwise normalized beam formation / null beam formation (BFNF) for two pairs of microphone arrays.</figref><figref num="15A">[0041] A diagram showing two pairs of microphone arrays.</figref><figref num="15B">[0042] A diagram showing an example of pairwise normalized minimum variance distortionless respopnse (MVDR) BFNF.</figref><figref num="16A">[0043] Matrix A<sup>H</sup>The figure which shows an example of the pairwise BFNF about the frequency which A is not a bad condition.</figref><figref num="16B">[0044] A diagram illustrating an example of steering vectors.</figref><figref num="17">[0045] A flowchart of an example of a method of integrating source direction estimates as described herein.</figref><figref num="18">[0046] A diagram illustrating examples of practical results of DOA estimates, source discrimination, and source tracking as described herein.</figref><figref num="19">The figure which shows the example of the practical result of the DOA estimate, the source distinction, and the source tracking described herein.</figref><figref num="20">The figure which shows the example of the practical result of the DOA estimate, the source distinction, and the source tracking described herein.</figref><figref num="21A">The figure which shows the example of the practical result of the DOA estimate, the source distinction, and the source tracking described herein.</figref><figref num="21B">The figure which shows the example of the practical result of the DOA estimate, the source distinction, and the source tracking described herein.</figref><figref num="22">The figure which shows the example of the practical result of the DOA estimate, the source distinction, and the source tracking described herein.</figref><figref num="23">The figure which shows the example of the practical result of the DOA estimate, the source distinction, and the source tracking described herein.</figref><figref num="24">The figure which shows the example of the practical result of the DOA estimate, the source distinction, and the source tracking described herein.</figref><figref num="25">The figure which shows the example of the practical result of the DOA estimate, the source distinction, and the source tracking described herein.</figref><figref num="26">The figure which shows the example of the practical result of the DOA estimate, the source distinction, and the source tracking described herein.</figref><figref num="27">The figure which shows the example of the practical result of the DOA estimate, the source distinction, and the source tracking described herein.</figref><figref num="28A">The figure which shows the example of the practical result of the DOA estimate, the source distinction, and the source tracking described herein.</figref><figref num="28B">The figure which shows the example of the practical result of the DOA estimate, the source distinction, and the source tracking described herein.</figref><figref num="29">The figure which shows the example of the practical result of the DOA estimate, the source distinction, and the source tracking described herein.</figref><figref num="30">The figure which shows the example of the practical result of the DOA estimate, the source distinction, and the source tracking described herein.</figref><figref num="31">The figure which shows the example of the practical result of the DOA estimate, the source distinction, and the source tracking described herein.</figref><figref num="32A">[0047] A diagram showing a telephone design.</figref><figref num="32B">Diagram showing the use of such a design in various modes with corresponding visualization displays.</figref><figref num="32C">Diagram showing the use of such a design in various modes with corresponding visualization displays.</figref><figref num="32D">Diagram showing the use of such a design in various modes with corresponding visualization displays.</figref><figref num="33A">[0048] Flowchart for method M10 with general configuration.</figref><figref num="33B">[0049] The figure which shows the implementation form T12 of a task T10.</figref><figref num="33C">[0050] The figure which shows the implementation form T14 of a task T10.</figref><figref num="33D">[0051] A flowchart relating to the implementation form M20 of the method M10.</figref><figref num="34A">[0052] A flowchart relating to the implementation form M25 of the method M20.</figref><figref num="34B">[0053] A flowchart relating to the implementation form M30 of the method M10.</figref><figref num="34C">[0054] A flowchart relating to the implementation form M100 of the method M30.</figref><figref num="35A">[0055] A flowchart relating to the implementation form M110 of the method M100.</figref><figref num="35B">[0056] Block diagram of device A5 with general configuration.</figref><figref num="35C">[0057] A block diagram of a mounting embodiment A10 of the device A5.</figref><figref num="35D">[0058] The block diagram of the mounting form A15 of the apparatus A10.</figref><figref num="36A">[0059] A block diagram of the device MF5 according to a general configuration.</figref><figref num="36B">[0060] The block diagram of the mounting form MF10 of the apparatus MF5.</figref><figref num="36C">[0061] The block diagram of the mounting form MF15 of the apparatus MF10.</figref><figref num="37A">[0062] A diagram illustrating the use of a device to represent a three-dimensional direction coming into the plane of the device.</figref><figref num="37B">[0063] A diagram illustrating the intersection of cones of confusion representing the respective responses of a microphone array having a non-orthogonal axis to a point source located outside the plane of the axis.</figref><figref num="37C">[0064] A diagram illustrating the intersecting lines of the conical area of Figure 37B.</figref><figref num="38A">[0065] Block diagram of the audio preprocessing stage.</figref><figref num="38B">[0066] Block diagram of a 3-channel implementation of the audio preprocessing stage.</figref><figref num="39A">[0067] A block diagram of an implementation of an apparatus including means for indicating the direction of arrival.</figref><figref num="39B">[0068] A diagram showing an example of ambiguity caused by the one-dimensionality of DOA estimates from a linear array.</figref><figref num="39C">[0069] A diagram showing an example of a conical area that is confused.</figref><figref num="40">[0070] A diagram illustrating an example of source confusion in a speakerphone application in which three sources are arranged in different directions for a device having a linear microphone array.</figref><figref num="41A">[0071] A diagram showing a 2D microphone array containing two pairs of microphone pairs with orthogonal axes.</figref><figref num="41B">[0072] Flowchart of the general configuration method including tasks.</figref><figref num="41C">[0073] The figure which shows an example of the DOA estimate shown on the display.</figref><figref num="42A">[0074] A diagram showing an example of the correspondence between the 1-D estimate symbol and the corresponding quadrant of the plane defined by the array axis.</figref><figref num="42B">[0075] A diagram showing another example of the correspondence between the 1-D estimate symbol and the corresponding quadrant of the plane defined by the array axis.</figref><figref num="42C">[0076] Tuple (sign (θ)<sub>x</sub>), Sign (θ<sub>y</sub>)) Diagram showing the correspondence between the four values and the plane quadrant.</figref><figref num="42D">[0077] Diagram showing a 360 degree display with alternative mapping.</figref><figref num="43A">[0078] A diagram showing an example showing a more general case, similar to FIG. 41A, but with the source located at the top of the xy plane.</figref><figref num="43B">[0079] A diagram showing another example of a 2-D microphone array whose axis defines the xy plane and a source located on top of the xy plane.</figref><figref num="43C">[0080] A diagram showing an example of such a common case where a point sound source rises from a plane defined by an array axis.</figref><figref num="44A">[0081] (θ) to an angle in the array plane<sub>x</sub>, θ<sub>y</sub>) Derivation of the transformation.</figref><figref num="44B">(Θ) to an angle in the array plane<sub>x</sub>, θ<sub>y</sub>) Derivation of the transformation.</figref><figref num="44C">(Θ) to an angle in the array plane<sub>x</sub>, θ<sub>y</sub>) Derivation of the transformation.</figref><figref num="44D">(Θ) to an angle in the array plane<sub>x</sub>, θ<sub>y</sub>) Derivation of the transformation.</figref><figref num="44E">[0082] A diagram showing an example of projection p and elevation angle.</figref><figref num="45A">[0083] A diagram showing plots obtained by applying alternative mappings.</figref><figref num="45B">[0084] A diagram showing an example of the intersection of confused conical regions associated with the response of a linear microphone array with non-orthogonal axes x and r to a common point sound source.</figref><figref num="45C">[0085] The figure which shows the intersecting line of a conical area.</figref><figref num="46A">[0086] The figure which shows an example of the microphone array.</figref><figref num="46B">Observations from the array shown in Figure 46A (θ)<sub>x</sub>, θ<sub>y</sub>) Is a diagram showing an example of obtaining an estimate of the combined direction in the xy plane with respect to the orthogonal axes x and y.</figref><figref num="46C">[0088] A diagram showing an example of projection.</figref><figref num="46D">[0089] The figure which shows an example which determines a value from a dimension of a projection vector.</figref><figref num="46E">[0090] A diagram showing another example of determining a value from the dimension of a projection vector.</figref><figref num="47A">[0091] A flowchart of another general configuration method that includes an instance of the task.</figref><figref num="47B">[0092] A flowchart of an implementation form of a task including a subtask.</figref><figref num="47C">[0093] The figure which shows an example of the apparatus which provided the component for performing the function corresponding to FIG. 47A.</figref><figref num="47D">[0094] A diagram illustrating an example of an apparatus including means for performing the function corresponding to FIG. 47A.</figref><figref num="48A">[0095] A flowchart of an implementation of a method that includes a task.</figref><figref num="48B">[0096] A flowchart relating to an implementation of another method.</figref><figref num="49A">[0097] Flowchart of another implementation of the method.</figref><figref num="49B">[0098] The figure which shows an example of the display of the elevation angle estimated with respect to the display plane.</figref><figref num="49C">[0099] Flowchart of such an implementation of another method involving a task.</figref><figref num="50A">[00100] Figure showing an example of a display before rotation.</figref><figref num="50B">The figure which shows the example of the display after rotation.</figref><figref num="51A">[00101] A diagram showing another example of a display before rotation.</figref><figref num="51B">The figure which shows the other example of the display after rotation.</figref><figref num="52A">[00102] The figure which shows an example which the device coordinate system E is matched with the world world coordinate system.</figref><figref num="52B">[00103] A diagram showing an example in which a device is rotated and a matrix F corresponding to a direction.</figref><figref num="52C">[00104] The figure which shows the fluoroscopic mapping of the projection of DOA on the world reference plane on the display plane of a device.</figref><figref num="53A">[00105] The figure which shows an example of the mapped display of DOA projected on the world reference plane.</figref><figref num="53B">[00106] Flowchart of such another implementation of the method.</figref><figref num="53C">[00107] A diagram illustrating an example of an interface including a linear slider potentiometer, a rocker switch, and a wheel or knob.</figref><figref num="54A">[00108] The figure which shows an example of the user interface.</figref><figref num="54B">[00109] The figure which shows another example of a user interface.</figref><figref num="54C">[00110] The figure which shows another example of a user interface.</figref><figref num="55A">[00111] A diagram illustrating a further example in which a directional sensor is used to track the orientation of a device.</figref><figref num="55B">The figure which shows a further example in which a direction sensor is used to track the direction of a device.</figref><figref num="56">[00112] A block diagram illustrating a configuration of an electronic device in which a system and method for mapping source locations may be implemented.</figref><figref num="57">[00113] A flow diagram illustrating one configuration of a method for mapping source positions.</figref><figref num="58">[00114] A block diagram illustrating a more specific configuration of an electronic device in which a system and method for mapping source locations may be implemented.</figref><figref num="59">[00115] A flow diagram illustrating a more specific configuration of a method for mapping source locations.</figref><figref num="60">[00116] A flow chart illustrating one configuration of a method for performing operations based on mapping.</figref><figref num="61">[00117] A flow chart illustrating another configuration of methods for performing operations based on mapping.</figref><figref num="62">[00118] A block diagram illustrating a configuration of a user interface in which a system and method for displaying the user interface on an electronic device may be implemented.</figref><figref num="63">[00119] A flow diagram illustrating one configuration of a method for displaying a user interface on an electronic device.</figref><figref num="64">[00120] A block diagram illustrating a configuration of a user interface in which a system and method for displaying the user interface on an electronic device may be implemented.</figref><figref num="65">[00121] A flow diagram illustrating a more specific configuration of a method for displaying a user interface on an electronic device.</figref><figref num="66">[00122] The figure which shows the example of the user interface for displaying the directionality of at least one audio signal.</figref><figref num="67">[00123] A diagram illustrating another example of a user interface for displaying the orientation of at least one audio signal.</figref><figref num="68">[00124] A diagram illustrating another example of a user interface for displaying the orientation of at least one audio signal.</figref><figref num="69">[00125] A diagram illustrating another example of a user interface for displaying the orientation of at least one audio signal.</figref><figref num="70">[00126] A diagram showing another example of a user interface for displaying the orientation of at least one audio signal.</figref><figref num="71">[00127] The figure which shows an example of the sector selection function of a user interface.</figref><figref num="72">[00128] The figure which shows another example of the sector selection function of a user interface.</figref><figref num="73">[00129] The figure which shows another example of the sector selection function of a user interface.</figref><figref num="74">[00130] A diagram showing more examples of the sector selection function of the user interface.</figref><figref num="75">[00131] A diagram showing more examples of the sector selection function of the user interface.</figref><figref num="76">[00132] A flow diagram illustrating one configuration of a method for editing a sector.</figref><figref num="77">[00133] The figure which shows the example of the sector editing function of a user interface.</figref><figref num="78">[00134] A diagram showing more examples of the sector editing function of the user interface.</figref><figref num="79">[00135] A diagram showing more examples of the sector editing function of the user interface.</figref><figref num="80">[00136] A diagram showing more examples of the sector editing function of the user interface.</figref><figref num="81">[00137] A diagram showing more examples of the sector editing function of the user interface.</figref><figref num="82">[00138] A diagram showing an example of a user interface having a coordinate system oriented independently of the direction of an electronic device.</figref><figref num="83">[00139] A diagram illustrating another example of a user interface having a coordinate system oriented independently of the electronic device orientation.</figref><figref num="84">[00140] A diagram illustrating another example of a user interface having a coordinate system oriented independently of the electronic device orientation.</figref><figref num="85">[00141] A diagram illustrating another example of a user interface having a coordinate system oriented independently of the electronic device orientation.</figref><figref num="86">[00142] A diagram showing more examples of a user interface having a coordinate system oriented independently of the electronic device orientation.</figref><figref num="87">[00143] A diagram showing another example of a user interface having a coordinate system oriented independently of the electronic device orientation.</figref><figref num="88">[00144] A block diagram illustrating another configuration of a user interface in which a system and method for displaying the user interface on an electronic device may be implemented.</figref><figref num="89">[00145] A flow diagram illustrating another configuration of a method for displaying a user interface on an electronic device.</figref><figref num="90">[00146] The figure which shows an example of the user interface combined with the database.</figref><figref num="91">[00147] A flow diagram illustrating another configuration of a method for displaying a user interface on an electronic device.</figref><figref num="92">[00148] A block diagram illustrating a configuration of a wireless communication device in which a system and method for mapping source locations may be implemented.</figref><figref num="93">[00149] A diagram illustrating various components that may be used in an electronic device.</figref><figref num="94">[00150] A diagram showing another example of a user interface.</figref>
[00151] The 3rd Generation Partnership Project (3GPP) is a collaboration between groups of the Telecommunications Association that aims to define globally applicable 3rd Generation (3G) mobile phone specifications. .. 3GPP Long Term Evolution (LTE) is a 3GPP project aimed at improving the Universal Mobile Telecommunications System (UMTS) mobile phone standard. 3GPP can define specifications for next-generation mobile networks, mobile systems and mobile devices.
[00152] In some cases, the systems and methods disclosed herein are 3GPP Release 8 (Rel-8), 3GPP Release 9 (Rel-9), 3GPP Release 10 (Rel-10), LTE, LTE Advanced ( LTE-A), Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data Rates for GSM (EDGE) GSM® Evolution), Time Division Long-Term Evolution (TD-LTE), Time Division Syncronous Code Division Multiple Access (TD-SCDMA), Frequency Division Multiple Access (TD-SCDMA) Long Term Evolution (FDD-LTE: Frequency-Division Duplexing) Note that it may be described in terms of one or more specifications, such as Long-Term Evolution), UMTS, GSM EDGE Radio Access Network (GERAN), Global Positioning System (GPS), etc. I want to be. However, at least some of the concepts described herein may apply to other wireless communication systems. For example, the term electronic device may be used to refer to a user device (UE). In addition, the term base station may be used to refer to at least one of node B, evolutionary node B (eNB), home evolutionary node B (HeNB), and the like.
[00153] Unless expressly limited by its context, the term "signal" as used herein refers to the state of a memory location (or set of memory locations) represented on a wire, bus, or other transmission medium. Used to indicate any of its usual meanings, including. Unless expressly limited by its context, the term "generating" is used herein in any of its usual meanings, such as computing or generating otherwise. Is also used to indicate. Unless expressly limited by its context, the term "calculating" is used herein to compute, evaluate, estimate, and / or choose from multiple values. It is used to indicate any of its usual meanings, such as that. Unless explicitly limited by its context, the term "obtaining" refers to calculating, deriving, receiving (eg, from an external device), and / or (eg, storing). Used to indicate any of its usual meanings, such as searching (from an array of elements). Unless explicitly limited by its context, the term "selecting" identifies, directs, or applies at least one of two or more sets, and less than all. , And / or used to indicate any of its usual meanings, such as use. Unless explicitly limited by its context, the term "determining" refers to determining, establishing, concluding, calculating, selecting, And / or used to indicate any of its usual meanings, such as evaluating. The term "comprising" is used herein and in the claims. When used in, it does not exclude other elements or operations. The terms "based on" (such as "A is based on B") are (i) derived from "" (for example, "B is a precursor of A"), (ii) "at least ~ Based on (eg, "A is at least based on B"), and (iii) "equal to" (eg, "A is equal to B" or "A is equal to B", as appropriate in a particular context. Is used to indicate any of its usual meanings, including the case of "). Similarly, the term "in response to" is used to indicate any of its usual meanings, including "at least in response to." Unless otherwise specified, the terms "at least one of A, B and C" and "one or more of A, B and C" are referred to as "A and / or B and / or C." Is shown.
[00154] References to the "position" of a microphone in a multi-microphone audio sensing device indicate the position of the center of the acoustically sensitive surface of the microphone, unless otherwise specified by the context. The term "channel" is sometimes used to indicate a signal path, and at other times, to indicate a signal carried by such a path, depending on the particular context. Unless otherwise specified, the term "series" is used to refer to a sequence of two or more items. The term "logarithm" is used to refer to a base 10 logarithm, but the extension of such operations to other bases is within the scope of the present disclosure. The term "frequency component" refers to a sample (or "bin") of the frequency domain representation of a signal (eg, generated by a Fast Fourier Transform), or a subband of the signal (eg, a Burke or Mel scale subband). , Used to indicate one of a set of signal frequencies or frequency bands.
[00155] Unless otherwise specified, any disclosure of the operation of a device having a particular function is expressly intended to disclose a method having a similar function (and vice versa), and a particular configuration. Any disclosure of the operation of the device by is also expressly intended to disclose a method of similar configuration and vice versa. The term "configuration" may be used with respect to methods, devices, and / or systems, as indicated by its particular context. The terms "method," "process," "process," and "technique" are generally used interchangeably unless otherwise specified by the specific context. A "task" with multiple subtasks is also a method. The terms "device" and "device" are also generally used interchangeably, unless otherwise specified by a particular context. The terms "element" and "module" are commonly used to refer to parts of a larger configuration. Unless expressly limited by its context, the term "system" is used herein to refer to any of its usual meanings, including "a group of elements that interact to serve a common purpose." used.
[00156] Any embedding by reference to a part of a document is in any figure in which the definitions of terms or variables mentioned within that part appear elsewhere in the document and are referenced in the incorporated part. If so, it should also be understood that it incorporates such a definition. Unless first introduced by a definite article, the order-indicating terms used to modify a claim element (eg, "first," "second," "third," etc.) are different in their own right. It does not indicate the priority or order of the claiming elements for one, it merely distinguishes the claiming element from another claiming element with the same name (apart from the use of ordering terms). Unless expressly limited by its context, each of the terms "plurality" and "set" is used herein to indicate an integer quantity greater than one.
A. Systems, methods, and equipment for estimating arrival directions [00157] A method of processing a multi-channel signal is to obtain multiple phase differences, with respect to each of the multiple different frequency components of the multi-channel signal, the frequencies within each of the first pair of channels of the multi-channel signal. Includes calculating the difference between the phases of the components. The method also includes estimating the error between the candidate direction and the vector based on the plurality of phase differences for each of the plurality of candidate directions. The method also includes selecting the candidate direction corresponding to the lowest value of the estimated errors from the plurality of candidate directions. In this method, each of the first pair of channels is based on the signal produced by the corresponding one of the first pair of microphones, and at least one of the different frequency components is the first pair. Has a wavelength of less than twice the distance between the microphones.
[00158] Within the short-range and long-range regions of the emitted sound field, the wave planes can be assumed to be spherical and flat, respectively. A short distance can be defined as a region of space that is less than one wavelength away from a voice receiver (eg, a microphone array). Under this definition, the distance to the boundary of the region changes in inverse proportion to the frequency. For example, at frequencies of 200, 700, and 2000 hertz, the distance to one wavelength boundary is about 170, 49, and 17 centimeters, respectively. Instead, the short-range / long-range boundary is at a specific distance from the microphone array (for example, 50 centimeters from the array microphone or array centroid, or 1 meter or 1.5 meters from the array microphone or array centroid). It may be useful to consider it as.
[00159] Next, various configurations will be described with reference to the figures, in which case similar reference numbers may indicate functionally similar elements. The systems and methods generally described and illustrated herein can be configured and designed in a wide variety of different configurations. Therefore, the following more detailed description of some of the configurations shown in the figure is not limiting the scope of claims, but merely represents the system and method. One or more functions and / or elements shown in the figure may be combined with at least one function and / or element shown in at least one other figure.
[00160] Figure 1 shows the first microphone pairs MV10-1, MV10-3 whose axes are left and right on the front of the device, and the first microphones whose axes are front and back (ie, orthogonal to the front). An example of a multi-microphone handset H100 (eg, a multi-microphone device) including two microphone pairs MV10-1 and MV10-2 is shown. Such a configuration can be used to determine when the user is speaking in front of the device (eg, browse-talk mode). Front and back pairs can be used to resolve ambiguities between the front and back directions that the left and right pairs generally cannot resolve on their own. In some implementations, the handset H100 may include one or more loudspeakers LS10, L20L, LS20R, touch screen TS10, lens L10, and / or one or more additional microphones ME10, MR10.
[00161] In addition to the handset shown in FIG. 1, another example of an audio sensing device that includes a multi-microphone array and can be implemented to perform the methods as described herein is a portable computing device. (For example, laptop computers, notebook computers, netbook computers, ultra-portable computers, tablet computers, mobile internet devices, smart books, smartphones, etc.), audio or video conferencing devices, and display screens (for example, computer monitors, televisions) There is John Set).
[00162] The device shown in FIG. 1 measures the difference (eg, phase difference) between microphone channels for each frequency bin in order to obtain the directional reading, and the estimated directional is with all bins. It may be configured to determine the direction of arrival (DOA) of the source signal by averaging the directional readings of all bins to determine if they are consistent. The range of frequency bins that may be available for tracking is generally spatial aliasing with respect to the microphone pair. Constraint by frequency). This upper limit can be defined as the frequency at which the wavelength of the signal is twice the distance d between the microphones. Such methods cannot support accurate tracking of source DOA greater than 1 meter and may generally only support low DOA resolution. Moreover, relying on front and back pairs to resolve ambiguity is a significant constraint on the microphone placement shape, as placing the device on a surface effectively blocks the front or back microphone. Can be. Such schemes also generally use only one fixed pair for tracking.
[00163] A multi-microphone device can be arbitrarily placed (for example, on a table for a conference call, on a car seat, etc.) to track and / or enhance the voice of an individual speaker. It may be desirable to provide a general purpose speakerphone application application. Such a scheme may be able to address any target speaker position with respect to any direction of available microphones. It may also be desirable for such a scheme to provide simultaneous multi-speaker tracking / separation capabilities. Unfortunately, the current state-of-the-art technology is the single microphone system.
[00164] It may be desired to support source tracking in long range applications, which can be used to provide solutions for tracking sources in unknown directions and large distances with respect to multi-microphone devices. Multi-microphone devices in such applications may include arrays mounted in televisions or set-top boxes, which may be used to support telephone communications. Examples include arrays of Kinect devices (Microsoft Corp., Redmond, WA), as well as arrays from Skype (Microsoft Skype Division) and Samsung Electronics (Seoul, KR). In addition to large source-to-device distances, such applications also generally suffer from poor signal-to-noise ratio (SINR) and room reverberation.
[00165] It is a challenge to provide a method of estimating the three-dimensional direction of arrival (DOA) for each frame of an audio signal for multiple simultaneous sound events that are robust enough under background noise and reverberation. Robustness can be obtained by maximizing the number of reliable frequency bins. It may be desired for such a method to be suitable for an arbitrarily shaped microphone array shape such that certain restrictions on the microphone shape can be circumvented. The pairwise 1D scheme described herein can be appropriately incorporated into any shape.
[00166] The systems and methods disclosed herein may be implemented with respect to such general purpose speakerphone application applications or ranged applications. Such a scheme can be implemented to operate without microphone placement constraints. Such schemes also use frequency bins that are available up to the Nyquist frequency (for example, by supporting the use of microphone pairs with larger microphone-to-microphone distances) and down to lower frequencies. Can be performed to track the source. Rather than limiting to a single pair for tracking, such a scheme can be implemented to select the best pair among all available pairs. Such schemes can be used to support source tracking and provide higher DOA resolution, even in long-distance scenarios at distances from 3 meters to 5 meters and above. Other potential features include getting an accurate 2D representation of the active source. For best results, each source is sparse broadband audio Source), and it may be desirable that each frequency bin is largely dominated by only one source.
[00167] FIG. 33A shows a flow chart for method M10 with a general configuration involving tasks T10, T20, and T30. Task T10 calculates the difference between channel pairs of multi-channel signals (eg, based on the signal generated by the corresponding microphone for each channel). For each of the K candidate directions, task T20 calculates the error in the corresponding direction based on the calculated difference. Based on the error of K directions, task T30 selects candidate directions.
[00168] Method M10 may be configured to process a multichannel signal as a series of segments. Typical segment lengths range from about 5 or 10 ms to about 40 or 50 ms, and segments may or may not overlap (for example, 25% or 50% overlap with adjacent segments). In one particular example, the multi-channel signal is divided into a series of non-overlapping segments, or "frames," each having a length of 10 milliseconds. In another particular example, each frame has a length of 20 milliseconds. Also, the segment processed by method M10 can be a segment of a larger segment (ie, a "subframe") processed by a different operation, and vice versa.
[00169] Examples of differences between channels are gain differences or gain ratios, arrival time differences, and phase differences. For example, task T10 may be performed to calculate the difference between a pair of channels as the difference or ratio (eg, magnitude or energy difference) between the corresponding gain values of the channels. FIG. 33B shows an implementation form T12 of the task T10.
[00170] Task T12 may be in the time domain (eg, for each of multiple subbands of the signal) or in the time domain (eg, Fast Fourier Transform (FFT), Discrete Cosine Transform (DCT), or Modified DCT (MDCT) domain, etc. It can be performed to calculate the gain measurements of a segment of a multichannel signal in the frequency domain (for each of the multiple frequency components of the signal in the transform domain). Examples of such gain measurements are, but not limited to, the magnitude of the sum (eg, the sum of the absolute values of the sample values), the magnitude of the average (eg, per sample), the root mean square. There is a root mean square (RMS) amplitude, median magnitude, maximum amplitude, peak energy, total energy (eg sum of squares of sample values), and average energy (eg per sample).
[00171] It may be desirable to calibrate the responses of the two microphone channels against each other in order to obtain accurate results using the gain difference technique. It may be desirable to apply a low frequency filter to a multichannel signal so that the calculation of gain measurements is limited to the audio frequency components of the multichannel signal.
[00172] Task T12 is the ratio between the gain measurements in the linear region as the difference between the corresponding gain measurements (eg, in decibels) for each channel in the log region, or equally. Can be performed to calculate as. For calibrated microphone pairs, a gain difference of zero can be taken to indicate that the source is equidistant from each microphone (ie, placed laterally to the pair), and the source is one microphone. A gain difference with a large positive value can be taken to indicate that it is closer (ie, placed one of the pairs vertically) and the source is closer to the other microphone. A gain difference with a large negative value can be taken to indicate (ie, arranged vertically on the other side of the pair).
In another example, task T10 from Figure 33A performs cross-correlation on channels to determine the difference (eg, calculates the arrival time difference based on the delay between channels in a multichannel signal). Can be given to).
[00174] In a further example, task T10 is performed to calculate the difference between pairs of channels as the difference between the phases of each channel (eg, at a particular frequency component of the signal). FIG. 33C shows such an implementation T14 of task T10. As discussed below, such calculations can be performed for each of the multiple frequency components.
[00175] For signals received by a pair of microphones directly from a point source in a particular direction of arrival (DOA) with respect to the axis of the pair of microphones, the phase delay is different for each frequency component and is the spacing between the microphones. Also depends. The observed value of the phase delay at a particular frequency component (ie, "bin") can be calculated as the inverse tangent (also called arc tangent) of the ratio of the imaginary term of the complex FFT coefficient to the real term of the complex FFT coefficient.
[00176] As shown in FIG. 2A, the phase delay value Δφ for the source S01 for at least one microphone MC10, MC20 at a particular frequency f.<sub>f</sub>Is<maths num="1"><img id="000002" he="28" wi="76" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
As long distance (ie, plane wave) assumptions can relate to the source DOA, where d is the distance (meters) between the microphones MC10 and MC20 and θ is the angle of arrival in the direction orthogonal to the array axis. (Radian), f indicates frequency (Hz), c indicates sound velocity (meters per second). As described below, the DOA estimation principle described herein can be extended to multiple microphone pairs within a linear array (eg, shown in FIG. 2B). For an ideal single-point sound source with no reverberation, the frequency Δφ<sub>f</sub>The ratio of phase delay to to is the same across all frequencies<maths num="2"><img id="000003" he="29" wi="70" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Will have. As discussed in more detail below, DOA, θ for a pair of microphones is a one-dimensional measurement that defines the surface of the conical region in space (for example, the axis of the conical region is the axis of the array). is there.
[00177] Such a scheme is generally limited in practice by the spatial aliasing frequency with respect to the microphone pair, which can be defined as the frequency at which the wavelength of the signal is twice the distance d between the microphones. Spatial aliasing causes phase wrapping, which imposes an upper limit on the range of frequencies that can be used to provide reliable phase delay measurements for a particular microphone pair.
[00178] Figure 3A shows a frequency plot for the non-folding phase delay for four different DOAs D10, D20, D30, D40. Figure 3B shows frequency plots for folded phase delays for the same DOA D10, D20, D30, D40, in which case the initial part of each plot is shown in bold (ie, until the first wrapping occurs). Is done. Attempting to extend the useful frequency range of phase delay measurements by not folding back the measured phase is generally unreliable.
[00179] Task T20 may be performed to calculate the directional error in terms of phase difference. For example, task T20 determines the directional error at frequency f for each of the inventory of K DOA candidates, the square difference between the observed phase difference and the phase difference corresponding to the DOA candidate eph_f_k = (Δφob_f-Δφk_f). )<sup>2</sup>(Or, it can be performed to calculate as the absolute difference eph_f_k = | Δφob_f-Δφk_f |), where 1 <k <K.
[00180] Instead of not folding the phase, the proposed method compares the measured (eg, folded) phase delay with the precomputed value of the folded phase delay for each of the DOA candidate's inventory. .. Figure 4A shows such an example including a measured phase delay value MPD10 (noisy) and a frequency plot (solid and dashed) for the angles of the phase delay values PD10 and PD20 for the two DOA candidates in the inventory. , In this case, the phase wraps in the range pi to -pi. Then each DOA candidate θ<sub>i</sub>By calculating the corresponding directional errors with respect to and identifying the DOA candidate value corresponding to the lowest of these directional errors, the DOA candidate that best matches the observed signal can be determined. The error in such a direction is, for example, the phase delay value Δφ for the kth DOA candidate.<sub>k_f</sub>The observed phase delay value Δφ<sub>ob_f</sub>Error between<sub>ph_k</sub>Can be calculated as. In one example, the error e<sub>ph_k</sub>With respect to the desired range of frequency components or other set F<maths num="3"><img id="000004" he="29" wi="90" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
That is, the sum of the square differences between the observed phase delay value and the candidate phase delay value for F.<maths num="4"><img id="000005" he="29" wi="112" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Expressed as. Each DOA candidate, θ<sub>k</sub>Phase delay value ΔΦ<sub>k_f</sub>Is calculated before run time (eg, during design or manufacture) and retrieved from storage during device use, according to known values of c and d, as well as the desired range f of the frequency component. It is possible. Such pre-computed inventory has a range and resolution of desired angles (eg, uniform resolution, such as 1 degree, 2 degrees, 5 degrees, 6 degrees, 10 degrees, or 12 degrees, or desired non-uniform resolution). , And may be configured to support the desired frequency range and resolution (which may also be uniform or non-uniform).
[00181] Directional errors across as many frequency bins as possible (eg e<sub>ph_f</sub>, E<sub>ph_k</sub>) May be desired. For example, it may be desirable for the error calculation to include terms from frequency bins above the spatial aliasing frequency. In practical applications, the maximum frequency bin can include other memory available, computational complexity, strong reflections from rigid bodies at high frequencies (eg objects in its environment, device housings), etc. It can be limited by factors.
[00182] Audio signals are generally sparse in the time frequency domain. If the sources are independent in the frequency domain, the two sources can be tracked at the same time. If the sources are independent in the time domain, the two sources can be tracked at the same frequency. It may be desirable for the array to include several microphones that are at least equal to the number of different source directions that will be distinguished at any given time. The microphone can be omnidirectional (as is common with devices such as cellular phones or dedicated conferencing devices) or directional (as is common with devices such as set-top boxes).
[00183] Such multi-channel processing is generally applicable, for example, to source tracking for speakerphone applications. Such techniques can be used to calculate DOA estimates for frames of received multichannel signals. Such a scheme can calculate the error for each candidate angle with respect to the observed angle, as indicated by the phase delay, in each frequency bin. The target angle in that frequency bin is a candidate with the lowest error. In one example, the errors are then summed over the frequency bins to get possible measurements for that candidate. In another example, one or more of the most frequently occurring target DOA candidates across all frequency bins are identified as one DOA estimate (or multiple DOA estimates) for a given frame. ..
[00184] Such a method may be applied to obtain simultaneous tracking results (eg, with a delay of less than one frame). The delay depends on the FFT size and the degree of duplication. For example, for a 512 point FFT with 50% overlap and a sampling frequency of 16 kHz (kHz), the resulting 256 sample delay corresponds to 16 milliseconds. Such methods can generally be used to support source orientation distinctions up to a source array distance of up to 2-3 meters, or even up to 5 meters.
[00185] Errors may also be considered variances (eg, to the extent that individual errors deviate from the expected values). Converting the time domain received signal into the frequency domain (eg, by applying an FFT) has the effect of averaging the spectra in each bin. This averaging becomes even more apparent when subband representations (eg, the Mel or Bark scale) are used. In addition, it may be desirable to perform time domain smoothing on DOA estimates (by applying a recursive smoother, such as a first-order infinite impulse response filter). Computation of error calculation operations (for example, by using a search strategy such as a binary tree and / or by applying known information such as DOA candidate selection from one or more previous frames) It may be desirable to reduce the above complexity.
Information in the [00186] direction can be measured with respect to phase delay, but it is generally desired to obtain results that indicate source DOA. As a result, it may be desirable to perform task T20 to calculate the directional error at frequency f for each of the inventory of K DOA candidates in terms of DOA rather than phase delay.
[00187] The directional error equation for DOA is a folded delay at frequency f (eg, for example).<maths num="5"><img id="000006" he="27" wi="104" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Etc., DOA of signal, function Ψ of θ<sub>f_wr</sub>As observed phase delay Δφ<sub>ob_f</sub>) Can be derived. This equation, except for near discontinuities due to phase wrapping,<maths num="6"><img id="000007" he="30" wi="63" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Equal to the corresponding expression for the non-folding phase delay, depending on the DOA function. Directional error e<sub>ph_f_k</sub>Then e<sub>ph_f_k</sub>= | Ψ<sub>f_wr</sub>(θ<sub>ob</sub>)-Ψ<sub>f_wr</sub>(θ<sub>k</sub>) | | Ψ<sub>f_un</sub>(θ<sub>ob</sub>)-Ψ<sub>f_un</sub>(θ<sub>k</sub>) | Or e<sub>ph_f_k</sub>= (Ψ<sub>f_wr</sub>(θ<sub>ob</sub>)-Ψ<sub>f_wr</sub>(θ<sub>k</sub>))<sup>2</sup> (Ψ)<sub>f_un</sub>(θ<sub>ob</sub>)-Ψ<sub>f_un</sub>(θ<sub>k</sub>))<sup>2</sup>Etc., observed DOA, θ<sub>ob</sub>And candidate DOA, θ<sub>k</sub>In the equation, the difference between the phase delay observed at frequency f and the candidate phase delay is the DOA, θ observed at frequency f.<sub>ob_f</sub>And candidate DOA, θ<sub>k</sub>Regarding<maths num="7"><img id="000008" he="29" wi="149" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Expressed as. Directional error across F e<sub>ph_k</sub>Then then<maths num="8"><img id="000009" he="27" wi="159" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
As observed DOAθ<sub>ob</sub>And candidate DOA, θ<sub>k</sub>Can be expressed with respect to.
[00188] We have observed DOAθ at frequency f.<sub>ob_f</sub>And DOA candidates<maths num="9"><img id="000010" he="30" wi="137" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
The following first-order approximation used to obtain the equation for the difference between: <maths num="10"><img id="000011" he="30" wi="117" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
To obtain, perform a Taylor series expansion on this result. This equation has a phase delay<maths num="11"><img id="000012" he="39" wi="159" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths><maths num="12"><img id="000013" he="42" wi="150" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Not DOA (e<sub>DOA_f_k,</sub>e<sub>DOA_k</sub>To express the directional error with respect to), it can be used (eg, in task T20) with the assumed equivalence of the observed folded phase delay for the non-wrapped phase delay, and the equation During,<maths num="13"><img id="000014" he="27" wi="99" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
The value of<maths num="14"><img id="000015" he="28" wi="54" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Is defined as.
[00189] In order to avoid division by zero in the vertical direction (θ = +/- 90 °), instead, to perform such an expansion using a quadratic approximation, as follows: , It may be desirable to carry out task T20.<maths num="15"><img id="000016" he="38" wi="111" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
In the formula, A = (πfd sin θ<sub>k</sub>) / C, B = (-2πfd cos θ<sub>k</sub>) / C, and C =-(Ψ<sub>f_un</sub>(θ<sub>ob</sub>)-Ψ<sub>f_un</sub>(θ<sub>k</sub>)). As in the first-order example above, this equation does not wrap to represent the directional error with respect to DOA, depending on the function of the observed folded phase delay value and the candidate folded phase delay value. It can be used with the assumed equivalence of the observed folded phase delay for the phase delay.
[00190] FIGS. 5A-5C show a plurality of frames 502. As shown in Figure 5A, the directional error based on the difference between the observed DOA and the candidate DOA for a given frame of the received signal is the received microphone signal (eg, f F). At each of the plurality of frequencies f of, and a plurality of DOA candidates θ<sub>k</sub>For each of these, it can be calculated in such a manner (eg, by task T20). (Also known as first-order IIR or recursive filter) e<sub>s</sub>(n) = βe<sub>s</sub>It may be desirable to perform task T20 to perform a time smoothing operation on the error e in each direction according to an equation such as (n + 1) + (1-β) e (n), where e<sub>s</sub>(n-1) indicates the error in the smoothed direction with respect to the previous frame, and e<sub>s</sub>(n) indicates the current unsmoothed value of the directional error, e<sub>s</sub>(n) indicates the current smoothing value of the directional error, and β is a smoothing coefficient whose value can be selected from the range of zero (no smoothing) to 1 (no update). Typical values for the smoothing factor β include 0.1, 0.2, 0.25, 0.3, 0.4, and 0.5. It is typical, but not always necessary, for such an implementation of task T20 to use the same value of β to smooth out the error in the direction corresponding to the different frequency components. Similarly, it is typical, but not always necessary, for such an implementation of task T20 to use the same value of β to smooth out errors in directions corresponding to different candidate directions. As shown in Figure 5B, the DOA estimate for a given frame is a directional error (eg, e).<sub>ph_k</sub>Or e<sub>DOA_k</sub>) Can be determined by summing the square differences for each candidate across all frequency bins in the frame and selecting the DOA candidate with the lowest error. Alternatively, as shown in Figure 5C, such differences can be used to identify the most matching (ie, minimum square difference) DOA candidate at each frequency. DOA estimates for frames can then be determined as the most frequent DOA across all frequency bins.
[00191] Based on the direction error, task T30 selects candidate directions for frequency components. For example, task T30 may be performed to select a candidate direction associated with the lowest directional error among the K directional errors generated by task T20. In another example, task T30 is performed to calculate the likelihood based on the error in each direction and select the candidate direction associated with the highest likelihood.
As shown in FIG. 6B, error term 604 can be calculated for each of the set F of frequencies for each candidate angle 606I and each frame 608k. It may be desirable to indicate the likelihood of source activity with respect to the calculated DOA difference or error term 604. An example of such a likelihood L is for a particular frame, frequency, and angle.<maths num="16"><img id="000017" he="28" wi="93" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Can be expressed as.
[00193] For this equation, a very good match at a particular frequency can have the corresponding likelihood dominant over all other likelihoods. In order to reduce this sensitivity, it may be desirable to include the regulatory term λ, as in the following equation.<maths num="17"><img id="000018" he="28" wi="100" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
[00194] Audio tends to be sparse in both time and frequency so that the sum for set F of frequencies can include results from bins dominated by noise. It may be desirable to include the bias term β, as in the equation below.<maths num="18"><img id="000019" he="34" wi="76" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Bias terms that can vary over frequency and / or time can be based on a hypothesized noise distribution (eg Gauss). In addition, or instead, the bias term can be based on an initial noise estimate (eg, from a noise-only initial frame). In addition, or instead, the bias term can be dynamically updated based on information from noise-only frames, for example, as indicated by the voice activity detection module. 7 and 8 show examples of likelihood plots before and after bias removal, respectively. In FIG. 7, a signal frame number 710, an arrival angle 712, and an amplitude 714 are illustrated. Similarly, in FIG. 8, the signal frame number 810, the arrival angle 812, and the amplitude 814 are illustrated.
[00195] Frequency-specific likelihood results are frame-by-frame DOA estimates that are robust against noise and reverberation, as only target-dominant frequency bins contribute to the estimates.<maths num="19"><img id="000020" he="27" wi="99" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Can be projected onto a (frame, angle) plane (as shown in Figure 8) to obtain. In this sum, terms with large errors have values close to zero and are therefore less important to the estimates. If the sound source in one direction predominates among several frequency bins, the error value in those frequency bins can be close to zero with respect to that angle. Also, if sound sources in different directions dominate among the other frequency bins, the error values in those other frequency bins can be closer to zero with respect to the other angles.
[00196] Likelihood results are also shown in the lower panel 918 of FIG. 9 to show likelihood information per frequency bin based on directional membership (eg, for voice activity detection). Can be projected onto a (frame, frequency) plane. The bottom panel 918 shows the corresponding likelihood for the estimated DOA (eg, for each frequency and frame).<maths num="20"><img id="000021" he="26" wi="90" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
) Is shown. This likelihood can be used to indicate the likelihood of voice activity. In addition, or instead, such information is to support time and / or frequency selective masking of the received signal, for example, by classifying the frame and / or frequency components according to their direction of arrival. Can be used for.
[00197] The anglogram representation is similar to the spectrogram representation, as shown in the lower panel 918 of FIG. As shown in the top panel 916 of FIG. 9, the spectrogram can be obtained by plotting the magnitude of each frequency component in each frame. Anglogram can be obtained by plotting the likelihood of the current DOA candidate at each frequency in each frame.
[00198] FIG. 33D shows a flowchart of implementation M20 of method M10 including tasks T100, T200, and T300. Such a method can be used, for example, to select a candidate arrival direction of the source signal based on information from a channel pair of the multichannel signal for each of the multiple F frequency components of the multichannel signal. For each of the multiple F frequency components, task T100 calculates the difference between the pairs of channels. Task T100 may be performed, for example, to perform a corresponding instance of task T10 (eg, task T12 or T14) for each of the multiple F frequency components.
[00199] For each of the multiple F frequency components, task T200 calculates errors in multiple directions. Task T200 may be performed to calculate the error in K directions for each frequency component. For example, task T200 may be performed to execute a corresponding instance of task T20 for each of the multiple F frequency components. Alternatively, the task T200 calculates an error in K directions for each of one or more of the frequency components and for each of the different one or more of the frequency components ( For example, it can be performed to calculate errors in different numbers of directions (more than or less than K).
[00200] For each of the multiple F frequency components, task T300 selects a candidate direction. Task T300 may be performed to execute a corresponding instance of task T30 for each of the multiple F frequency components.
[00201] The energy spectrum of uttered speech (eg, vowels) tends to have local peaks at the harmonics of the pitch frequency. On the other hand, the energy spectrum of background noise tends to be relatively unstructured. As a result, the components of the input channel at the harmonics of the pitch frequency can be expected to have a higher signal-to-noise ratio (SNR) than the other components. It may be desirable to configure method M20 to consider only the frequency components that correspond to multiples of the estimated pitch frequency.
[00202] Typical pitch frequencies range from about 70-100Hz for male speakers and about 150-200Hz for female speakers. The current pitch frequency can be estimated by calculating the pitch period as the distance between adjacent pitch peaks (eg, within the primary microphone channel). The sample in the input channel measures its energy (eg, based on the ratio between the sample energy and the frame average energy) and / or how well the sample neighborhood is with a similar neighborhood of the known pitch peak. It can be identified as a pitch peak based on a measurement of whether it correlates. The pitch estimation procedure can be found, for example, in Section 4.6.3 (pages 4-44 to 4-49) of the EVRC (Enhanced Variable Rate Codec) document C.S0014-C available online at www.3gpp.org. Explained. Current estimates of pitch frequency (in the form of pitch periods, or "pitch lag" estimates) are generally voice-coded and / or decoded (eg, code-excited linear predictions (CELPs)). It will already be available in applications that include prediction) and voice communications using codecs that include pitch estimates, such as prototype waveform interpolation (PWI).
[00203] For example, it may be desirable to configure task T100 so that at least 25%, 50%, or 75% of the calculated channel difference (eg, phase difference) corresponds to a multiple of the estimated pitch frequency. .. The same principle can be applied to other desired harmonic signals. In a related method, task T100 is performed to calculate the phase difference for each of the frequency components of at least the subbands of the channel pair, and task T200 is based solely on the phase difference corresponding to a multiple of the estimated pitch frequency. It is carried out to calculate the directional error.
[00204] FIG. 34A shows a flowchart of implementation M25 of method M20 including task T400. Such a method can be used, for example, to indicate the direction of arrival of a source signal based on information from a channel pair of multichannel signals. Based on the selection of F candidate directions generated by task T300, task T400 indicates the direction of arrival. For example, task T400 may be performed to indicate the most frequently selected candidate direction among the F candidate directions as the arrival direction. In the case where the source signals are independent in frequency, task T400 may be performed to indicate more than one direction of arrival (eg, to indicate direction for each of more than one source). Method M25 can be repeated over time to indicate one or more directions of arrival for each of the sequences of frames of a multichannel signal.
[00205] Spatial aliasing begins at low frequencies for microphone pairs with wide spacing, so such pairs are generally not suitable for high frequencies. However, the DOA estimation scheme described herein allows the use of phase delay measurements up to the Nyquist frequency (ie, half the sampling rate) above the frequency at which phase wrapping begins. By relaxing spatial aliasing constraints, such schemes allow the use of microphone pairs with larger inter-microphone spacing. Arrays with large microphone distances generally provide better orientation at lower frequencies than arrays with smaller microphone distances, so using larger arrays is generally a useful phase delay measurement. The range of is similarly extended to lower frequencies.
[00206] The DOA estimation principle described herein can be extended to multiple microphone pairs MC10a, MC10b, MC10c in a linear array (eg, shown in FIG. 2B). An example of such an application for long-range scenarios is a linear array of microphones MC10a-e placed along the edges of a television TV 10 or other large video display screen (eg, shown in Figure 4B). As in the examples of FIGS. 2B and 4B, it may be desirable to configure such an array so that there is a non-uniform (eg, logarithmic) spacing between the microphones.
[00207] For long-range sources, multiple microphone pairs in a linear array will have essentially the same DOA. Therefore, one option is to estimate DOA as the average of DOA estimates from two or more pairs in the array. However, the averaging scheme can even be affected by an inconsistency in one of the pairs, which can reduce the accuracy of DOA estimation. Alternatively, the best microphone pair for each frequency (eg, the lowest error e at that frequency) out of two or more pairs of microphones in the array so that different microphone pairs can be selected for different frequency bands.<sub>i</sub>It may be desirable to select a pair) that presents. The error is large at the spatial aliasing frequency of the microphone pair. As a result, such schemes tend to automatically avoid microphone pairs when the frequency is close to their wrapping frequency, and thus avoid the associated uncertainty in DOA estimates. Pairs with shorter distances between microphones for higher frequency bins generally provide better estimates and can be automatically advantageous, while for lower frequency bins. Pairs with longer distances between microphones will generally provide better estimates and can be automatically advantageous. In the example of four microphones shown in Figure 2B, six different pairs of microphones (ie,<maths num="21"><img id="000022" he="29" wi="65" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Is possible.
[00208] In one example, the best pair for each axis is selected by calculating the PxI value for each frequency f, where P is the number of pairs, I is the size of the inventory, and each value e.<sub>pi</sub>Is the observed angle θ (with respect to p and frequency f)<sub>pf</sub>And candidate angle θ<sub>if</sub>Is the absolute difference in squares between. Minimum error value e for each frequency f<sub>pi</sub>The pair p corresponding to is selected. This error value is also the best DOA candidate θ at frequency f (as shown in Figure 6A).<sub>if</sub>Is shown.
[00209] FIG. 34B shows a flowchart regarding the implementation form M30 of the method M10 including the implementation form T150 of the task T10 and the implementation form T250 of the task T20. Method M30 can be used, for example, to indicate a candidate direction for the frequency component of a multichannel signal (eg, in a particular frame).
[00210] For each of the plurality of channels vs. P of the multichannel signal, task T250 calculates errors in multiple directions. Task T250 may be performed to calculate the error in K directions for each channel pair. For example, task T250 may be performed to execute a corresponding instance of task T20 for each of a plurality of channels vs. P. Alternatively, task T250 calculates an error in K directions for each of one or more of the channel pairs and for each of the different one or more of the channel pairs ( For example, it can be performed to calculate an error in a different number of directions (more than or less than K).
[00211] Method M30 also includes task T35, which selects candidate directions based on errors in multiple directions. For example, task T35 may be performed to select a candidate direction that corresponds to the lowest directional error among the directional errors.
[00212] FIG. 34C shows a flowchart of the implementation form M100 of the method M30 including the implementation form T170 of the tasks T100 and T150, the implementation form T270 of the tasks T200 and T250, and the implementation form T350 of the task T35. Method M100 can be used, for example, to select candidate directions for each of the multiple F frequency components of a multichannel signal (eg, in a particular frame).
[00213] For each of the multiple F frequency components, task T170 calculates a plurality of differences P, in which case each of the plurality of differences P corresponds to a different channel pair of multichannel signals. And the difference between the 21 channels (eg, gain-based or phase-based). For each of the multiple F frequency components, task T270 calculates errors in multiple directions for each of the multiple pairs of P. For example, task T270 may be performed to calculate the error of K directions for each of the P pairs for each of the frequency components, or the total P × K directions of error for each frequency component. The task T350 selects the corresponding candidate direction for each of the multiple F frequency components and based on the error in the corresponding directions.
[00214] FIG. 35A shows a flowchart relating to the implementation form M110 of the method M100. The implementation M110 may include tasks T170, T270, T350, and T400, which may be examples of the corresponding elements described with respect to at least one of FIGS. 34A and 34C.
[00215] FIG. 35B shows a block diagram of device A5 in a general configuration, including an error calculator 200 and a selector 300. The error calculator 200 describes for each of the calculated differences between the channel pairs of the multichannel signal and in each of the plurality of K candidate directions (as described herein with reference to the embodiment of task T20). It is configured to calculate the error in the corresponding direction based on the calculated difference. The selector 300 is configured to select a candidate direction based on an error in the corresponding direction (for example, as described herein with reference to an implementation of task T30).
[00216] FIG. 35C shows a block diagram of an implementation form A10 of the apparatus A5 including the difference calculator 100. Device A10 may be implemented, for example, to perform instances of methods M10, M20, M30, and / or M100 as described herein. The computer 100 calculates the difference between channel pairs of a multichannel signal (eg, gain-based or phase-based difference) (eg, as described herein with reference to the embodiment of task T10). It is configured as follows. Computer 100 can be implemented, for example, to calculate such differences for each of the multiple F frequency components of a multichannel signal. In such cases, computer 100 also applies a subband filter bank to the signal and / or before calculating the difference, the frequency transform of each channel (eg, Fast Fourier Transform (FFT) or modified discrete cosine transform). Transform (MDCT)) can be performed to calculate.
[00217] FIG. 35D shows a block diagram of implementation A15 of apparatus A10 including indicator 400. The indicator 400 is configured to indicate the direction of arrival based on the selection of multiple candidate directions generated by the selector 300 (for example, as described herein with reference to the implementation of task T400). .. Device A15 may be implemented, for example, to perform instances of methods M25 and / or M110 as described herein.
[00218] FIG. 36A shows a block diagram of the device MF5 in a general configuration. The apparatus MF5 describes the calculated differences between the channel pairs of the multi-channel signal and for each of the multiple K candidate directions (as described herein with reference to the implementation of task T20). ) Includes means F20 for calculating the error or suitability measure in the corresponding direction based on the calculated difference. The device MF5 also includes means F30 for selecting candidate directions based on the error in the corresponding directions (eg, as described herein with reference to the implementation of task T30).
[00219] FIG. 36B shows the difference between channel pairs of multichannel signals (eg, gain-based or phase-based differences) (eg, as described herein with reference to the implementation of task T10). The block diagram of the implementation form MF10 of the device MF5 including the means F10 for calculating the above is shown. Means F10 can be implemented, for example, to calculate such differences for each of the multiple F frequency components of a multichannel signal. In such cases, means F10 also performs a subband analysis and / or frequency transform of each channel (eg, Fast Fourier Transform (FFT) or Modified Discrete Cosine Transform (MDCT)) before calculating the difference. ) Can be implemented to include means for calculating. The device MF10 may be implemented, for example, to perform instances of methods M10, M20, M30, and / or M100 as described herein.
[00220] FIG. 36C includes means F40 for indicating the direction of arrival based on the selection of multiple candidate directions generated by means F30 (as described, for example, with reference to the implementation of task T400). The block diagram of the mounting form MF15 of the apparatus MF10 is shown. The device MF15 may be implemented, for example, to perform an instance of method M25 and / or M110 as described herein.
[00221] The signal received by the microphone pair can be processed as described herein to provide an estimated DOA over a range of up to 180 degrees with respect to the axis of the microphone pair. The desired angular span and resolution may be arbitrary within that range (eg, uniform (linear) or non-uniform (non-linear), limited to the selected sector, etc.). In addition, or instead, the desired frequency range and resolution may be arbitrary (eg, linear, logarithmic, Mel scale, Bark scale, etc.).
In a model such as that shown in Figure 2B, each DOA estimate between 0 and +/- 90 degrees from a pair of microphones indicates an angle to a plane that is orthogonal to the axis of that pair. Such estimates describe a cone around the axis of the pair, and the actual orientation of the source along the surface of this cone is uncertain. For example, a DOA estimate from a single microphone pair does not indicate whether the source is before or after (or above or below) the microphone pair. Therefore, more than two microphones are used in a linear array to improve DOA estimation performance over the frequency range, but the range of DOA estimation supported by the linear array is generally limited to 180 degrees. ..
The DOA estimation principle described herein can also be extended to a two-dimensional (2D) array of microphones. For example, a 2D array can be used to extend the range of source DOA estimates up to a complete 360 ° (for example, to provide a range similar to that in applications such as radar and biomedical scanning). Such arrays can be used, for example, in speakerphone applications to support good performance even with respect to any placement of the phone for one or more sources.
[00224] Multiple microphone pairs in a 2D array will generally not share the same DOA even for long-range point sources. For example, the source height of the array (for example, in the z-axis) relative to the plane can play an important role in 2D tracking. FIG. 10A shows an example of a speakerphone application in which the xy plane defined by the microphone axis is parallel to the surface on which the phone is placed (eg, a table top). In this example, source 1001 is a person who is along the x-axis 1010 but is speaking from a position that is offset in the direction of the z-axis 1014 (for example, the speaker's mouth is on the table top). With respect to the xy plane defined by the microphone array, the orientation of the source 1001 is along the x-axis 1010, as shown in FIG. 10A. A pair of microphones along the y-axis 1012 estimates the source DOA as zero degrees from the xz plane. However, due to the height of the speaker above the xy plane, the microphone pair along the x-axis will cause the source DOA to be 30 ° to 30 ° (ie, from the yz plane) instead of along the x-axis 1010. 60 degrees). Figures 11A and 11B show two views of the confused conical region CY10 associated with this DOA estimate, which creates ambiguity in the estimated speaker orientation with respect to the microphone axis. FIG. 37A shows another example of a point sound source 3720 (ie, the speaker's mouth) rising from the plane of device H100 (eg, the plane defined by the display plane and / or the microphone array axis).
[00225] In the equation, θ<sub>1</sub>And θ<sub>2</sub>Are the estimated DOAs for 1: 1 and 2, respectively,<maths num="22"><img id="000023" he="29" wi="106" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Expressions such as can be used to project all pairs of DOA into a 360 ° range in the plane where the three microphones are located. Such projections can be used to make it possible to track the orientation of the active speaker over a 360 ° range around the microphone array, regardless of height difference. Applying the above equation to project the DOA estimate (00,600) in Figure 10A onto the xy plane is a direction estimation. <maths num="23"><img id="000024" he="33" wi="95" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Can be mapped to an estimate of 270 ° combined orientation 1022 (eg, azimuth) as shown in Figure 10B.
[00226] In a typical use case, the source is placed in a non-projected orientation on the microphone axis. FIGS. 12A-12D show an example in which the source S01 is placed above the surface of the microphones MC10, MC20, MC30. In this example, the DOA of the source signal passes through the point (x, y, z) = (5,2,5), and FIG. 12A shows the xy plane as seen from the + z direction. 12B and 12D show the xz plane seen from the direction of the microphone MC30, and FIG. 12C shows the yz plane seen from the direction of the microphone MC10. The shaded area in Figure 12A is the DOAθ observed by the y-axis microphone vs. MC20-MC30.<sub>1</sub>Shows the confusing conical region CY associated with, and the shaded region in Figure 12B is the DOA S01 θ observed by the x-axis microphone vs. MC10 to MC20.<sub>2</sub>Shows the confused conical region CX associated with. In Figure 12C, the shaded area shows the conical region CY, and the dashed circle shows the line of intersection between the plane passing through the source and orthogonal to the x-axis and the circular thrust region CX. The two points on the circle that indicate the intersection of the conical region CY and its circle are candidate positions for the source. Similarly, in Figure 12D, the shaded area shows the conical region CX, and the dashed circle shows the line of intersection of the plane orthogonal to the y-axis through the source and the conical region CY, the conical region CX. The two points on the circle that indicate the intersection of and the circle are candidate positions for the source. It will be understood that in this 2D case there remains ambiguity as to whether the source is above or below the xy plane.
[00227] In the example shown in FIGS. 12A-12D, the DOA observed by the x-axis microphone vs. MC10-MC20 is<maths num="24"><img id="000025" he="30" wi="84" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
And the DOA observed by the y-axis microphone vs. MC20 ~ MC30 is<maths num="25"><img id="000026" he="28" wi="114" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Is. An equation to project these directions onto the xy plane<maths num="26"><img id="000027" he="29" wi="106" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Using produces the desired angular magnitudes (21.8 °, 68.2 °) with respect to the x and y axes, respectively, which gives a given source position (x, y, z) = (5,2,5). ) Corresponds. The observed angle signal indicates the xy quadrant where the source (indicated by the microphones MC10, MC20, and MC30) is located, as shown in Figure 11C.
[00228] In fact, the 2D microphone array provides nearly 3D information, except for top and bottom confusion. For example, the directions of arrival observed by the microphones vs. MC10-MC20 and MC20-MC30 can also be used to estimate the magnitude of the ascending angle of the source with respect to the xy plane. If d represents the vector from the microphone MC20 to the source, the length of projection of the vector d on the x-axis, y-axis, and xy plane is dsin (θ), respectively.<sub>2</sub>), Dsin (θ<sub>1</sub>),and<maths num="27"><img id="000028" he="27" wi="91" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Can be expressed as. The magnitude of the ascending angle is then<maths num="28"><img id="000029" he="28" wi="75" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Can be presumed as.
[00229] Microphone pairs in the particular examples of FIGS. 10A-10B and 12A-12D have orthogonal axes, but for microphone pairs with non-orthogonal axes, project DOA estimates for those non-orthogonal axes. Because of the formula<maths num="29"><img id="000030" he="29" wi="85" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Note that can be used, from which it is easy to obtain an estimate of the combined directions with respect to the orthogonal axes. FIG. 37B shows an example of the intersection of confused cones C1 and C2 related to the response of a microphone array with (shown) non-orthogonal axes to a common point source. Figure 37C shows one of the lines of intersection L1 of these cones C1 and C2, which defines one of the two possible directions of a point source with respect to the axis of the array in three dimensions. ..
[00230] In FIG. 13A, the axes 1 of the MC20 and MC30 are in the xy plane, and the bevel angle θ.<sub>0</sub>Only an example of microphone arrays MC10, MC20, MC30 diagonal to the y-axis is shown. FIG. 13B shows the observed values (θ) from the array of microphones MC10, MC20, and MC30 shown in FIG. 13A.<sub>1</sub>, θ<sub>2</sub>) Is used to obtain an estimate of the combined directions in the xy plane with respect to the orthogonal axes x and y. If d represents the vector from the microphone MC20 to the source, the length of projection of the vector d on the x-axis and axis 1 is dsin (θ), respectively.<sub>2</sub>), Dsin (θ<sub>1</sub>) Can be expressed. The vector (x, y) shows the projection of the vector d onto the xy plane. An estimate of x is known, and that estimate still estimates the value of y.
[00231] The estimation of y is the projection of the vector (x, y) on axis 1.<sub>1</sub>= (dsinθ<sub>1</sub>sinθ<sub>0</sub>, Dsinθ<sub>1</sub>cosθ<sub>0</sub>) Can be used. Vector (x, y) and vector p<sub>1</sub>The difference is p<sub>1</sub>Noting that they are orthogonal to<maths num="30"><img id="000031" he="28" wi="97" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Calculate y as. The desired arrival angle of the xy plane with respect to the orthogonal x and y axes is, in this case,<maths num="31"><img id="000032" he="29" wi="144" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths><maths num="32"><img id="000033" he="29" wi="97" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Can be expressed as.
Deploying DOA estimates to 2D arrays is generally suitable for speakerphone applications and sufficient for speakerphone application applications. However, further deployment to N-dimensional arrays is possible and can be done in a simple way. For tracking applications where one target predominates, it may be desirable to select N pairs to represent the N dimension. Once the 2D results are obtained for a particular microphone pair, another available pair can be used to increase the degree of freedom. For example, FIGS. 12A-12D and 13A, 13B illustrate the use of DOA estimates observed from different microphone pairs in the xy plane to obtain source orientation estimates projected on the xy plane. .. Similarly, DOA estimates observed from x-axis microphone pairs and z-axis microphone pairs (or other pairs in the xz plane) are source orientation estimates projected onto the xz plane, as well as the yz plane, Or it can be used to obtain estimates for any other plane that intersects three or more microphones.
[00233] Estimates of DOA error from different dimensions are, for example,<maths num="33"><img id="000034" he="35" wi="81" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Or<maths num="34"><img id="000035" he="30" wi="97" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
It can be used to obtain a combined likelihood estimate using an expression such as, where θ<sub>0, i</sub>Indicates the selected DOA candidates for vs i. It is close to only one of the confusing cones, and therefore close to only one confusing cone, and therefore to use the maximum of different errors rather than the estimates that can indicate false peaks. It may be desired to facilitate the selection of estimates close to the conical region where both observations are confused. Such combined results are shown in the (frame, angle) plane described herein and / or at the bottom of FIG. 9 and the (frame, frequency) plot described herein. Can be used to obtain.
[00234] The DOA estimation principles described herein can be used to support selection from multiple speakers. For example, the location of multiple sources can be manual selection of a particular speaker (for example, pressing a particular button to select a particular corresponding user) or recognition of a particular speaker (eg, speaker recognition). Can be combined with automatic selection (according to). In one such application, the phone is configured to recognize its owner's voice and automatically select the direction corresponding to that voice in preference to the direction of the other source.
[00235] For a one-dimensional (1D) array of microphones, the direction of arrival DOA10 with respect to the source can be easily defined, for example, in the range -90 ° to 90 °. For example, with respect to the phase difference in the signals produced by the various microphones in the array, it is easy to obtain a closed form solution for the coming direction DOA10 over a range of angles (eg, shown in Cases 1 and 2 of Figure 13C). ..
[00236] For an array containing more than two microphones in any relative position (eg, an array that does not have a common axis), a simple extension of the one-dimensional principle described above, eg, two-dimensional two. It may be desirable to use (θ1, θ2) for paired cases and (θ1, θ2, θ3) for three-dimensional three-pair cases. The main question is how to apply spatial filtering to so combine the paired 1D direction of arrival DOA10 estimates. For example, with respect to the phase difference in the signals produced by the various microphones of the array (eg, shown in Cases 3 and 4 of Figure 13C), the closed form for the direction of arrival DOA10 over a range of angles for arrays that do not have a common axis Obtaining a solution can be difficult or impractical.
[00237] Figure 14A shows an example of a simple one-dimensional (1D) pairwise beam-forming null-forming (BFNF) BF10 configuration for spatially selective filtering based on robust 1D DOA estimates. In this example, the notation<maths num="35"><img id="000036" he="27" wi="58" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Is each pair<maths num="36"><img id="000037" he="27" wi="68" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Represents the microphone pair number i, the microphone number j in the pair, and the source number k so that the ellipse represents the steering vector for each source and microphone pair (the ellipse indicates the steering vector for source 1 and microphone pair 1). Indicates the regulation factor. The number of sources is less than or equal to the number of microphone pairs. Such a configuration avoids the need to use all of the microphones at once to define the DOA.
[00238] We can apply the beam-former / null-former (BFNF) BF10 shown in FIG. 14A by expanding the steering vector for each pair. In this figure, A<sup>H</sup>Indicates the conjugate transpose of A, x indicates the microphone channel, and y indicates the spatially filtered channel. Pseudo-inverse operation A shown in Fig. 14A<sup>+</sup>= (A<sup>H</sup>A)<sup>-1</sup>A<sup>H</sup>The use of allows the use of non-square matrices. For example, in the case of three microphones MC10, MC20, MC30 (ie, two microphone pairs) illustrated in Figure 15A, the number of rows is three so that the additional rows are non-square to the matrix. Instead, 2<sup>*</sup>2 = 4.
[00239] Since the method shown in Figure 14A is based on robust 1D DOA estimation, no complete knowledge of microphone geometry is required, nor is DOA estimation using all microphones required at the same time. Such a method is suitable for use in the anglogram-based DOA estimation described herein, but any other 1D DOA estimation method may also be used. FIG. 14B also includes a normalized N10 (eg, by the denominator) to prevent adverse condition reversal at spatial aliasing frequencies (ie, wavelengths twice the distance between microphones), as shown in FIG. 14A. An example of BF10 is shown.
[00240] FIG. 15B shows an example of a pairwise (PW) normalized MVDR (minimum dispersion distortion-free response) BFNF BF10 in which a steering vector (array manifold vector) is acquired unlike the conventional method. In this case, a microphone between the two pairs (for example, x in Figure 15A)<sub>1,2</sub>And x<sub>2,1</sub>A common channel is presumed by sharing a microphone (labeled with). The noise coherence matrix Γ can be obtained by measurement or by theoretical calculation using the sinc function. The examples of FIGS. 14A, 14B, and 15B can be generalized to any number N of sources such that N M, where M is the number of microphones.
[00241] Figure 16A shows the matrix A.<sup>H</sup>Here is another example of BFNF BF10 that can be used if A is not a bad condition, it can be determined using the determinant of the matrix or the conditional number. In this example, the notation is the same as in Figure 14A, where the number of sources N is less than or equal to the number of microphones vs. M. If the matrix is in bad condition, matrix A<sup>H</sup>While A continues to apply methods for spatial filtering of other non-adverse frequency bins, it may be desirable to bypass one microphone signal for that frequency bin for use as a source channel. .. This option omits the calculation to calculate the denominator for normalization. The methods in Figures 14A-16A show the BFNF BF10 technique that can be applied independently in each frequency bin. Steering vectors are constructed using DOA estimates for each frequency and microphone pair described herein. For example, the elements of the steering vector for DOA θi, frequency f, and pair p for macrophone number m (1 or 2) and source n are:<maths num="37"><img id="000038" he="25" wi="56" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths><maths num="38"><img id="000039" he="33" wi="76" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Can be calculated, where l<sub>p</sub>Indicates the distance between the pair of microphones, ω indicates the frequency bin number, and f<sub>s</sub>Indicates the sampling frequency. FIG. 16B shows an example of the steering vectors SV10a to b of the array shown in FIG. 15A.
[00242] The PWBFNF scheme suppresses the direct path of interference to the available degrees of freedom (instantaneous suppression without the assumption of a smooth orbit, additional noise suppression gain using directional masking, additional bandwidth expansion. Can be used to suppress noise). Single-channel post-processing of the quadrant framework can be used for stationary noise and noise reference processing.
[00243] It may be desirable to obtain instantaneous suppression and to minimize artifacts such as music noise. It may be desirable to maximize the degrees of freedom available for BFNF. One DOA can be fixed over all frequencies, or some inconsistent matching over frequencies can be tolerated. Only the current frame can be used, or a feed-forward network can be implemented. BFNF can be set for all frequencies up to the Nike straight (eg, except for frequencies in adverse conditions). Natural masking techniques can be used (for example, to obtain a smooth natural seamless change in aggressiveness). FIG. 31 shows an example of DOA tracking for targets and moving interferers for the scenarios shown in FIGS. 21B and 22. In FIG. 31, the fixed source S10 at D is shown, and the mobile source S20 is also shown.
[00244] FIG. 17 shows an example flowchart of the integration method 1700 described herein. This method has an inventory matching task T10 for phase delay estimation, an error calculation task T20 for obtaining DOA error values, a dimension matching and / or pair selection task T30, and a DOA error for selected DOA candidates. Includes task T40 and for mapping to source activity likelihood estimates. Pairwise DOA estimation results can also be used to track one or more active speakers, to perform pairwise spatial filtering operations, and / or to perform time and / or frequency selective masking. Activity likelihood estimates and / or spatial filtering operations can also be used to obtain noise estimates to support single-channel noise suppression operations. 18 and 19 show an example of observations obtained using a 2D microphone configuration for tracking the movement of a source (eg, a human speaker) within the directional ABCD shown in FIG. 21A. As shown in FIG. 21A, three microphones MC10, MC20, MC30 can be used to record the audio signal. In this example, FIG. 18 shows observations A to D on the y-axis pair MC20-MC30, where the distance dx is 3.6 centimeters, and FIG. 19 shows observations A to MC20 on the x-axis pair MC10-MC20. Indicates D, in which case the distance dy is 7.3 centimeters and the inventory of DOA estimates covers the range from -90 degrees to +90 degrees with a resolution of 5 degrees.
[00245] It will be appreciated that when the source is in the vertical direction of the microphone pair, the rise of the source above or below the plane of the microphone limits the observed angle. As a result, when the source is outside the plane of the microphone, it is common that no substantial vertical orientation is observed. In FIGS. 18 and 19, the ascent of the source with respect to the microphone plane also causes the source to pass in the corresponding vertical direction (ie, direction A with respect to the x-axis vs. MC10-MC20 and direction B with respect to the y-axis vs. MC20-MC30). Regardless, it will be understood that the observed direction does not reach -90 degrees.
[00246] FIG. 20 shows +/ from an orthogonal axis, as shown in FIGS. 18 and 19, for the scenario shown in FIG. 21A to generate DOA estimates in the microphone plane ranging from zero to 360 degrees. An example in which the observation values A to D of -90 degrees are combined is shown. In this example, one-degree resolution is used. FIG. 22 shows a microphone of a source (eg, a human speaker) in directions A to B to C shown in FIG. 21B in the presence of another source of direction D (eg, a static human speaker). An example of combined observations A to D using a 2D microphone configuration to track movement by MC10, MC20, MC30, where distance dx is 3.6 cm and distance dy is 7.3 cm. It is a meter.
[00247] As described above, DOA estimates can be calculated based on the sum of likelihoods. When combining observations from different microphone axes (eg, as shown in Figure 20), there can be more than one directional source (eg, two speakers, or a speaker and an interpreter). In some cases, it may be desirable to perform a coupling on each individual frequency bin before calculating the sum of likelihoods. Assuming that only one of the sources predominates in each frequency bin, calculating the combined observations for each frequency component maintains a distinction between the predominances of different sources at different corresponding frequencies. .. This distinction can be lost if sums for frequency bins dominated by different sources are performed for the observations before the observations are combined, and the combined observations will be at the location of any actual source. Can also show erroneous peaks in uncorresponding directions. For example, summing the observations from the orthogonal microphone pair of the first source at 45 degrees and the second source at 225 degrees, and then combining the combined observations is desired at 45 degrees and 225 degrees. In addition to the peaks, false peaks at 135 and 315 degrees can be produced.
[00248] FIGS. 23 and 24 show an example of combined observations for a conference call scenario, such as that shown in FIG. 25, where the telephone is stationary on a table top. In FIG. 25, the device may include three microphones MC10, MC20, MC30. In FIG. 23, the signal frame number 2310, the arrival angle 2312, and the amplitude 2314 are illustrated. At about frame 5500, speaker 1 stands up and the movement of speaker 1 is apparent in about frame 9000. The movement of speaker 3 in the near frame 9500 is also apparent. The rectangle of FIG. 24 can be rejected or attenuated for frequency components arriving from outside this sector, or is treated differently from frequency components arriving from within the selected sector. The target sector selection TSS10 to obtain is shown. In this example, the target sector is a quadrant of 180-270 degrees and is selected by the user from four quadrants of the microphone plane. This example also includes acoustic interference from the air conditioning system.
[00249] FIGS. 26 and 27 show an example of combined observations for a dynamic scenario as shown in FIG. 28A. In FIG. 28A, the device may be placed between the first speaker S10, the second speaker S20, and the third speaker S30. In FIG. 26, the signal frame number 2610, the arrival angle 2612, and the amplitude 2614 are illustrated. In this scenario, Speaker 1 picks up the handset at About Frame 800 and places the handset on the tabletop at About Frame 2200. It will be appreciated that when the phone is in this browse talk position, the angular span is wider, but the spatial response is still at the center of the specified DOA. The movement of Speaker 2 after About Frame 400 is also clear. As shown in FIG. 24, the rectangle of FIG. 27 shows the user selection of the 180-270 degree quadrant as the target sector TSS10. 29 and 30 show an example of combined observations for a dynamic scenario with traffic noise, as shown in Figure 28B. In Figure 28B, the telephone can receive an audio signal from speaker S10. In FIG. 29, the signal frame number 2910, the arrival angle 2912, and the amplitude 2914 are illustrated. In this scenario, the speaker picks up the handset between the about frames 200 and 100, and again between the about frames 1400 and 2100. In this example, the rectangle in FIG. 30 shows the user selection in the 270-360 degree quadrant as the interference sector IS10.
[00250] (VAD) The anglogram-based technique described herein is used to support voice activity detection (VAD), which can be applied for noise suppression in various use cases (eg, speakerphones). Can be done. Such techniques, which can be implemented as a sector-based scheme, may include "vadall" statistics based on the maximum likelihood (likelihood_max) of all sectors. For example, if the maximum value is well above the noise-only threshold, the value of the vadall statistic is 1 (or zero). It may be desirable to update the noise-only threshold only during the noise-only period. Such periods are based, for example, on single-channel VAD (eg, from the primary microphone channel) and / or utterance onset and / or offset (eg, based on the time derivative of energy for each of the set of frequency components). ) Can be indicated by VAD.
[00251] In addition, or instead, such techniques may include sector-by-sector "vad [sector]" statistics based on the maximum likelihood of each sector . Such statistics show a value of only 1 when the single-channel VAD and onset offset VAD are 1, vadall is 1, and the maximum value for that sector exceeds a portion of likelihood_max (eg, 95%). Can be implemented to have. This information can be used to select the sector with the highest likelihood. Applicable scenarios include user-selected interfering sectors with moving interfering objects and user-selected interfering sectors with moving targets.
[00252] It may be desirable to choose a trade-off between instantaneous tracking (PWBFNF performance) and prevention of too frequent switching of interfering sectors. For example, it may be desirable to combine vadall statistics with one or more other VAD statistics. The vad [sector] can be used to specify the interfering sector and / or to trigger an update of the transient noise reference. For example, using minimal statistics-based normalization techniques (for example, as described in US Patent Application Publication No. 2012/0130713 published May 24, 2012), vadall statistics and / or vad [ sector] It may also be desirable to normalize the statistics.
[00253] The anglogram-based techniques described herein can be used to support directional masking, which can be applied for noise suppression in various use cases (eg, speakerphones). Such techniques provide additional noise suppression gain by using DOA estimates to control directional masking techniques (for example, to pass the target quadrant and / or prevent interference quadrants). Can be used to obtain. Such a method can be useful for handling reverberations and can generate an additional 6-12 dB gain. An interface from the anglogram can be provided for quadrant masking (eg, by assigning a maximum likelihood to the angle for each frequency bin). It may be desirable to control masking aggressiveness based on the target dominance indicated by the anglogram. Such techniques can be designed to obtain a natural masking response (eg, a smooth, natural, seamless change in aggressiveness).
[00254] PW for source tracking and / or with directional masking It may be desirable to provide a multi-view user interface (GUI) for the deployment of BFNF. Various examples of three-microphone (two pairs) two-dimensional (eg, 360 °) source tracking and extension schemes that can be applied to desktop hands-free speakerphone use cases are presented herein. However, it may be desirable to implement universal methods to provide seamless coverage of use cases ranging from desktop hands-free use cases to handheld hand-free use cases, or even handset use cases. The three microphone scheme can be used for handheld hands-free use cases, but it may be desirable to also use a fourth microphone (if already present) on the back of the device. For example, it may be desirable that at least four microphones (three microphone pairs) be available to represent the (x, y, z) dimension. The design shown in FIG. 1 has this function, and the design shown in FIG. 32A with three front microphones MC10, MC20, MC30 and a rear microphone MC40 (shaded circle) also has this function.
[00255] It may be desirable to provide visualization of the active source on the display screen of such devices. The extension principles described herein can be applied to obtain a simple extension from 2D to 3D by using a pair of front and back microphones. Utilize one of a variety of position detection methods to support a multi-view GUI, such as accelerometers, gyrometer, proximity sensors, and / or variance of likelihood provided by a 2D anglogram for each retention pattern. By doing so, the retention pattern of the user can be determined. Depending on the current retention pattern, you can switch between two non-coaxial microphone pairs for such retention pattern, and if the user wants to see, a corresponding 360 ° 2D display on the display. It is also possible to provide to.
[00256] For example, such a method is performed to support switching within a mode range that may include desktop hands-free (eg, speakerphone) mode, vertical browse talk mode, and horizontal browse talk mode. Can be done. FIG. 32B shows an example of a desktop hands-free mode with three front microphones MC10, MC20, MC30 and corresponding visualizations on the display screen of the device. FIG. 32D shows an example of a handheld hands-free mode with two front microphones MC10, MC20, an activated rear microphone MC40 (shaded circle), and a corresponding display. FIG. 32C shows an example of a handheld hands-free mode with different pairs of front microphones MC10 and MC20, one activated rear microphone MC40 (shaded circle), and a corresponding display. In some configurations, the rear microphone MC40 may be located on the back of the device, approximately directly behind the front microphone MC10.
[00257] It may be desirable to implement a factory for the target source. The extension principles described herein can also be applied to obtain a simple extension from 2D to 3D, also by using front and back microphone pairs. Only two DOA estimates (θ)<sub>1</sub>, θ<sub>2</sub>), We have added estimates (θ) from different magnitudes for a total of three DOA estimates.<sub>1</sub>, θ<sub>2</sub>, θ<sub>3</sub>) Can be obtained. In this case, the PWBFNF coefficient matrix as shown in Figures 14A and 14B is expanded from 4x2 to 6x2 (using the added microphone pair) and the masking gain function is f (θ).<sub>1</sub>) f (θ<sub>2</sub>) To f (θ<sub>1</sub>) f (θ<sub>2</sub>) f (θ<sub>3</sub>). Using the position-sensitive selection described above, we found that all three, regardless of the current retention pattern, to obtain seamless transitions between modes with respect to source enhancement performance. The microphone pair can be used optimally. Of course, more than three pairs at a time can be used as well.
[00258] Each of the microphones for direction estimation discussed herein (see location and tracking of one or more users, or other sources) is omnidirectional, bidirectional, or unidirectional. It can have a response that is (eg, cardioid). Various types of microphones that can be used include, but are not limited to, piezoelectric microphones, dynamic microphones, and electret microphones. It should be noted in particular that microphones can be more commonly implemented as transducers that are sensitive to non-audio radiation or emission. In one such example, the microphone array may include one or more ultrasonic transducers (eg, transducers sensitive to acoustic frequencies greater than 15, 20, 25, 30, 40, or 50 kHz and above). It will be carried out in.
[00259] The devices disclosed herein can be implemented as a combination of hardware (eg, a processor) and software and / or firmware. Such devices also generate preprocessed microphone signals (eg, the corresponding microphone signal of the left and right microphone signals) for input to task T10 or a different computer 100 (for example). For example, an audio pre-processing step as shown in Figure 38A, which performs one or more pre-processing operations on the signals produced by each of the microphones MC10 and MC20 (in some implementations with one or more microphone arrays). May include AP10. Such preprocessing operations may include (but not limited to) impedance matching, analog-to-digital conversion, gain control, and / or filtering in the analog and / or digital domain.
[00260] FIG. 38B shows a block diagram of a 3-channel implementation AP20 of an audio preprocessing stage AP10 including analog preprocessing stages P10a, P10b, and P10c. In one example, steps P10a, P10b, and P10c are each configured to perform a high frequency filtering operation on the corresponding microphone signal (eg, with a cutoff frequency of 50, 100, or 200 Hz). In general, steps P10a, P10b and P10c will be configured to perform the same function for each signal.
[00261] The audio pre-processing stage AP10 may desire to generate each microphone signal as a digital signal, i.e., as a sequence of samples. The audio preprocessing stage AP20 includes, for example, analog-to-digital converters (ADCs) C10a, C10b, and C10c, each configured to sample the corresponding analog signal. Typical sampling rates for acoustic applications include 8kHz, 12kHz, 16kHz, and other frequencies in the range of about 8 to about 16kHz, but sampling rates similar to about 44.1, 48, or 192kHz are also used. obtain. Generally, the transducers C10a, C10b and C10c will be configured to sample each signal at the same rate.
[00262] In this example, the audio preprocessing stage AP20 also has the corresponding microphone signals of the left microphone signal AL10, the central microphone signal AC10, and the right microphone signal AR10, respectively, for input to task T10 or a different computer 100. Includes digital preprocessing steps P20a, P20b, and P20c configured to perform one or more preprocessing operations (eg, spectrum formation) on the corresponding digitized channel to generate .. In general, steps P20a, P20b and P20c will be configured to perform the same function for each signal. Preprocessing Stage AP10 is a microphone for content use (eg, at different sampling rates and / or with different spectrum formation), such as to provide near-end voice signals in voice communications (eg, telephone calls). Also note that it can be configured to produce different versions of the signal from at least one of them. Although FIGS. 38A and 38B show two-channel and three-channel implementations, respectively, it will be appreciated that the same principle can be extended to any number of microphones.
[00263] FIG. 39A is a means F40 for indicating the direction of arrival based on the selection of multiple candidate directions generated by the means F30 (eg, described herein with reference to an implementation of task T400). The block diagram of the mounting form MF15 of the apparatus MF10 including the above is shown. The device MF15 may be implemented, for example, to perform an instance of method M25 and / or M110 as described herein.
[00264] Signals received by a pair of microphones or other linear arrays of microphones can be processed as described herein to provide an estimated DOA that indicates an angle relative to the axis of the array. .. As described above (eg, in connection with methods M20, M25, M100, and M110), more than two microphones can be used in a linear array to improve DOA estimation performance over the frequency range. However, even in such cases, the DOA estimation range supported by the linear (ie, one-dimensional) array is generally limited to 180 degrees.
[00265] Figure 2B shows a measurement model showing the angle (within the 180 degree range of +90 to -90 degrees) with respect to the plane where the one-dimensional DOA estimate is orthogonal to the axis of the array. Implementations of methods M200 and M300, as well as task TB200, are described below with reference to the situation shown in Figure 2B, but such implementations are not limited to this situation (DOA estimate is microphone MC10). Corresponding implementations that reference other situations (indicating an angle of 0-180 degrees with respect to an axis in the direction of, or an axis away from the microphone MC10) are expressly contemplated and disclosed herein. It will be recognized that it will be done.
[00266] The desired angular span can be arbitrary within the 180 degree range. For example, the DOA estimate may be limited to the selected sector within that range. The desired angular resolution may also be arbitrary (eg, it may be uniformly distributed or unevenly distributed over that range). In addition, or instead, the desired frequency range may be arbitrary (eg, limited to the audio range) and / or the desired frequency resolution is arbitrary (eg, linear, logarithmic, Mel scale). , Bark scale, etc.).
[00267] Figure 39B shows an example of the ambiguity that arises from the one-dimensionality of DOA estimates from a linear array. In this example, the DOA estimates from the microphone vs. MC10, MC20 (for example, the candidate direction generated by selector 300 or the DOA estimate generated by indicator 400) indicate an angle θ relative to the array axis. .. Even if this estimate is very accurate, it does not indicate whether the source is placed along line d1 or along line d2.
[00268] As a result of its one-dimensionality, the DOA estimate from the linear microphone array is not in any particular direction in space, but in fact (assuming the microphone response is completely omnidirectional). Describe the right-angled surface around the array axis in space. The actual position of the source on this conical surface (also known as the "confused conical region") is uncertain. FIG. 39C shows an example of such a surface.
[00269] Figure 40 shows source confusion in a speakerphone application where three sources (eg, the mouth of a human speaker) are placed in different directions with respect to a device D100 (eg, a smartphone) that has a linear microphone array. An example is shown. In this example, the source directions d1, d2, and d3 all happen to be on the confusing conical region defined in the microphone MC20 at an angle (θ + 90 degrees) with respect to the array axis in the direction of the microphone MC10. Since all three source orientations have the same angle with respect to the array axis, the microphone pair produces the same DOA estimates for each source and cannot distinguish between them.
[00270] It may be desirable to extend the DOA estimation principles described herein to a two-dimensional (2D) array of microphones in order to provide estimates with higher dimensions. FIG. 41A shows a 2D microphone array containing two microphone pairs with orthogonal axes. In this example, the axes of the first pair MC10 and MC20 are the x-axis, and the axes of the second pair MC20 and MC30 are the y-axis. An instance of the method M10 implementation is the 1D DOA estimate θ to which the first pair corresponds.<sub>x</sub>An instance of the implementation of method M10 can be performed to generate the 1D DOA estimate θ to which the second pair corresponds.<sub>y</sub>Can be executed to generate. For signals coming from sources located in the plane defined by the microphone axis, θ<sub>x</sub>And θ<sub>y</sub>The confused conical regions described by coincide in the signal arrival direction d to indicate a unique direction within that plane.
[00271] FIG. 41B shows a flow chart of method M200 with a general configuration including tasks TB100a, TB100b, and TB200. Task TB100a calculates a first DOA estimate for the multi-channel signal for the axis of the microphone's first linear array, and task TB100a calculates a second DOA for the multi-channel signal for the axis of the microphone's second linear array. Calculate the estimate. Each of the tasks TB100a and TB100b can be performed, for example, as an instance of an implementation of method M10 (eg, method M20, M30, M100, or M110) as described herein. Based on the first and second DOA estimates, task TB200 calculates the combined DOA estimates.
[00272] The range of combined DOA estimates may be greater than the range of either the first DOA estimate and the second DOA estimate. For example, task TB200 shows individual ranges up to 180 degrees generated by tasks TB100a and TB100b to generate combined DOA estimates that show DOA as angles in the range up to 360 degrees. Can be performed to combine 1D DOA estimates with. Task TB200 combines one angle with information from another (for example, signal information).<maths num="39"><img id="000040" he="29" wi="105" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Etc., by applying the mapping, 1D DOA estimate θ<sub>x</sub>, Θ<sub>y</sub>Can be implemented to map directions within a larger angle range. As shown in Figure 41A, the 1D estimate (θ)<sub>x</sub>, θ<sub>y</sub>) = (45 °, 45 °), for example, TB200 is a combined estimate of 45 degrees relative to the x-axis θ<sub>c</sub>Can be implemented to apply such a mapping to obtain. In the case where the range of DOA estimates is 0 to 180 degrees instead of -90 to +90 degrees, the axial polarity (ie, positive or negative) condition of Eq. (1) is that the DOA estimate for testing is It will be understood that it will be expressed as to whether it is less than 90 degrees or more than 90 degrees.
[00273] DOA estimate θ coupled on a 360 degree range display<sub>c</sub>May be desirable to indicate. For example, it may be desirable to display the DOA estimate as an angle on a 2D polar plot. Two-dimensional polar plots are well known for applications such as radar scanning and biomedical scanning. FIG. 41C shows an example of DOA estimates shown on such a display. In this example, the direction of the line indicates the DOA estimate and the length of the line indicates the current intensity of the component arriving from that direction. As shown in this example, a polar plot can also contain one or more concentric circles to indicate the intensity of the directional component on a linear or logarithmic (eg, decibel) scale. If more than one DOA estimate is available at a time (for example, for independent sources in terms of frequency), a corresponding line for each DOA estimate may be displayed. Alternatively, the DOA estimate may be displayed on a rectangular coordinate system (eg, Cartesian coordinates).
[00274] FIGS. 42A and 42B show 1D estimates θ, respectively.<sub>x</sub>And θ<sub>y</sub>Indicates a match between the symbol of and the corresponding quadrant of the plane defined by the array axis. Figure 42C shows a tuple (sign (θ)).<sub>x</sub>), Sign (θ<sub>y</sub>)) Shows the correspondence between the four values and the plane quadrant. Figure 42D shows a 360 degree display with alternative mapping (for example, to the y-axis).<maths num="40"><img id="000041" he="29" wi="119" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Is shown.
[00275] Fig. 41A shows θ<sub>x</sub>And θ<sub>y</sub>Note that the confused conical region described by points to a unique direction in this plane, exemplifying a special case where the source is placed in a plane defined by the microphone axis. For most practical applications, it can be predicted that the confusing cones of a pair of non-linear microphones in a 2D array will generally not match in the plane defined by the array, even for distance point sources. For example, the source height with respect to the plane of the array (eg, the displacement of the source along the z-axis) can play an important role in 2D tracking.
[00276] It may be desirable to generate an accurate 2D representation of the direction of arrival for a signal received from a source at any location in 3D space. For example, the combined DOA estimate generated by task TB200 is the DOA of the source signal in a plane that does not contain the DOA (for example, the plane defined by the microphone array or the display plane of the device). May be desirable to indicate. Such a display, for example, to support any placement of the audio sensing device with respect to the source and / or any relative movement of the device and source (for example, in the case of speakerphone and / or source tracking applications). Can be used.
[00277] FIG. 43A shows an example similar to FIG. 41A, but showing a more common case where the source is located on the xy plane. In such cases, the intersection of the confused conical regions of the array indicates two possible directions of arrival: the direction d1 that extends above the xy plane and the direction d2 that extends below the xy plane. In many applications, this ambiguity can be resolved by ignoring the second direction d2, where the direction d1 is accurate. For example, in the case of a speakerphone application where the device is placed on a tabletop, it can be assumed that no source is placed under the device. In both cases, the projections in directions d1 and d2 on the xy plane are the same.
[00278] 1D estimate θ (eg, as in equation (1) or (2))<sub>x</sub>And θ<sub>y</sub>A mapping to a 360 degree range of can produce a proper DOA display when the source is placed in the microphone plane, but the mapping is more such that the source is not placed in that plane. In the general case, it may produce inaccurate results. For example, θ as shown in Fig. 41B<sub>x</sub>= θ<sub>y</sub>In the case of, it will be understood that the corresponding direction in the xy plane is 45 degrees with respect to the x-axis. However,<sub>x</sub>, θ<sub>y</sub>Applying the mapping of Eq. (1) to) = (30 °, 30 °) does not correspond to the source direction projected on the plane, a combined estimate of 30 degrees with respect to the x-axis θ<sub>c</sub>To create.
[00279] Figure 43B shows another example of a 2D microphone array whose axis defines the xy plane and a source located on the xy plane (for example, a speakerphone application with the speaker's mouth on the table top). .. With respect to the xy plane, the sources are placed along the y-axis (at an angle of 90 degrees with respect to the x-axis). The x-axis pairs MC10 and MC20 show a zero degree DOA that coincides with the source direction projected onto the xy plane with respect to the yz plane (ie, the sides of the pair of axes). The source is placed directly on the y-axis, but the source is also offset in the z-axis direction by an angle of attack of 30 degrees. The rise of the source from the xy plane causes the y-axis vs. MC20, MC30 to show a DOA of 60 degrees (ie, relative to the xz plane) instead of 90 degrees. Value (θ<sub>x</sub>, θ<sub>y</sub>Applying the mapping of Eq. (1) to) = (0 °, 60 °) does not correspond to the source direction projected on the plane, a combined estimate of 60 degrees with respect to the x-axis θ<sub>c</sub>To create.
[00280] In a typical use case, the source will be placed in a direction that is neither the plane defined by the array axes nor the plane directly above the array axes. FIG. 43C shows an example of such a common case where a point source (ie, the speaker's mouth) rises on a plane defined by the array axes. To get an accurate representation in the array plane in the source direction outside that plane, to convert the 1D DOA estimate to an angle in the array plane to get the corresponding DOA estimate in that plane. It may be desirable to carry out the task TB200.
[00281] Figures 44A-44D show (1) to an angle in the array plane.<sub>θx</sub>,<sub>θy</sub>) Shows the derivation of such a transformation. In FIGS. 44A and 44B, the source vector d is projected on the x-axis and the y-axis, respectively. The length of these projections (d sin θ, respectively)<sub>x</sub>, d sin θ<sub>y</sub>) Is the magnitude of the projection p of the source vector d on the xy plane, as shown in FIG. 44C. These magnitudes are DOA estimates (θ), as shown in Figure 44D.<sub>x</sub>, θ<sub>y</sub>), The angle of p in the xy plane with respect to the y-axis and the x-axis, respectively.<maths num="41"><img id="000042" he="27" wi="65" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Enough to decide to convert to.<maths num="42"><img id="000043" he="28" wi="159" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Where ε is a small value that can be included to avoid division by zero errors. (See FIGS. 43B, 43C, 44A-E, and also 46A-E discussed below, the relative amplitude of d as shown is for convenience of illustration only, and the amplitude of d is the microphone. Note that for the amplitude of the array, the long-range assumption of the flat wave surface should be large enough to remain valid.) [00282] Task TB200 transforms the DOA estimate into the corresponding angle in the array plane according to such an equation, and the combined DOA estimate θ in that plane.<sub>c</sub>Can be performed to apply a mapping (eg, as in equation (1) or (2)) to the transformed angle. The value θc is (for example, as shown in equations (1) and (2)).<maths num="43"><img id="000044" he="27" wi="57" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Combined with<maths num="44"><img id="000045" he="27" wi="37" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Such an implementation of task TB200 is included in equation (3), as it can be determined from<maths num="45"><img id="000046" he="26" wi="56" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
(Or<maths num="46"><img id="000047" he="26" wi="55" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Note that the calculation of can be omitted.<maths num="47"><img id="000048" he="27" wi="58" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
If the value of is still desired<maths num="48"><img id="000049" he="27" wi="51" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Can be calculated as (<maths num="49"><img id="000050" he="29" wi="39" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
The same is true for).
[00283] FIG. 43C shows an example of which DOA of the source signal passes through the point (x, y, z) = (5,2,5). In this case, the DOA observed by the x-axis microphone vs. MC10 ~ MC20 is<maths num="50"><img id="000051" he="27" wi="110" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
And the DOA observed by the y-axis microphone vs. MC20 ~ MC30 is<maths num="51"><img id="000052" he="27" wi="113" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Is. Using equation (3) to convert these angles to the corresponding angles in the xy plane corresponds to the given source position (x, y) = (5,2), the transformed DOA. Estimated value<maths num="52"><img id="000053" he="26" wi="100" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
To generate.
[00284] Values (θx, θ) as shown in Figure 41B.<sub>y</sub>Applying equation (3) to) = (30 °, 30 °) is a transformed estimate that is mapped by equation (1) to a predicted value of 45 degrees with respect to the x-axis.<maths num="53"><img id="000054" he="27" wi="59" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
To generate. Values (θx, θ) as shown in Figure 43B<sub>y</sub>Applying equation (3) to) = (0 °, 60 °) is a transformed estimate that is mapped by equation (1) to a predicted value of 90 degrees with respect to the x-axis.<maths num="54"><img id="000055" he="21" wi="106" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
To generate.
[00285] Task TB200 performs the transformations and mappings described above for the purpose of projecting DOA onto the plane in which the array is located, as indicated by any such pair of DOA estimates from a 2D orthogonal array. Can be implemented to apply. Such projections can be used to allow the tracking direction of the active speaker to span 360 ° around the microphone array, regardless of height difference. Figure 45A shows an estimate of the combined orientation of 270 degrees (eg, azimuth) to obtain. Converted estimates from Figure 43B<maths num="55"><img id="000056" he="26" wi="84" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Alternative mapping to<maths num="56"><img id="000057" he="29" wi="109" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Shows the plot obtained by applying. In this figure, the labels on the concentric circles indicate the relative amplitude in decibels.
[00286] Task TB200 may also be performed to include a validity test on the observed DOA estimates prior to the calculation of the combined DOA estimates. For example, the value (| θ) (for example, to ensure that the confusing cones associated with the two observed estimates will intersect along at least one line).<sub>x</sub>| + | θ<sub>y</sub>It may be desirable to make sure that |) is at least equal to 90 degrees.
[00287] In practice, the information provided by such DOA estimates from a 2D microphone array is nearly complete in three dimensions, except for top and bottom confusion.
For example, the directions of arrival observed by the microphone pairs MC10-MC20 and MC20-MC30 can also be used to estimate the magnitude of the source elevation with respect to the xy plane. If d represents the vector from the microphone MC20 to the source, the length of projection of the vector d onto the x-axis, y-axis, and xy planes is d sin (eg, as shown in Figures 44A-44E), respectively. (θ<sub>2</sub>), D sin (θ)<sub>1</sub>),and<maths num="57"><img id="000058" he="29" wi="93" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Can be expressed as. The size of the elevation angle, in that case<maths num="58"><img id="000059" he="31" wi="81" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Can be presumed as.
[00288] In some specific cases, it may be desirable to implement method M200 for more general cases where the linear microphone array has orthogonal axes, but the axes of the microphone array are not orthogonal. FIG. 45B shows an example of the intersection of confused conical regions associated with the response of a linear microphone array with non-orthogonal axes x and r to a common point source. Figure 45C shows the intersecting lines of these conical regions that define the two possible directions d1 and d2 of the point source with respect to the three-dimensional array axis.
[00289] In Fig. 46A, the axis of the MC10 to MC20 is the x-axis, the axis r of the MC20 to MC30 is on the xy plane, and the microphone array MC10 to MC20 ~ is oblique to the y-axis by the bevel angle α. An example of MC30 is shown. FIG. 46B shows the observations from the array shown in FIG. 46A.<sub>θx</sub>,<sub>θr</sub>) Is shown as an example of obtaining an estimate of the combined directions in the xy plane with respect to the orthogonal axes x and y. If d represents the vector from the microphone MC20 to the source, then the vector d (d) on the x-axis<sub>x</sub>) And the vector d (d) on the axis r<sub>r</sub>) Projection lengths are d sin (θ), respectively, as shown in Figures 46B and 46C.<sub>x</sub>) And d sin (θ<sub>y</sub>) Can be expressed. Vector p = (px, p<sub>y</sub>) Indicates the projection of the vector d onto the xy plane. p<sub>x</sub>= d sin θ<sub>x</sub>Estimates are known, and those estimates are still p<sub>y</sub>Determine the value of.
[00290] We assume that the value of α is within the range (-90 ° C, + 90 °), and arrays with any other value α are easily mapped to such cases. obtain. p<sub>y</sub>The value of is the projection vector d, as shown in Figures 46D and 46E.<sub>r</sub>= (d sin θ<sub>r</sub> d sin α, d sin θ<sub>r</sub>It can be judged from the size of cosα). Vector p and vector d<sub>r</sub>The difference between and is orthogonal to dr (ie, the product <(pd)<sub>r</sub>), D<sub>r</sub>Note that> is equal to zero)<maths num="59"><img id="000060" he="29" wi="119" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
As p<sub>y</sub>Is calculated (this is p when α = 0)<sub>y</sub>= d sin θ<sub>r</sub>Will be easier). The desired angle in the xy plane with respect to the orthogonal x and y axes is, in this case,<maths num="60"><img id="000061" he="24" wi="159" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Can be expressed as.
Note that Eq. (3) is a special case of Eq. (4) where α = 0. The magnitude of the projection p (p<sub>x</sub>, p<sub>y</sub>) Is the source elevation / elevation θ with respect to the xy plane (eg, as explained above with reference to Figure 44E).<sub>h</sub>Can also be used to estimate.
[00291] FIG. 47A shows a flow chart of method M300 in a general configuration involving instances of tasks TB100a and TB100b. The method M300 may also include an implementation TB300 of task TB200 that calculates the projection of the direction of arrival into a plane that does not include the direction of arrival (eg, the plane defined by the array axes). In such a fashion, a 2D array can be used to extend the range of source DOA estimates from linear 180 degree estimates to planar 360 degree estimates. FIG. 47C shows an example of an apparatus A300 equipped with components for performing the functions corresponding to FIG. 47A (eg, first DOA estimator B100a, second DOA estimator B100b, and projection calculator B300). .. FIG. 47D shows the means for performing the functions corresponding to FIG. 47A (eg, the means for calculating the first DOA estimate for the axis of the first array, FB100a, the second for the axis of the second array. An example of the device MF300 including the means FB100b for calculating the DOA estimate and the means FB300) for calculating the projection of the DOA on a plane containing no DOA is shown.
[00292] FIG. 47B shows a flowchart of an implementation form TB302 of the task TB300 including the subtasks TB310 and TB320. Task TB310 has a first DOA estimate (eg, θ).<sub>x</sub>) Is an angle in the projective plane (for example)<maths num="61"><img id="000062" he="26" wi="56" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
). For example, task TB310 may perform the transformation as shown in equation (3) or (4), for example. Task TB320 combines the transformed angle with information from a second DOA estimate (eg, symbolic information) to obtain a projection in the direction of arrival. For example, task TB320 may perform mapping according to equation (1) or (2), for example.
As explained above, extending the source DOA estimate to two dimensions is an elevation angle of DOA over a range of 90 degrees (eg, to provide a measurement range that describes a region on the array plane). It may also include estimating the elevation angle. FIG. 48A shows a flowchart of such an implementation M320 of method M300 including task TB400. Task TB400 calculates an estimate of the elevation angle of DOA with reference to the plane containing the array axes (eg, described herein with reference to FIG. 44E). Method M320 can also be performed to combine projected DOA estimates with estimated elevation elevations to generate 3D vectors.
[00294] It may be desirable to implement the implementation of Method M300 within an audio sensing device having a 2D array that includes two or more linear microphone arrays. An example of a portable audio sensing device that can be implemented to include such a 2D array and that can be used to perform such methods for audio recording and / or voice communication applications is a telephone headset (eg, for example). , Cellular phone handset), wired or wireless headset (eg Bluetooth® headset), handheld audio and / or video recorder, personal media player, mobile device configured to record audio and / or video content There are information terminals (PDAs) or other handheld computing devices, and notebook computers, laptop computers, netbook computers, tablet computers, or other portable computing devices. Types of portable computing devices now include devices with names such as laptop computers, notebook computers, netbook computers, ultra-portable computers, tablet computers, mobile internet devices, smartbooks, and smartphones. Such a device may have an upper panel that includes a display screen and a lower panel that may include a keyboard, and the two panels may be connected in a clamshell or other hinged relationship. Such a device can be similarly implemented as a tablet computer with a touch screen display on top.
Deploying DOA estimates (as described herein with reference to, eg, a method M200 implementation and a method M300 implementation) in a 2D array is generally well suited for speakerphone applications. Sufficient for speakerphone applications. However, such a principle can be further extended to N-dimensional (N-dimensional 2) arrays and can be implemented in a simple manner. For example, FIGS. 41A-46E illustrate using DOA estimates observed from different microphone pairs in the xy plane to obtain source orientation estimates projected in the xy plane. Similarly, an instance of Method 200 or Method 300 gets an estimate of the source direction projected in the xz plane, as well as an estimate for the yz plane or any other plane that intersects three or more microphones. To do so, it can be performed to combine DOA estimates observed from x-axis microphone pairs and z-axis microphone pairs (or other pairs in the xz plane). The 2D projected estimates can then be combined to obtain a 3D estimated DOA. For example, to get a combined DOA estimate as a vector in (x, y, z) space, the DOA estimate for the source projected on the xy plane is for the source projected on the xz plane. Can be combined with DOA estimates.
[00296] For tracking applications where one target predominates, it may be desirable to select N linear microphone arrays (eg, pairs) to represent each of the N dimensions. Method M200 or M300 DOA from each of one or more linear arrays in other planes the 2D results obtained using a particular pair of such linear arrays to provide additional degrees of freedom. Can be performed to combine with estimates.
[00297] Estimates of DOA error from different dimensions are, for example,<maths num="62"><img id="000063" he="29" wi="116" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Or<maths num="63"><img id="000064" he="33" wi="87" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Can be used to obtain a combined likelihood estimate using an expression such as.
In the formula, θ<sub>0, i</sub>Indicates the selected DOA candidates for vs i. Using the maximum of different errors is close to only one of the conical regions to be confused, and therefore confuses both observations in preference to estimates that may indicate false peaks. It may be desired to facilitate the selection of estimates close to the conical region. Such combined results are shown in the (frame, angle) plane described herein and / or at the bottom of FIG. 9 and the (frame, frequency) plot described herein. Can be used to obtain.
[00298] FIG. 48B shows a flowchart of the implementation form M325 of the method M320 including the task TB100c and the implementation form T410 of the task T400. Task TB100c calculates a third estimate of the direction of arrival with respect to the axis of the third microphone array. Task TB410 estimates elevation and elevation based on information from DOA estimates from tasks TB100a, TB100b, and TB100c.
[00299] Methods M200 and M300 calculate their DOA estimates based on some type of difference (eg, phase-based difference) between the microphone channels that Task TB100a corresponds to, and Task TB100b (or, or Task TB100b). It should be noted in particular that task TB100c) can be performed to calculate its DOA estimate based on another type of difference between the corresponding microphone channels (eg, gain-based difference). In one application of such an example of Method M325, the array defining the xy plane was placed at an offset along the z-axis with respect to the front and back pairs (eg, microphone MC10, MC20, or MC30 4). Expanded to include). The DOA estimates generated by Task TB100c for this pair ambiguity of the anterior and posterior surfaces within elevation and elevation so that this method provides a complete spherical measurement range (eg, 360 degrees in any plane). Used in task TB400 to resolve. In this case, method M325 may be implemented such that the DOA estimates produced by tasks TB100a and TB100b are based on the phase difference and the DOA estimates produced by task TB100c are based on the gain difference. In a particular example (for example, to track only one source), the DOA estimate generated by task TB100c has two states: the first step indicating that the source is on a surface. Has a second stage, indicating that the sauce is below the surface.
[00300] FIG. 49A shows a flowchart of the implementation form M330 of the method M300. Method M330 presents task TB500 displaying the calculated projection to the user of the audio sensing device. The task TB500 may be configured to display, for example, the calculated projection on the display screen of the device in the form of a polar plot (eg, as shown in Figures 41C, 42D, and 45A). Examples of such display screens, which may be touch screens as shown in FIG. 1, include liquid crystal displays (LCDs), organic light emitting diode (OLED) displays, electrowetting displays, electrophoretic displays, and interference modulators. There is a display. Such a display may also include an estimated elevation angle display (eg, as shown in FIG. 49B).
[00301] Task TB500 may be performed to display the projected DOA relative to the device reference direction (eg, the device spindle). In such cases, the direction in which they are displayed will change as the device rotates with respect to the fixed source, even if the position of the source does not change. Figures 50A and 50B show examples of such displays before and after such rotations, respectively.
[00302] Alternatively, it is desired to perform task TB500 to display the projected DOA with respect to the external reference direction so that the indicated direction remains constant as the device rotates with respect to the fixed source. Can be rare. 51A and 51B show examples of such displays before and after such rotations, respectively.
[00303] To support such an implementation of Task TB500, device D100 is placed in an external reference direction, such as the axis of gravity (eg, the axis perpendicular to the surface of the earth) or the magnetic axis (eg, the magnetic axis of the earth). Can be configured to include a directional sensor (not shown) that indicates the current spatial orientation of the device with reference to. Directional sensors may include one or more inertial sensors, such as gyroscopes and / or accelerometers. The gyroscope uses the principle of angular momentum to detect changes in direction with respect to one axis, or with respect to each of two or three (generally orthogonal) axes (eg, changes in pitch, roll and / or twist). use. An example of a gyroscope that can be made as a microelectromechanical system (MEMS) device is a vibrating gyroscope. Accelerometers detect acceleration along one axis, or along each of two or three (generally orthogonal) axes. Accelerometers can also be made as MEMS devices. It is also possible to combine a gyroscope and an accelerometer into a single sensor. In addition, or instead, the directional sensor measures one or more magnetic field strengths along one axis, or along each of two or three (generally orthogonal) axes. (For example, a magnetometer) may be included. In one example, device D100 includes a magnetic field sensor that indicates the current orientation of the device with respect to the magnetic axis (eg, the earth). In such cases, task TB500 may be performed to display the projected DOA on a grid rotated to align with its axis (eg, compass).
[00304] FIG. 49C shows a flowchart of such an implementation M340 of method M330 including an implementation T510 of task TB600 and task T500. Task TB600 refers to an external reference axis (eg, gravity axis or magnetic axis) to determine the orientation of the audio sensing device. Task TB510 displays a projected projection based on the determined direction.
[00305] Task TB500 may be performed to display the DOA as an angle projected onto the array plane. For many portable audio sensing devices, the microphone used for DOA estimation will be placed on the same device surface as the display (eg, microphones ME10, MV10-1, and MV10-3 in Figure 1). Or will be placed closer to their surface than to each other (eg, the microphones ME10, MR10, and MV10-3 in Figure 1). For example, the thickness of a tablet computer or smartphone is generally small compared to the dimensions of the display surface. In such cases, the error between the DOA projected on the array plane and the DOA projected on the display plane may be expected to be negligible, and the DOA projected on the array plane It may be acceptable to configure task TB500 to display.
[00306] If the display plane is significantly different from the array plane, task TB500 can be performed to project the estimated DOA from the plane defined by the axes of the microphone array onto the plane of the display surface. For example, such an implementation of task TB500 can display the result of applying a projection matrix to the estimated DOA, in which case the projection matrix describes the projection from the array plane on the surface of the display. .. Alternatively, task TB300 may be performed to include such a projection.
[00307] As described above, an audio sensing device may include a directional sensor that indicates the current spatial orientation of the device with reference to an external reference direction. It may be desirable to combine the DOA estimates described herein with such directional information in order to indicate DOA estimates with reference to external reference directions. FIG. 53B shows a flowchart of such an implementation M350 of method M300 including an instance of task TB600 and an implementation TB310 of task TB300. Method M350 may also be implemented to include an instance of the display task TB500 described herein.
[00308] FIG. 52A shows an example in which the device coordinate system E is aligned with the world world coordinate system. FIG. 52A also shows the device direction matrix F corresponding to this direction (eg, as indicated by the direction sensor). FIG. 52B shows an example of device rotation (eg, when used in browse talk mode) and a matrix F corresponding to this new direction (eg, indicated by a direction sensor).
[00309] Task TB310 may be performed to use the device direction matrix F to project DOA estimates in any plane defined with reference to the world world coordinate system. In one such example, the DOA estimate is the vector g in the device coordinate system. In the first operation, the vector g is transformed into the vector h in the world world coordinate system by the inner product using the device direction matrix F. Such a transformation, for example,<maths num="64"><img id="000065" he="26" wi="81" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Can be executed according to an expression such as. In the second action, the vector h is projected<maths num="65"><img id="000066" he="27" wi="76" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Projected onto the plane P defined with reference to the world world coordinate system, where A is the basis matrix of the plane P in the world world coordinate system.
[00310] In a typical example, the plane P is parallel to the xy plane of the world world coordinate system (ie, the "world reference plane"). FIG. 52C shows the perspective mapping of DOA projections onto the world reference plane onto the display plane of the device, which can be performed by task TB500, in which case the orientation of the display plane with respect to the world reference plane is determined by the device orientation matrix F. Shown. Figure 53A shows an example of such a mapped display of DOA projected onto a world reference plane.
In another example, task TB310 is configured to project a DOA estimate vector g into plane P using a less complex interpolation of the component vector g projected in plane P. To. In this case, the projected DOA estimation vector P<sub>g</sub>Is<maths num="66"><img id="000067" he="26" wi="118" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
It can be calculated according to the following formula.
During the ceremony<maths num="67"><img id="000068" he="27" wi="68" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Indicates the basis vector of the device coordinate system,<maths num="68"><img id="000069" he="27" wi="64" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
θ<sub>α</sub>, θ<sub>β</sub>, θ<sub>γ</sub>Are the planes P and<maths num="69"><img id="000070" he="22" wi="63" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Indicates the angle between the plane and the plane stretched by, and α, β, and γ are their respective cosine (α).<sup>2</sup>+ β<sup>2</sup>+ γ<sup>2</sup>) = 1 and g<sub>xy (p)</sub>, g<sub>xz (p)</sub>Are component vectors, respectively<maths num="70"><img id="000071" he="25" wi="159" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Shows the projection of on the plane P. The plane corresponding to the minimum value in α, β, and γ is the plane closest to P, and the alternative implementation of task TB310 identifies this minimum value and corresponds to the projected component vector. Component vector P<sub>g</sub>Generated as an approximation of.
It may be desirable to configure the audio sensing device to distinguish between source signals with different DOAs. For example, a direction on a multi-channel signal to allow directional components coming from within a certain angle of passage and / or to block or attenuate directional components coming from within a certain angle of stop. It may be desirable to configure the audio sensing device to perform a selectable filtering operation.
[00313] To support a graphical user interface to allow users of audio sensing devices to configure directionally selective processing actions (eg, beam forming actions described herein). It may be desirable to use the display described herein. FIG. 54A shows an example of such a user interface showing the directional range through which the shaded portion of the circle will pass and the directional range through which the shaded portion will be blocked. A circle represents a point on the touch screen that the user can slide along the circumference of the circle to change the selected range. Touch points can be linked so that moving one point moves the other point in the same angular direction, or instead, in the opposite angular direction, by equal angles. Alternatively, the touch points (eg, as shown in FIG. 54B) may be independently selectable. It is also possible to provide one or more additional pairs of touch points to support selection of a range of angles greater than one (for example, as shown in Figure 54C).
As an alternative to the touchpoints shown in Figures 54A-C, the user interface is another physical or virtual selection interface for obtaining user input to select the pass / stop band position and / or width. Can include (for example, clickable or touchable icons on the screen). Examples of such interfaces are linear slider potentiometers, rocker switches (for example, for binary inputs to indicate up / down, left / right, clockwise / counterclockwise), and wheels or knobs, as shown in Figure 53C. ..
[00315] In use cases where the audio sensing device is expected to remain stationary during use (eg, the device is placed on a flat surface for speakerphone use), it is fixed to the device. It may be sufficient to indicate a range of selected directions. However, if the orientation of the device with respect to the desired source changes during use, components arriving from that source can no longer be allowed. 55A and 55B show further examples where the directional sensor is used to track the orientation of the device. In this case, the directional displacement of the device (indicated by the directional sensor) is to update the directional filtering configuration selected by the user so that the desired directional response can be maintained regardless of changes in the device orientation. Used (and to update the corresponding display).
The array is desired to include several microphones that are at least equal to the number of different source directions that will be distinguished at any one time point (eg, the number of beams that will be formed). obtain. The microphone may be omnidirectional (as is common with cellular phones or dedicated conferencing devices, for example) or directional (as is common with devices such as set-top boxes). May be good.
The DOA estimation principles described herein can be used to support selection from multiple speakers. For example, the location of multiple sources can be manually selected by a particular speaker (for example, pressing a particular button to select a particular corresponding speaker or active source orientation, or touching a particular screen area. (To do) or can be combined with the automatic selection of a particular speaker (eg, by speaker recognition). In one such application, an audio sensing device (eg, a phone) recognizes its owner's voice and automatically chooses the direction corresponding to that voice in preference to the direction of the other source. It is configured as follows.
B. Systems and methods for mapping source locations It should be noted that one or more of the functions, devices, methods, and / or algorithms described above may be performed in accordance with the systems and methods disclosed herein. Some configurations of the systems and methods disclosed herein are multimodal sensor fusions for seamless audio processing. fusion) will be explained. For example, the systems and methods described herein use sensor data and a set of microphones placed on a 3D device to capture multiple DOA information from a 3D sound source captured by a microphone in a physical 2D plane. The microphone signal can be selected based on DOA information retrieved from the microphone, which maximizes the spatial resolution of the sensor in the 2D physical plane. , Sensor data provides a reference for the orientation of the 3D device with respect to the physical 2D plane. There are many use cases that can benefit from the fusion of sensors such as accelerometers and proximity sensors with multi-microphones. One example (eg, "Case 1") may include a robust Handset Intelligent Switch (IS). Another example (eg, "Case 2") may include robust support for various speakerphone retention patterns. Another example (eg, "Case 3") may include seamless speakerphone handset retention pattern support. Yet another example (eg, "Case 4") may include multi-view visualization of the active source and coordination passing.
[00319] Some configurations of the systems and methods described herein may optionally include at least one statistical model to distinguish the desired use case with pre-obtainable sensor data. .. The available sensor data can be tracked along with the multi-microphone data and can be utilized for at least one of these use cases. Some configurations of the systems and methods disclosed herein may, in addition, or instead, track sensor data along with other sensor data (eg, camera data) for at least one use case.
[00320] Next, various configurations will be described with reference to the figures, in which case similar reference numbers may indicate functionally similar elements. The systems and methods generally described and illustrated herein can be configured and designed in a wide variety of different configurations. Therefore, the following more detailed description of some of the configurations shown in the figure is not limiting the scope of claims, but merely represents the system and method. One or more functions and / or elements shown in the figure may be combined with at least one function and / or element shown in at least one other figure.
[00321] FIG. 56 is a block diagram showing a configuration of an electronic device 5602 in which a system and method for mapping source locations may be implemented. The systems and methods disclosed herein can be applied to various electronic devices 5602. Examples of electronic devices 5602 include mobile phones, smartphones, voice recorders, camcorders, audio players (eg Moving Picture Experts Group-1 (MPEG-1) or MPEG-2 Audio). Layer3 (MP3) players), video players, audio recorders, desktop computers, laptop computers, personal digital assistants (PDAs), game systems, etc. A type of electronic device 5602 is a communication device that can communicate with another device. Examples of communication devices include phones, laptop computers, desktop computers, cellular phones, smartphones, wireless or wired modems, electronic readers, tablet devices, gaming systems, cellular telephone base stations or nodes, access points, wireless gateways and wireless routers. and so on.
[00322] Electronic devices 5602 (eg, communication devices) are International Telecommunication Union (ITU) standards and / or Institute of Electrical and Electronics Engineers (IEEE) standards (eg 802.11a, 802.11b, 802.11g, 802.11n, It can operate according to several industrial standards, such as 802.11ac, 802.11 Wireless Fidelity or the "Wi-Fi®" standard). Other examples of standards that communication devices can comply with are IEEE802.16 (eg, Worldwide Interoperability for Microwave Access, or "WiMAX®"), 3GPP, 3GPP LTE, and 3rd Generation Partnership. Can comply with Project2 (3GPP2), GSM, and others (in this case, communication devices are, for example, user equipment (UE), NodeB, evolved NodeB (eNB), mobile devices, mobile stations, subscriber stations, remote stations. , Access terminals, mobile terminals, terminals, user terminals, and / or sometimes referred to as subscriber units, etc.). Some of the systems and methods disclosed herein can be described with respect to at least one standard, as this is because those systems and methods may be applicable to many systems and / or standards. The scope of disclosure should not be limited.
[00323] The electronic device 5602 may include at least one sensor 5604, a mapper 5610, and / or an operating block / module 5614. The phrase "block / module" as used herein indicates that a particular component can be implemented in hardware (eg, a circuit), software, or a combination of both. For example, the operating block / module 5614 may be implemented using hardware components such as circuits and / or software components such as instructions or codes. In addition, one or more of the components or elements of the electronic device 5602 may be implemented in hardware (eg, circuits), software, firmware, or a combination thereof. For example, the mapper 5610 can be implemented in circuits such as application specific integrated circuits (ASICs), field programmable gate arrays (FFPGAs), and / or one or more processors.
[00324] At least one sensor 5604 can collect data about the electronic device 5602. At least one sensor 5604 can be included within the electronic device 5602 and / or coupled to the electronic device 5602. Examples of sensors 5604 include microphones, accelerometers, gyroscopes, compasses, infrared sensors, tilt sensors, Global Positioning System (GPS) receivers, proximity sensors, cameras, ultrasonic sensors and the like. In some implementations, at least one sensor 5604 can provide sensor data 5608 to the mapper 5610. Examples of sensor data 5608 include audio signals, accelerometer readings, gyroscope readings, position information, orientation information, position information, proximity information (eg, whether an object is detected near the electronic device 5602), an image. and so on.
[00325] In some configurations (discussed in more detail below), the mapper 5610 can use sensor data 5608 to improve audio processing. For example, a user can hold an electronic device 5602 (eg, a telephone) in different orientations (eg, portrait, landscape, or even desktop hands-free) for speakerphone use. Depending on the retention pattern (eg, electronic device 5602 orientation), the electronic device 5602 can select the appropriate microphone configuration (including the single microphone configuration) to improve spatial audio processing. With the addition of accelerometer / proximity sensor data 5608, the electronic device 5602 can switch seamlessly.
[00326] Sensor 5604 (eg, multiple microphones) can receive one or more audio signals (eg, multi-channel audio signals). In some implementations, depending on the configuration, the microphone can be placed in various positions on the electronic device 5602. For example, the microphone may be placed on the front, side, and / or back of the electronic device 5602, as illustrated above in FIG. In addition, or instead, the microphone may be located near the top and / or bottom of the electronic device 5602. In some cases, the microphone may be configured to be inoperable (eg, not receive an audio signal). For example, in some cases, the electronic device 5602 may include a circuit that renders at least one microphone inoperable. In some implementations, one or more microphones can be inoperable based on the electronic device 5602 orientation. For example, when the electronic device 5602 is horizontally facing up on the surface (eg, tabletop mode), the electronic device 5602 operates at least one microphone located on the back of the electronic device 5602. Can be disabled. Similarly, if the direction of the electronic device 5602 changes (eg, in large numbers), the electronic device 5602 can render at least one microphone inoperable.
[00327] Given a small number of examples of various microphone configurations: In one example, the electronic device 5602 may be configured to use a dual microphone configuration when possible. As the user orients the electronic device 5602 (eg, the phone) so that the normal vector is parallel to the display or nearly parallel to the ground (for example, the electronic device 5602 is oriented vertically). The electronic device 5602 can use a dual microphone configuration in a Category A configuration, as long as it does not retain its appearance (which can be determined based on sensor data 5608). In some implementations, in a Category A configuration, the electronic device 5602 is a dual microphone in which one microphone can be located near the upper back of the electronic device 5602 and the other microphone can be located near the lower front of the electronic device 5602. May include configurations. In this configuration, the electronic device 5602 may be able to distinguish between audio signal sources in the plane containing the lines formed by the location of the microphone (eg, determine the direction of arrival of the audio signal). Based on this configuration, the electronic device 5602 may be able to distinguish between 180 degree audio sources. Therefore, the direction of arrival of an audio signal arriving within the 180 degree range can be distinguished based on the two microphones in the Category A configuration. For example, the audio signal received from the left of the display of the electronic device 5602 can be distinguished from the audio signal received from the right. In some configurations, the orientation of one or more audio signals can be determined as described in Section A above.
In another example, the user orients the electronic device 5602 (for example, a telephone) so that the normal vector to the display is orthogonal or approximately orthogonal to the ground (for example, the electronic device 5602 is oriented horizontally). The electronic device 5602 can use a dual microphone configuration in a Category B configuration, as long as it does not retain its appearance (which can be signaled by sensor data 5608). In this configuration, the electronic device 5602 may include a dual microphone configuration in which one microphone may be located near the lower back of the electronic device 5602 and the other microphone may be located near the lower front of the electronic device 5602. In some implementations, in a Category B configuration, one microphone may be located near the top back of the electronic device 5602 and the other microphone may be located near the top front of the electronic device 5602.
In a Category B configuration, the audio signal can be distinguished in a plane containing the lines formed by the position of the microphone (eg, the direction of arrival of the audio signal can be determined). Based on this configuration, there can be a 180 degree audio source distinction. Therefore, the direction of arrival of an audio signal arriving within the 180 degree range can be distinguished based on the two microphones in the Category B configuration. For example, the audio signal received from the top of the display of the electronic device 5602 can be distinguished from the audio signal received from the bottom. However, the two audio signals to the left or right of the display of the electronic device 5602 cannot be identified. Audio signals from the left and right of the electronic device's display can be identified if the electronic device's orientation 102 is redirected so that the electronic device 5602 is oriented vertically instead of horizontally. Please note. In the case of a three-microphone configuration, i.e. Category C, the electronic device 5602 can use a pair of front and back microphones in the vertical direction and a pair of top and bottom microphones in the horizontal direction. it can. Using configurations such as those in Category C, the electronic device 5602 may be able to distinguish audio sources at 360 degrees (eg, distinguish directions of arrival from different audio signals).
[00330] Mapper 5610 is based on sensor data 5608, the source position relative to the electronic device 5602 coordinates, and the mapping from the electronic device 5602 coordinates to physical coordinates (eg, a two-dimensional plane corresponding to real world world coordinates or earth coordinates). 5612 can be judged. Mapping 5612 may include data indicating a mapping (eg, projection) of the source position to and / or physical coordinates to electronic device coordinates. For example, Mapper 5610 may implement at least one algorithm to map source positions to physical coordinates. In some implementations, the physical coordinates can be two-dimensional physical coordinates. For example, the mapper 5610 determines the direction of the electronic device 5602 (eg, retention pattern) and performs actions (eg, displaying source position, switching microphone configurations, and / or configuring noise suppression settings). Sensor data 5608 (eg, integrated acceleration data, proximity data, and microphone data) from at least one sensor 5604 may be used to instruct the electronic device 5602 to do so.
[00331] The mapper 5610 can detect changes in the direction of the electronic device 5602. In some implementations, the movement of an electronic device 5602 (eg, a telephone) can be detected via a sensor 5604 (eg, an accelerometer and / or a proximity sensor). The mapper 5610 can take advantage of these movements and the electronic device 5602 can adjust the microphone configuration and / or noise suppression settings based on the range of rotation. For example, the mapper 5610 may receive sensor data 5608 from at least one sensor 5604 indicating that the electronic device 5602 has changed from horizontal (eg, tabletop mode) to vertical (eg, browse talk mode). In some implementations, the mapper 5610 redirects the electronic device 5602 (eg, a wireless communication device) from handset mode (eg, the side of the user's head) to browse talk mode (eg, in front of the user). Can show that you did.
[00332] The electronic device may also include an action block / module 5614 that performs at least one action based on the mapping 5612. For example, the motion block / module 5614 can be coupled to at least one microphone and the microphone configuration can be switched based on the mapping 5612. For example, if the mapping 5612 indicates that the electronic device 5602 has changed from vertical (eg, browse talk mode) to table-up horizontal (eg, tabletop mode), the motion block / module 5614 is electronic. At least one microphone located on the back of the device can be disabled. Similarly, as described below, the operating block / module 5614 can switch a multi-microphone configuration to a single-microphone configuration. Other examples of operation include tracking a 2D or 3D source, projecting the source into a 3D display space, and performing transient noise suppression.
[00333] FIG. 57 is a flow diagram illustrating a configuration of method 5700 for mapping the coordinates of an electronic device 5602. Method 5700 can be performed by electronic device 5602. Electronic device 5602 can acquire sensor data 5608 (5702). At least one sensor 5604 coupled to the electronic device 5602 can provide sensor data 5608 to the electronic device 5602. Examples of sensor data 5608 include audio signals (eg from one or more microphones), accelerometer readings, position information, directional information, position information, proximity information (eg objects detected near electronic device 5602). Whether it will be done), images, etc. In some implementations, the electronic device 5602 uses sensor data 5608 (eg, for example) with pre-acquired data and corresponding microphone identification for each specified electronic device 5602 direction (eg, retention pattern). Accelerometer xyz coordinates) can be obtained (5702).
[00334] Electronic device 5602 can map source positions to electronic device coordinates based on sensor data (5704). This can be achieved as described above for one or more of FIGS. 41-48. For example, the electronic device 5602 can estimate the source arrival direction (DOA) with respect to electronic device coordinates based on a multichannel signal (eg, multiple audio signals from two or more microphones). In some schemes, mapping the source position to electronic device coordinates (5704) projects the direction of arrival onto a plane (eg, projective plane and / or array plane), as explained above. Can include. In some configurations, the electronic device coordinates can be the microphone array plane corresponding to the device. In other configurations, the electronic device coordinates may be another coordinate system corresponding to the electronic device 5602, to which the source position (eg, DOA) can be mapped (eg, transformed and / or rotated) by the electronic device 5602.
[00335] Electronic device 5602 can map source positions from electronic device coordinates to physical coordinates (eg, two-dimensional physical coordinates) (5706). This can be achieved as described above for one or more of FIGS. 49-53. For example, an electronic device can be implemented to utilize a directional matrix to project DOA estimates into a plane defined with reference to a world (or earth) coordinate system.
[00336] In some configurations, the mapper 5610 contained within the electronic device 5602 maps the source position to the electronic device coordinates (5704), and the source position from the electronic device coordinates electronic device 5602 coordinates to the physical coordinates. At least one algorithm can be implemented for mapping (5706). In some configurations, the mapping 5612 may be applied to a "3D audio map". For example, in some configurations, the compass (eg, sensor 5604) can provide compass data (eg, sensor data 5608) to mapper 5610. In this example, the electronic device 5602 can obtain a sound distribution map in four pi directions (eg, a sphere) converted to physical (eg, real world or earth) coordinates. This may allow the electronic device 5602 to describe the 3D audio space. This type of ascending information can be used to reproduce the ascending sound via a loudspeaker located within the ascending position (eg, such as a 22.2 surround system).
[00337] In some implementations, mapping the source position from electronic device coordinates to physical coordinates (5706) detects the electronic device 5602 direction and / or detects any change in the electronic device 5602 direction. Can include that. For example, the mapper 5610 has sensor data 5608 (eg, integrated accelerometer data, proximity data, and microphone data) from at least one sensor 5604 to determine the electronic device 5602 direction (eg, retention pattern). ) Can be used. Similarly, the mapper 5610 may receive sensor data 5608 from at least one sensor 5604 indicating that the electronic device 5602 has changed from horizontal (eg, tabletop mode) to vertical (eg, browse talk mode). ..
[00338] Electronic device 5602 may perform operations based on mapping 5612 (5708). For example, electronic device 5602 may perform at least one operation based on the electronic device 5602 direction (indicated by mapping 5612, for example) (5708). Similarly, the electronic device 5602 may perform an operation based on the detected change in the direction of the electronic device 5602 (indicated by, for example, mapping 5612) (5708). Specific examples of behavior include switching electronic device 5602 microphone configurations, tracking audio sources (for example, in 2D or 3D), mapping source positions from physical coordinates to 3D display space, and non- These include stationary noise suppression, filtering, and displaying images based on audio signals.
[00339] An example of mapping the source position from electronic device coordinates to physical coordinates (5706) is given as follows. According to this example, the electronic device 5602 (eg, mapper 5610) monitors sensor data 5608 (eg, accelerometer coordinate data) and smoothes sensor data 5608 (simple recursive weighting). ) Or Kalman smoothing), and the motion block / module 5614 can perform actions based on the mapping 5612 (eg, mapping or projecting an audio source).
[00340] The electronic device 5602 has a xyz basis vector in the coordinate system given by the form factor (eg FLUID) (eg by using a gyro sensor).<maths num="71"><img id="000072" he="27" wi="78" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
You can get the three-dimensional (3D) space defined by. The electronic device 5602 is a basis vector in a physical (eg, real-world) coordinate system based on xyz position sensor data 5608.<maths num="72"><img id="000073" he="26" wi="81" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Can be specified. The electronic device 5602 is then<maths num="73"><img id="000074" he="28" wi="44" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Can be obtained, which is the basis vector space for obtaining any two-dimensional plane in the coordinate system. Search grid<maths num="74"><img id="000075" he="27" wi="44" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Considering that, the electronic device 5602 takes the basis vector space by utilizing the first two elements of the projective motion defined by utilizing the first two elements (x ", y"). Can project onto a surface (x ", y"), where<maths num="75"><img id="000076" he="27" wi="104" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Is.
[00341] For example, assuming the device (eg, phone) is held in browse talk mode, E = ([1 0 0]]<sup>T</sup>,[0 1 0]<sup>T</sup>,[0 0 1]<sup>T</sup>) And E'= ([0 0 1]<sup>T</sup>,[0 1 0]<sup>T</sup>,[1 0 0]<sup>T</sup>). In this case, within the device (eg phone) coordinate system<maths num="76"><img id="000077" he="27" wi="77" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
and<maths num="77"><img id="000078" he="27" wi="91" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Is. It is the real xy plane (for example, physical coordinates), A = ([1 0 0]<sup>T</sup>,[0 1 0]<sup>T</sup>) To project<maths num="78"><img id="000079" he="33" wi="94" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
Is. The first two elements [1 0]<sup>T</sup>Can be used after the projection operation. Therefore, of E<maths num="79"><img id="000080" he="26" wi="39" file="JP6400566B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
At this time, [1 0]<sup>T</sup>Note that it can be projected onto A as. This has a browse talk mode in the device (eg phone) xyz shape [0 0 1]<sup>T</sup>Is about the real world xy plane [1 0]<sup>T</sup>Corresponds to.
[00342] For a less complex approximation of projection, the electronic device 5602 has P (x', y') = αP.<sub>xy</sub>(x', y') + βP<sub>xz</sub>(x', y') + γP<sub>yz</sub>A simple interpolation method in the three set representations defined as (x', y') can be applied, where α + β + γ = 1, which is the real xy plane and each set plane. It is a function of the angle between and. Alternatively, the electronic device 5602 has P (x', y') = min (P).<sub>xy</sub>(x', y;), P<sub>xz</sub>(z', y'), P<sub>yz</sub>(x', y<sub>’</sub>) Can be used. In this mapping example, the coordinate change position is shown before the projection operation.
[00343] In addition, or instead, performing an operation (5708) may include mapping the source position from physical coordinates into a three-dimensional display space. This can be achieved as described above for one or more of FIGS. 52-53. Additional examples are provided below. For example, the electronic device 5602 can render a sound source representation corresponding to a source position in a three-dimensional display space. In some configurations, the electronic device 5602 can render plots containing sound source representations (eg, polar plots, rectangular plots) on a 2D plane that corresponds to physical coordinates in 3D display space, in this case. , The plane is rendered based on the device orientation. Performing an operation in this way (5708) may include keeping the source orientation within the 3D display space regardless of the device orientation (eg, rotation, tilt, pitch, yaw, roll, etc.). .. For example, the plot will be aligned with the physical coordinates regardless of how the device is oriented. In other words, the electronic device 5602 can compensate for device orientation changes to maintain plot behavior with respect to physical coordinates. In some configurations, displaying a 3D display space can include projecting the 3D display space onto a 2D display (for example, to display a 2D pixel grid).
[00344] FIG. 58 is a block diagram illustrating a more specific configuration of the electronic device 5802 in which a system and method for mapping the electronic device 5802 coordinates may be implemented. The electronic device 5802 may be an example of the electronic device 5602 described with respect to FIG. 56. The electronic device 5802 may include at least one sensor 5804, at least one microphone, a mapper 5810, and an operating block / module 5814, which may be examples of the corresponding elements described with respect to FIG. In some implementations, at least one sensor 5804 can provide the mapper 5810 with sensor data 5808, which may be an example of the sensor data 5608 described with respect to FIG. 56.
[00345] The operating block / module 5814 can receive the reference direction 5816. In some implementations, the reference direction 5816 can be contained within the electronic device 5802 and / or stored in memory that can be coupled to the electronic device 5802. The reference direction 5816 can indicate the reference electronic device 5602 direction. For example, reference direction 5816 can indicate the optimal electronic device 5602 orientation (eg, optimal retention pattern). The optimal electronic device 5602 orientation may correspond to the orientation in which a dual microphone configuration can be implemented. For example, the reference direction 5816 can be the direction in which the electronic device 5602 is located between the vertical and horizontal directions. In some embodiments, the horizontal and vertical electronic device 5602 (eg, telephone) orientation is an uncommon retention pattern (eg, non-optimal electronic device 5602 orientation). These positions (eg, vertical and / or horizontal) can be identified using a sensor 5804 (eg, accelerometer). In some implementations, the intermediate position (which may include the reference direction 5816) can be a position for vertical dual microphone noise suppression. By comparison, horizontal and / or vertical can be handled by lateral / single microphone noise suppression.
[00346] In some implementations, the working block / module 5814 is a 3D source projection block / module 5818, a 2D source tracking block / module 5820, a 3D source tracking block / module 5822, a microphone configuration switch 5824, and / Or may include a non-stationary noise suppression block / module 5826.
[00347] The 3D source tracking block / module 5822 can track an audio signal source in 3D. For example, as the audio source moves relative to the electronic device 5602, or as the electronic device 5602 moves relative to the audio source, the 3D source tracking block / module 5822 position the audio source with respect to the electronic device 5802. Can be tracked in three dimensions. In some implementations, the ternary source tracking block / module 5822 can track the audio signal source based on the mapping 5812. In other words, the 3D source tracking block / module 5822 can determine the position of the audio signal source with respect to the electronic device based on the electronic device 5802 direction shown in the mapping 5812. In some implementations, the 3D source projection block / module 5818 can project a source (eg, a source tracked in 3D) into 2D space. For example, the 3D Source Projection Block / Module 5818 can use at least one algorithm to project a 3D tracked source for 2D display purposes.
[00348] In this implementation, the 2D source tracking block / module 5820 can track the source in 2D. For example, as the audio source moves relative to the electronic device 5602, or as the electronic device 5602 moves relative to the audio source, the 2D source tracking block / module 5820 places the audio signal source relative to the electronic device 5802. Can be tracked in two dimensions. In some implementations, the 2D source tracking block / module 5820 can track audio sources based on the mapping 5812. In other words, the 2D source tracking block / module 5820 can determine the position of the audio signal source with respect to the electronic device based on the electronic device 5802 direction shown in the mapping 5812.
[00349] The microphone configuration switch 5824 can switch the electronic device 5802 microphone configuration. For example, the microphone configuration switch 5824 can enable / disable at least one of the microphones. In some implementations, the microphone configuration switch 5824 can switch the microphone configuration 306 based on the mapping 5812 and / or the reference direction 5816. For example, if the mapping 5812 indicates that the electronic device 5802 is horizontal on a face-up plane (eg, browse talk mode), then the microphone configuration switch 5824 is located on the back of the electronic device 5802. At least one microphone can be disabled. Similarly, when the mapping 5812 indicates that the direction of the electronic device 5802 differs from the reference direction 5816 (eg, by a certain amount), the microphone configuration switch 5824 is from a multi-microphone configuration (eg, a dual microphone configuration) to a single microphone configuration. Can be switched to.
[00350] In addition, or instead, the transient noise suppression block / module 326 may perform transient noise suppression based on the mapping 5812. In some implementations, the transient noise suppression block / module 5826 can perform transient noise suppression regardless of the direction of the electronic device 5802. For example, unsteady noise suppression can include spatial processing such as beam null formation and / or directional masking discussed above.
[00351] FIG. 59 is a flow diagram illustrating a more specific configuration of method 5900 for mapping the coordinates of an electronic device 5802. Method 5900 can be performed by electronic device 5802. The electronic device 5802 can acquire 5902 sensor data 5808. In some implementations this can be done as described with respect to FIG. 57.
[00352] The electronic device 5802 can determine the mapping (5812) of the electronic device 5802 coordinates from the multi-microphone configuration to the physical coordinates based on the sensor data 5808 (5904). In some implementations this can be done as described with respect to FIG. 57.
[00353] The electronic device 5802 can determine the orientation of the electronic device based on the mapping 5812 (5906). For example, the mapper 5810 can receive sensor data 5808 from a sensor 5804 (eg, an accelerometer). In this example, the mapper 5810 can use the sensor data 5808 to determine the orientation of the electronic device 5802. In some implementations, the orientation of the electronic device 5802 can be based on a reference plane. For example, electronic device 5802 can use polar coordinates to define the orientation of electronic device 5802. As described below, the electronic device 5802 can perform at least one operation based on the orientation of the electronic device 5802.
[00354] In some implementations, the electronic device 5802 can provide a real-time source activity map to the user. In this example, the electronic device 5802 can utilize a sensor 5804 (eg, an accelerometer and / or a gyroscope) to determine the orientation of the electronic device 5802 (eg, the user's retention pattern) (5906). .. The amount of change in likelihood (direction) can be given by a two-dimensional (2D) anglogram (or polar plot) for each direction (eg, retention pattern) of each electronic device 5802. In some cases, if the electronic device 5802 faces orthogonally to the plane created by the two pairs, the amount of change can be quite large (omnidirectional).
[00355] In some implementations, the electronic device 5802 can detect any change in the orientation of the electronic device 5802 based on the mapping 5812 (5908). For example, the mapper 5810 can monitor the orientation of the electronic device 5802 over time. In this example, the electronic device 5802 can detect any change in the orientation of the electronic device 5802 (5908). For example, the mapper 5810 indicates that the electronic device 5802 (for example, a wireless communication device) has changed direction from handset mode (for example, the side of the user's head) to browse talk mode (for example, in front of the user). Can be done. As described below, the electronic device 5802 can perform at least one operation based on any change in the direction of the electronic device 5802.
[00356] Optionally, the electronic device 5802 (eg, operating block / module 5814) can determine if there is a difference between the orientation of the electronic device 5802 and the reference direction 5816 (5910). For example, electronic device 5802 can receive a mapping 5812 indicating the direction of electronic device 5802. The electronic device 5802 may also include a reference direction 5816. If the orientation of the electronic device 5802 and the reference direction 5816 are not the same, the electronic device 5802 can determine that there is a difference between the orientation of the electronic device 5802 and the reference direction 5816. As described below, the electronic device 5802 can perform at least one operation based on the difference between the orientation of the electronic device 5802 and the reference direction 5816. In some implementations, determining if there is a difference between the direction of the electronic device 5802 and the reference direction 5816 (5910) determines if any difference is greater than the threshold amount. May include doing. In this example, the electronic device 5802 can perform an operation based on the difference when the difference is greater than the threshold amount.
[00357] In some implementations, electronic device 5802 can switch microphone configurations based on the orientation of electronic device 5802 (5912). For example, the electronic device 5802 can select a microphone signal that maximizes the spatial resolution of one or more sources in physical coordinates (eg, a 2D physical plane) based on DOA information. Switching microphone configurations (5912) can include enabling / inoperable microphones located at various locations on the electronic device 5802.
Switching microphone configurations (5912) may be based on mapping 5812 and / or reference direction 5816. In some configurations it is possible to switch between different microphone configurations (5912), but often there is some systematic delay, such as when switching from a dual microphone configuration to a single microphone configuration (5912). May include. For example, in the presence of abrupt changes in the direction of the electronic device 5802, the systematic delay can be approximately 3 seconds. Switching from a dual microphone configuration to a single microphone configuration (5912) can be done seamlessly by baseing the switch 5912 on the mapping 5812 (eg, also sensor data 5808). In some implementations, switching the microphone configuration based on the mapping 5812 and / or the reference direction 5816 (5912) can be the orientation of the electronic device 5802, any change in the orientation of the electronic device 5802, and the orientation of the electronic device 5802. It may include switching microphone configurations (5912) based on at least one of any differences between and the reference direction 5816.
[00359] Switching the microphone configuration (5912) A decimal example is given as follows. In one example, the electronic device 5802 can be in the reference direction 5816 (eg, optimal retention pattern). In this example, the electronic device 5802 may learn sensor data 5808 (eg, accelerometer xyz coordinates). This is, for example, a simple weighted average (eg alpha).<sup>*</sup>history + (1-alpha)<sup>*</sup>It can be based on current) or more elaborate Kalman smoothing. If the electronic device 5802 determines from the tracked accelerometer statistics and the reference direction 5816 that there is a significant difference (5910), the electronic device 5802 can switch from multiple microphone configurations to a single microphone configuration (5912). ).
[00360] Another example assumes that the user changes posture (for example, from sitting in a chair to lying on a bed). If the user holds the electronic device 5802 (eg, a telephone) in an acceptable retention pattern (eg, the electronic device 5802 is in reference direction 5816), the electronic device 5802 continues to have multiple microphone configurations (eg, dual microphone configurations). ), And can learn accelerometer coordinates (for example, acquire sensor data 5808). Further, the electronic device 5802 may detect the user's posture while the user is talking on the telephone, for example by detecting the direction of the electronic device 5802. Assume that the user does not speak while the user removes the electronic device 5802 (eg, telephone) from his mouth. In this case, the electronic device 5802 can switch from a plurality of microphone configurations to a single microphone configuration (5912), and the electronic device 5802 can remain in the single microphone configuration state. However, as soon as the user speaks while maintaining the electronic device 5802 in the optimal retention pattern (eg, in the reference direction 5816), the electronic device 5802 will switch back to multiple microphone configurations (eg, dual microphone configurations).
[00361] In another example, the electronic device 5802 can be in a horizontal orientation with the electronic device 5802 facing down (for example, while the display of the electronic device 5802 is facing down on the bed, the user holds the electronic device down. Lying on the bed while holding). Since the z coordinate is negative as sensed by the sensor 5804 (eg, accelerometer), the electronic device 5802 direction can be easily detected. In addition, or instead, if the user's posture changes from a sitting position to a lying position on the bed, the electronic device 5802 also uses a frame that uses phase and level differences to allow the user. Can learn the posture of. As soon as the user uses the electronic device 5802 in the reference direction 5816 (eg, holds the electronic device 5802 in an optimal holding pattern), the electronic device 5802 may perform optimal noise suppression. The sensor 5804 (eg, integrated accelerometer and microphone data) can then be used within the mapper 5810 to determine the direction of the electronic device 5802 (eg, the retention pattern of the electronic device 5802), and the electronic device 5802 operates. It is possible to do (for example, choose the appropriate microphone configuration). More specifically, the front and rear microphones can be made operational, or the front microphones can be made operational and at the same time the rear microphones can be inoperable. One of these configurations can be implemented while the electronic device 5802 is in the horizontal orientation (eg, speakerphone or tabletop mode).
[00362] In another example, the user may change the electronic device 5802 (eg, telephone) retention pattern (eg, electronic device 5802 direction) from using a handset to using a speakerphone, and vice versa. By adding accelerometer / proximity sensor data 5808, the electronic device 5802 seamlessly switches microphone configurations to adjust microphone gain and speaker volume (or switch earpieces to larger loudspeakers). Can be done. For example, suppose a user puts the electronic device 5802 (for example, a telephone) table down. In some implementations, the electronic device 5802 tracks the sensor 5804 so that the electronic device 5802 can track whether the electronic device 5802 (for example, a telephone) is facing down or up. Can be done. If the electronic device 5802 (eg, telephone) is face down, the electronic device 5802 may provide speakerphone functionality. In some implementations, the electronic device may prioritize proximity sensor results. In other words, if the sensor data 5808 indicates that the object (eg, hand or desk) is near the ear, the electronic device does not have to switch to the speakerphone (5912).
[00363] Optionally, the electronic device 5802 can track the source in three dimensions based on the mapping 5812 (5914). For example, as an audio source moves relative to an electronic device 5802, the electronic device 5802 can track the audio source in three dimensions. In this example, the electronic device 5802 can project a source (eg, source position) into two-dimensional space (5916). For example, electronic device 5802 can project a source tracked in 3D onto a 2D display within electronic device 5802 (5916). In addition, the electronic device 5802 can switch to tracking the source in two dimensions (5918). For example, as an audio source moves relative to an electronic device 5802, the electronic device 5802 can track the audio source in two dimensions. Depending on the orientation of the electronic device 5802, the electronic device 5802 can select the corresponding non-linear pair of microphones and use the appropriate 2D projection to provide a 360 degree 2D representation. For example, the electronic device 5802 can be a two-dimensional 360 degree regardless of the orientation of the electronic device 5802 (for example, the holding pattern (speakerphone mode, vertical browse talk mode, and horizontal browse talk mode, or any combination thereof)). Visualization of source activity can be achieved. If the electronic device 5802 is in the middle of each retention pattern, the visualization can be interpolated for a two-dimensional representation. In fact, the electronic device 5802 is a two-dimensional set of three. You can even use representations to render 3D visualizations.
[00364] In some implementations, the electronic device 5802 may perform transient noise suppression (5920). Performing transient noise suppression (5920) can suppress noise audio signals from the target audio signal in order to improve spatial audio processing. In some implementations, the electronic device 5802 can move during noise suppression. In these implementations, electronic device 5802 may perform unsteady noise suppression regardless of the orientation of electronic device 5802 (5920). For example, if a user accidentally rotates a phone but still wants to focus on some target orientation, it can be beneficial to maintain the target orientation regardless of the device orientation.
[00365] FIG. 60 is a flow diagram illustrating one configuration of method 6000 for performing an operation based on mapping 5812 (5708). Method 6000 can be performed by electronic device 5802. Electronic device 5802 may detect any change in sensor data 5808 (6002). In some implementations, detecting any change in sensor data 5808 (6002) may include detecting whether the change in sensor data 5808 is greater than a certain amount. For example, the electronic device 5802 can determine if there is a change in accelerometer data that is greater than the determined threshold amount (6002).
[00366] Electronic device 5802 indicates whether sensor data 5808 indicates that electronic device 5802 is in horizontal or vertical position, or that electronic device 5802 is in intermediate position. Can be judged (6004). For example, the electronic device 5802 indicates that the sensor data 5808 indicates that the electronic device 5802 is in tabletop mode (eg, face up and horizontal), or browse talk mode (eg, eye level). It can be determined whether it indicates that it is (perpendicular to) or that the electronic device 5802 is in a position other than vertical or horizontal (which may include, for example, the reference direction 5816).
[00367] If electronic device 5802 determines that sensor data 5808 indicates that electronic device 5802 is in an intermediate position (6004), electronic device 5802 may use a dual microphone configuration (6006). If the electronic device 5802 did not previously use a dual microphone configuration, using a dual microphone configuration (6006) may include switching to a dual microphone configuration. By comparison, if the electronic device 5802 previously used a dual microphone configuration, using the dual microphone configuration (6006) may include maintaining the dual microphone configuration.
[00368] If electronic device 5802 determines that sensor data 5808 indicates that electronic device 5802 is in a horizontal or vertical position (6004), electronic device 5802 is a short-range phase / gain voice activity detector (VAD). ) Can be determined if it is active (6008). In other words, the electronic device 5802 can determine if the electronic device 5802 is located near an audio signal source (eg, the user's mouth). If the electronic device 5802 determines that the short-range phase / gain voice activity detector is active (for example, the electronic device 5802 is near the user's mouth) (6008), the electronic device 5802 has a dual microphone configuration. Can be used (6006).
[00369] If the electronic device 5802 determines that the short-range phase / gain voice activity detector is inactive (for example, the electronic device 5802 is not located near the audio signal source) (6008), the electronic device 5802 will A single microphone configuration can be used (6010). If the electronic device 5802 did not previously use a single microphone configuration, using a single microphone configuration (6010) may include switching to a single microphone configuration. By comparison, if the electronic device 5802 previously used a single microphone configuration, using the single microphone configuration (6010) may include maintaining the single microphone configuration. In some implementations, using a single microphone configuration (6010) may include using lateral / single microphone noise suppression.
[00370] FIG. 61 is a flow diagram illustrating another configuration of method 6100 for performing an operation based on mapping 5812 (5708). Method 6100 can be performed by electronic device 5802. Electronic device 5802 can detect any changes in sensor data 5808 (6102). In some implementations this can be done as described with respect to FIG.
[00371] Electronic device 5802 may determine whether sensor data 5808 indicates that electronic device 5802 is in a tabletop position, or in an intermediate or vertical position (6104). For example, the electronic device 5802 indicates that the sensor data 5808 indicates that the electronic device 5802 is horizontal on the surface face up (eg, table top position), or that the electronic device 5802 is vertical (eg, vertical). , Browse talk position), or can be determined to indicate that it is in a position other than vertical or horizontal (which may include, for example, the reference direction 5816) (6104).
[00372] If electronic device 5802 determines that sensor data 5808 indicates that electronic device 5802 is in an intermediate position (6104), electronic device 5802 may use front and rear microphones (6106). In some implementations, using front and rear microphones (6106) can make at least one microphone operational / inoperable.
[00373] If electronic device 5802 determines that sensor data 5808 indicates that electronic device 5802 is in a tabletop position (6104), electronic device 5802 is whether the table of electronic device 5802 is facing up. Can be determined (6108). In some implementations, the electronic device 5802 may determine whether the table of the electronic device 5802 is facing up based on sensor data 5808 (6108). If the electronic device 5802 determines that the table of the electronic device 5802 is facing up (6108), the electronic device 5802 may use a front microphone (6110). For example, the electronic device may use at least one microphone located in front of the electronic device 5802 (6110). In some implementations, using a front microphone (6110) can make at least one microphone operational / inoperable. For example, using a front microphone (6110) can disable at least one microphone located on the back of the electronic device 5802.
[00374] If the electronic device 5802 determines that the table of the electronic device 5802 is not facing up (for example, the table of the electronic device 5802 is facing down) (6108), the electronic device 5802 uses the rear microphone. Can be used (6112). For example, the electronic device may use at least one microphone located on the back of the electronic device 5802 (6112). In some implementations, using a rear microphone (6112) can make at least one microphone operational / inoperable. For example, using a rear microphone (6112) may include disabling at least one microphone located in front of the electronic device 5802.
[00375] FIG. 62 is a block diagram illustrating a configuration of user interface 6228 in which a system and method for displaying user interface 6228 on electronic device 6202 may be implemented. In some implementations, user interface 6228 may be displayed on electronic device 6202, which may be an example of the electronic device 5602 described with respect to FIG. 56. User interface 6228 may be used with and / or independently of the multi-microphone configuration described herein. User interface 6228 may be presented on display 6264 (eg, screen) of electronic device 6202. Display 6264 can also present the sector selection function 6232. In some implementations, user interface 6228 may provide editable mode and fixed mode. In editable mode, user interface 6228 may respond to input for manipulating at least one function of user interface 6228 (eg, sector selection function). In fixed mode, user interface 6228 cannot respond to input to operate at least one function of user interface 6228.
[00376] User interface 6228 may include information. For example, user interface 6228 may include coordinate system 6230. In some implementations, the coordinate system 6230 can be a reference for audio signal source location. Coordinate system 6230 may correspond to physical coordinates. For example, sensor data 5608 (eg, accelerometer data, gyro data, compass data, etc.) can be used to map the electronic device 6202 coordinates to physical coordinates, as described in FIG. In some implementations, the coordinate system 6230 may correspond to a physical space independent of Earth coordinates.
[00377] User interface 6228 can display the directionality of the audio signal. For example, user interface 6228 may include an audio signal indicator indicating the direction of the audio signal source. The angle of the audio signal source can also be shown within the user interface 6228. The audio signal can be an audio signal. In some implementations, the audio signal can be captured by at least one microphone. In this implementation, the user interface 6228 may be coupled to at least one microphone. User interface 6228 may display a 2D anglogram of the captured audio signal. In some implementations, the user interface 6228 can display 2D plots in 3D perspective to convey the alignment of the plot with a plane based on the physical coordinates of the real world, such as a horizontal plane. In this implementation, user interface 6228 may display information irrelevant to the electronic device 6202 direction.
[00378] In some implementations, user interface 6228 may display audio signal indicators for different types of audio signals. For example, user interface 6228 may include anglograms of audio and noise signals. In some implementations, user interface 6228 may include icons corresponding to audio signals. For example, as described below, the display 6264 may include an icon corresponding to the type of audio signal displayed. Similarly, as described below, user interface 6228 may include an icon corresponding to the source of the audio signal. The position of these icons in the polar plot can be smoothed over time. As described below, user interface 6228 may include one or more elements for performing the functions described herein. For example, user interface 6228 may include indicators for selected sectors and / or may display icons for editing selected sectors.
[00379] The sector selection function 6232 may allow selection of at least one sector in the physical coordinate system 6230. The sector selection function 6232 may be implemented by at least one element contained within the user interface 6228. For example, user interface 6228 may include a selected sector indicator that indicates the selected sector. In some implementations, the sector selection function 6232 may operate on the basis of touch input. For example, the sector selection function 6232 selects a sector based on a single touch input (for example, touching, swiping, and / or drawing a circle in an area of user interface 6228 that corresponds to a sector). Can be made possible. In some implementations, the sector selection function 6232 may allow simultaneous selection of multiple sectors. In this example, the sector selection function 6232 may allow the selection of multiple sectors based on multiple touch inputs. Note that electronic device 6202 may include circuits, processors, and / or instructions for creating user interface 6228.
[00380] FIG. 63 is a flow diagram illustrating one configuration of method 6300 for displaying user interface 6228 on electronic device 6202. Method 6300 can be performed by electronic device 6202. The electronic device 6202 may acquire sensor data corresponding to physical coordinates (eg, accelerometer data, tilt sensor data, directional data, etc.) (6302).
[00381] Electronic device 6202 may present user interface 6228, for example, on display 6264 of electronic device 6202 (6304). In some implementations, user interface 6228 may include coordinate system 6230. As described above, the coordinate system 6230 can be a reference for the position of the audio signal source. Coordinate system 6230 may correspond to physical coordinates. For example, as described above , sensor data 5608 (eg, accelerometer data, gyro data, compass data, etc.) can be used to map the electronic device 6202 coordinates to physical coordinates .
[00382] In some implementations, presenting a user interface 6228 that may include coordinate system 6230 (6304) presents user interface 6228 and coordinate system 6230 in a direction independent of the electronic device 6202 direction. Can include that (6304). In other words, when the electronic device 6202 direction changes (eg, the electronic device 6202 rotates), the coordinate system 6230 can maintain its direction. In some implementations, the coordinate system 6230 may correspond to a physical space independent of Earth coordinates.
[00383] Electronic device 6202 may provide a sector selection function 6232 that allows selection of at least one sector in coordinate system 6230 (6306). As described above, electronic device 6202 may provide sector selection functionality via user interface 6228 (6306). For example, user interface 6228 may include at least one element that allows selection of at least one sector in coordinate system 6230. For example, user interface 6228 may include an indicator indicating the selected sector.
[00384] Electronic device 6202 may also include a touch sensor that allows touch input selection in at least one sector. For example, the electronic device 6202 may select (and / or edit) one or more sectors and / or one or more audio signal indicators based on one or more touch inputs. Some examples of touch input include one or more taps, swipes, patterns (eg symbols, shapes, etc.), pinches, spreads, multi-touch rotation, and so on. In some configurations, the electronic device 6202 (eg, user interface 6228) was displayed when one or more taps, swipes, patterns, etc. intersected the displayed audio signal indicator (and / or sector). Audio signal indicators (and / or sectors) can be selected. In addition, or instead, completely or partially encloses or includes an audio signal indicator (and / or sector) displaying a pattern (eg, a circular area, a rectangular area, or an area within the pattern). When the electronic device 6202 (eg, user interface 6228) may select the displayed audio signal indicator (and / or sector). Note that one or more audio signal indicators and / or sectors can be selected at one time.
[00385] In some configurations, the electronic device 6202 (eg, user interface 6228) may edit one or more sectors and / or audio signal indicators based on one or more touch inputs. For example, user interface 6228 edits an audio signal indicator or selected audio signal indicator (eg, selects an icon or image to label the audio signal indicator, colors, patterns, and / or with respect to the audio signal indicator. To select or change an image, set whether the corresponding audio signal should be filtered (for example, blocked or passed), zoom in or out on the displayed audio signal indicator, etc.) You may be presented with one or more options (eg, one or more buttons, drop-down menus, etc.) that provide the options for. In addition, or instead, user interface 6228 edits sectors (eg, selects or modifies colors, patterns, and / or images for sectors, and filters audio signals within sectors (eg, blocked). Provides one or more options for setting whether to (or pass), zooming in or out on a sector, adjusting the sector size (for example, by stretching or contracting a sector). You may be presented with one or more options to do (eg, one or more buttons, drop-down menus, etc.). For example, a pinch-touch input can correspond to reducing or narrowing the sector size, while a spread can correspond to increasing or increasing the sector size.
[00386] Electronic device 6202 may provide sector editing capabilities that allow editing of at least one sector (6308). For example, the sector editing function may allow sector adjustment (eg, enlargement, reduction, shift, etc.) as described herein.
[00387] In some configurations, the electronic device 6202 (eg, display 6264) may additionally, or instead, display the target audio signal and the interfering audio signal on the user interface. The electronic device 6202 (eg, display 6264) may display the orientation of the target audio signal and / or the interfering audio signal captured by one or more microphones. The target audio signal may include an audio signal.
[00388] FIG. 64 is a block diagram illustrating a configuration of a user interface 6428 in which a system and method for displaying the user interface 6428 on an electronic device 6402 may be implemented. In some implementations, the user interface 6428 may be included on the display 6464 of electronic device 6402, which may be an example of the corresponding elements described with respect to FIG. The electronic device 6402 may be an example of the corresponding element described in one or more of FIGS. 56 and 62, user interface 6428, at least one microphone 6406, working block / module 6414, display 6464, And / or may include sector selection function 6432.
[00389] In some implementations, the user interface 6428 may present a sector editing function 6436 and / or a user interface matching block / module 6440. The sector editing function 6436 may allow editing of at least one sector. For example, the sector editing function 6436 may allow editing of at least one selected sector in the physical coordinate system 6430. The sector editing function 6436 can be performed by at least one element contained within the display 6464. For example, the user interface 6428 may include at least one touchpoint that allows the user to adjust the size of the selected sector. In some implementations, the sector selection feature 6436 may operate on the basis of touch input. For example, the sector editing function 6436 may allow editing of selected sectors based on a single touch input. In some implementations, the sector editing feature 6436 has at least one of adjusting the size of the sector, adjusting the shape of the sector, adjusting the boundaries of the sector, and / or zooming in on the sector. Can be possible. In some implementations, the sector editing feature 6436 may allow simultaneous editing of multiple sectors. In this example, the sector editing function 6436 may allow editing of multiple sectors based on multiple touch inputs.
[00390] As described above, in certain implementations, at least one of the sector selection function 6432 and the sector editing function 6436 operates on the basis of a single touch input or multiple touch inputs. obtain. For example, the sector selection function 6432 may be based on one or more swipe inputs. For example, one or more swipe inputs can indicate a circular area. In some configurations, one or more swipe inputs can be a single swipe. The sector selection function 6432 may be based on single-touch input or multi-touch input. In addition, or instead, the electronic device 6402 may adjust sectors based on single-touch or multi-touch inputs.
[00391] In these examples, the display 6464 may include a touch sensor 6438 capable of receiving a touch input (eg, tap, swipe, or circular motion) to select a sector. The touch sensor 6438 may also receive touch inputs for editing sectors, for example by moving the touch points displayed on the display 6464. In some configurations, the touch sensor 6438 may be incorporated into the display 6464. In other configurations, the touch sensor 6438 may be implemented separately within the electronic device 6402 or coupled to the electronic device 6402.
[00392] The user interface alignment block / module 6440 may align all or part of the user interface 6428 with the reference plane. In some implementations, the reference plane can be horizontal (eg, parallel to the ground or floor). For example, the user interface alignment block / module 6440 may align part of the user interface 6428 that displays the coordinate system 6430. In some implementations, the user interface alignment block / module 6440 can align all or part of the user interface 6428 in real time.
[00393] In some configurations, the electronic device 6402 may include at least one image sensor 6434. For example, several image sensors 6434 (in addition to, or instead of, multiple microphones 6406) may be included within the electronic device 6402. At least one image sensor 6434 may collect data about the electronic device 6402 (eg, image data). For example, a camera (eg, an image sensor) can produce an image. In some implementations, at least one image sensor 6434 may provide image data 5608 to the display 6464.
[00394] The electronic device 6402 may pass an audio signal (eg, a target audio signal) contained within at least one sector. For example, the electronic device 6402 can pass an audio signal to the working block / module 6414. The working block / module may pass one or more signals of the audio shown in at least one sector. In some implementations, the working block / module 6414 may include an attenuator 6442 that attenuates the audio signal. For example, the working block / module 6414 (eg, attenuator 6442) attenuates (eg, blocks, reduces, and / or rejects) audio signals (eg, interfering audio signals) that are not contained in at least one selected sector. )can do. In some cases, the audio signal may include an audio signal. For example, the sector selection function may allow the attenuation of unwanted audio signals in addition to the user audio signal.
[00395] In some configurations, the electronic device (eg, display 6464 and / or motion block / module 6414) may display image data from the image sensor 6434. In one configuration, the electronic device 6402 (eg, operating block / module 6414) passes image data from at least one image sensor 6434 (and filters, for example, other image data) based on at least one sector. can do. In other words, at least one of the techniques described herein with respect to the user interface 6428 may be applied to image data in place of or in addition to the audio signal.
[00396] FIG. 65 is a flow diagram illustrating a more specific configuration of method 6500 for displaying the user interface 6428 on the electronic device 6402. The method can be performed by electronic device 6402. The electronic device 6402 may acquire a coordinate system 6430 corresponding to physical coordinates (6502). In some implementations, this can be done as described with respect to FIG. 63.
[00397] Electronic device 6402 may present a user interface 6428 including coordinate system 6430 (6504). In some implementations, this can be done as described with respect to FIG. 63.
[00398] Electronic device 6402 may display the directionality of at least one audio signal captured by at least one microphone (6506). In other words, the electronic device 6402 may display the position of the audio signal source with respect to the electronic device. The electronic device 6402 may also display the angle of the audio signal source within the display 6464. As described above, the electronic device 6402 may display a 2D anglogram of the captured audio signal. In some embodiments, the display 6464 may display a 2D plot in 3D perspective to convey the alignment of the plot with a plane based on the physical coordinates of the real world, such as a horizontal plane.
[00399] The electronic device 6402 may display an icon corresponding to at least one audio signal (for example, corresponding to the ripples displayed on the user interface 6428) (6508). According to some configurations, the electronic device 6402 (eg, display 6464) may display an icon that identifies the audio signal as being the target audio signal (eg, audio signal) (6508). In addition, or instead, the electronic device 6402 (eg, display 6464) displays an icon (eg, a different icon) that identifies the audio signal as noise and / or interference (eg, interference or interference audio signal). Can (6508).
[00400] In some implementations, the electronic device 6402 may display an icon corresponding to the source of the audio signal (6508). For example, an electronic device 6402 may display a source of audio signals, eg, an image icon indicating an image of an individual (6508). The electronic device 6402 may display multiple icons corresponding to at least one audio signal (6508). For example, an electronic device may display at least one image icon and / or icon that identifies an audio signal as a noise / interference signal, or audio signal.
[00401] Electronic device 6402 may align all or part of user interface 6428 with a reference plane (6510). For example, electronic device 6402 may align coordinate system 6430 with a reference plane (6510). In some configurations, aligning all or part of the user interface 6428 (6510) involves mapping a 2D plot (eg, a polar plot) to a 3D display space (eg, projection). obtain. In addition, or instead, the electronic device 6402 may align one or more of the sector selection function 6432 and the sector editing function 6436 with the reference plane. The reference plane may be horizontal (eg, it may correspond to Earth coordinates). In some implementations, a portion of the user interface 6428 that is aligned with the reference plane can be aligned with a reference plane that is independent of the electronic device 6402 orientation. In other words, as the electronic device 6402 transforms and / or rotates, all or part of the datum-aligned user interface 6428 may still remain datum-aligned. In some implementations, the electronic device 6402 may align all or part of the user interface 6428 in real time (6510).
[00402] The electronic device 6402 may provide a sector selection function 6432 that allows selection of at least one sector in the coordinate system 6430 (6512). In some implementations, this can be done as described with respect to FIG. 63.
[00403] In some implementations, the electronic device 6402 (eg, user interface 6428 and / or sector selection function 6432) may pad the selected sector (6514). For example, the electronic device 6402 may include additional information with an audio signal to improve spatial audio processing. For example, padding can refer to providing visual feedback that is provided as highlighted padding (eg, in bright colors) for a selected sector. For example, the selected sector 7150 (eg, sector contour), illustrated in FIG. 71, may be highlighted to allow easy identification of the selected sector.
[00404] An electronic device 6402 (eg, display 6464, user interface 6428, etc.) may provide a sector editing function 6436 that allows editing of at least one sector (6516). As described above, the electronic device 6402 may provide sector editing functionality 6436 via the user interface 6428 (6516). In some implementations, the sector editing feature 6436 may operate on the basis of touch input. For example, the sector editing function 6436 may allow editing of selected sectors based on a single touch input or multiple touch inputs. For example, the user interface 6428 may include at least one touchpoint that allows the user to adjust the size of the selected sector. In this implementation, the electronic device 6402 may provide a touch sensor 6438 that receives a touch input that allows editing of at least one sector.
[00405] Electronic device 6402 may provide fixed mode and editable mode (6518). In editable mode, the user interface 6428 may respond to inputs to operate at least one function of the user interface 6428 (eg, sector selection function 6432). In fixed mode, user interface 6428 cannot respond to input to operate at least one function of user interface 6428. In some implementations, the electronic device 6402 may allow a choice between fixed mode and editable mode. For example, a wireless button on the user interface 6428 may allow a choice between editable mode and fixed mode.
[00406] Electronic device 6402 may pass the audio signal shown in at least one sector (6520). For example, electronic device 6402 may pass the audio signal shown within the selected sector (6520). In some implementations, the electronic device 6402 can attenuate the audio signal (6522). For example, electronic device 6402 may attenuate (eg, reduce and / or reject) audio signals that are not contained within at least one selected sector (6522). For example, an audio signal can include an audio signal. In this example, the electronic device 6402 can attenuate unwanted audio signals in addition to the user audio signal (6522).
[00407] FIG. 66 shows an example of a user interface 6628a-b for displaying the directionality of at least one audio signal. In some implementations, user interfaces 6628a-b may be examples of user interface 6228 described with respect to FIG. 62. User interfaces 6628a-b may include coordinate systems 6630a-b, which may be examples of coordinate systems 6230 described with respect to FIG. 62.
[00408] In Figure 66, the electronic device 6202 (eg, a telephone) can be laid flat. This can happen, for example, in tabletop mode. In FIG. 66, the coordinate system 6630a-b may include at least one audio signal indicator 6646a-b capable of indicating the orientation of at least one audio signal (eg, according to an angle or range of angles). .. At least one audio signal can come from a person, a speaker, or anything that can produce an audio signal. Within the first user interface 6628a, the first audio signal indicator 6646a can indicate that the first audio signal is at approximately 180 degrees. By comparison, within the second user interface 6628b, the second audio signal indicator 6646b can indicate that the second audio signal is at approximately 270 degrees. In some embodiments, the audio signal indicators 6646a-b can indicate the strength of the audio signal. For example, the audio signal indicators 6646a-b may include at least one color gradient that indicates the strength of the audio signal.
[00409] A first user interface, 6628a, provides an example of one or more characteristics that may be included within one or more of the user interfaces described herein. For example, the first user interface 6628a includes the title portion 6601. The title portion 6601 may include a user interface or the title of an application that provides the user interface. In the example shown in FIG. 66, the title is "SFAST". Other titles may be used. In general, the title portion 6601 is optional and some configurations of the user interface may not include the title portion. Also note that the title portion may be placed anywhere on the user interface (eg top, bottom, left, right, and / or overlay).
[00410] In the example shown in FIG. 66, the first user interface 6628a includes control portion 6603. Control part 6603 includes an example of interactive control. In some configurations, one or more of these interactive controls may be included within the user interface described herein. In general, the control part 6603 is optional and some configurations of the user interface may not include the control part 6603. Further, the control parts may or may not be grouped as illustrated in FIG. For example, one or more of the interactive controls may be located within different parts of the user interface (for example, top, bottom, center, left, right, and / or overlay).
In the example shown in FIG. 66, the first user interface 6628a includes an activate / deactivate button 6607, a checkbox 6609, a target sector indicator 6611, a wireless button 6613, a smoothing slider 6615, and so on. Includes a reset button 6617 and a noise suppression (NS) enabled button 6619. However, it should be noted that interactive control can be implemented in a wide variety of configurations. For example, one or more of sliders, wireless buttons, buttons, toggle buttons, checkboxes, lists, dials, tabs, text boxes, drop-down lists, links, images, grids, tables, labels, and / or them. The combination of can be implemented within the user interface to control various functions.
[00412] The activate / deactivate button 6607 can generally activate or deactivate features relating to the first user interface 6628a. For example, when an event corresponding to the activate / deactivate button 6607 (eg, a touch event) occurs, in the case of activation, the user interface 6628a enables user interface interactivity and displays an audio signal indicator 6646a. Upon gaining or deactivating, the user interface interactivity may be disabled and the display of the audio signal indicator 6646a may be paused or discontinued.
[00413] Check box 6609 may enable or disable the display of the target audio signal and / or the interfering audio signal. For example, displaying an interferometer and displaying a target checkbox allow visual feedback on the detected / calculated interferometer and the detected angle of the target audio signal, respectively. For example, the "display interference" element can be paired with the "display target" element, which makes it possible to visualize the target position and points about the interference position in the user interface 6628a. In some configurations, the "Show Interfering" and "Show Target" elements are of the target source or interference source (eg, its actual photo, icon, etc.) on the angular position detected by the device. It can enable / disable the display of any actual photo.
[00414] The target sector indicator 6611 may provide a display of selected sectors or target sectors. In this example, all sectors are shown as target sectors. Another example is provided below with respect to Figure 71.
[00415] The radio button 6613 may allow the selection of fixed sector mode or editable sector mode. In fixed mode, one or more sectors (eg, selected sectors) cannot be adjusted. In editable mode, one or more sectors (eg, selected sectors) can be adjusted.
[00416] The smoothing slider 6615 may provide a selection of values used to filter the input. For example, a value of 0 can indicate no filter, while a value of 25 can indicate aggressive filtering. In some configurations, the smoothing slider 6615 represents the amount of smoothing to display the source activity polar plot. For example, the amount of smoothing can be obtained based on the value indicated by the smoothing slider 6615, in which case recursive smoothing is performed (eg polar = (1-alpha)).<sup>*</sup>polar + (alpha)<sup>*</sup>polar_current_frame, so less alpha means more smoothing).
[00417] Reset button 6617 may allow the erasure of one or more current user interface 6628a settings. For example, when a touch event corresponding to reset button 6617 occurs, user interface 6628a may erase any sector selection, erase whether a target and / or interfering audio signal is displayed, and / or smooth. The conversion slider can be reset to the default value. The noise suppression (NS) enable button 6619 may enable or disable noise suppression processing for the input audio signal. For example, electronic devices may enable or disable filtering of interfering audio signals based on the Noise Suppression (NS) Enable button 6619.
[00418] User interface 6628a may include coordinate system portion 6605 (eg, plot portion). In some configurations, the coordinate system portion 6605 may occupy the entire user interface 6628a (and / or the entire device display). In other configurations, the coordinate system may occupy a subsection of user interface 6628a. Although polar coordinate systems are given herein as an example, it should be noted that alternative coordinate systems, such as Cartesian coordinate systems, may be included within user interface 6628a.
[00419] Figure 94 shows another example of the user interface 9428. In this example, user interface 9428 includes an orthogonal (eg Cartesian) coordinate system 9430. An example of an audio signal indicator 9446 is also shown. As described above, the coordinate system 9430 may occupy the entire user interface 9428 (and / or the entire display 9464 contained within the electronic device 6202), as illustrated in FIG. In other configurations, the coordinate system 9430 may occupy a subsection of the user interface 9428 (and / or display 9464). It should be noted that the Cartesian coordinate system can be implemented in place of any of the polar coordinate systems described herein.
[00420] FIG. 67 shows another example of a user interface 6728 for displaying the directionality of at least one audio signal. In some implementations, the user interface 6728 may be an example of the user interface 6228 described with respect to FIG. 62. The user interface may include a coordinate system 6730 and at least one audio signal indicator 6746a-b, which may be examples of the corresponding elements described with respect to one or more of FIGS. 62 and 66. In FIG. 67, the user interface 6728 may include a plurality of audio signal indicators 6746a-b. For example, the first audio source 6746a may indicate that the first audio source 6715a is at about 90 degrees and the second audio source 6715b is at about 270 degrees. For example, FIG. 67 shows an example of voice detection for the left and right of an electronic device that includes a user interface 6728. More specifically, the user interface 6728 may represent audio detected from the left and right of the electronic device. For example, the user interface 6728 can display multiple (eg, two) different sources in different locations at the same time. In some configurations, the procedure described with respect to FIG. 78 below may allow the selection of two sectors corresponding to the audio signal indicators 6746a-b (and, for example, the audio signal source 6715a-b).
[00421] FIG. 68 shows another example of a user interface 6828 for displaying the directionality of at least one audio signal. In some implementations, the user interface 6828 may be an example of the user interface 6228 described with respect to FIG. The user interface may include a coordinate system 6830 and an audio signal indicator 6846, which may be examples of the corresponding elements described with respect to one or more of FIGS. 62 and 66. FIG. 68 shows an example in which the 2D coordinate system 6830 is projected into the 3D display space, in which case the coordinate system 6830 appears to extend inside the user interface 6828. For example, an electronic device 6202 (eg, a telephone) can be in the palm of the user. Specifically, the electronic device 6202 can be horizontal with the table up. In this example, part of the user interface 6828 can be aligned with the horizontal datum, as described above. The audio signal of FIG. 68 can come from a user holding an electronic device 6202 in his hand and speaking in front of it (eg, at approximately 180 degrees).
[00422] FIG. 69 shows another example of a user interface 6928 for displaying the directionality of at least one audio signal. In some implementations, the user interface 6928 may be an example of the user interface 6228 described with respect to FIG. The user interface may include a coordinate system 6930 and an audio signal indicator 6946, which may be examples of the corresponding elements described with respect to one or more of FIGS. 62 and 66. In FIG. 69, the electronic device 6202 (eg, a telephone) may be in the palm of the user. For example, the electronic device 6202 can be horizontal with the table up. In this example, part of the user interface 6928 can be aligned with the horizontal datum, as described above. The audio signal of FIG. 69 can originate from the back of the electronic device 6202 (eg, at approximately 0 degrees).
[00423] FIG. 70 shows another example of the user interface 7028 for displaying the directionality of at least one audio signal. In some implementations, user interface 7028 may be an example of user interface 6228 described with respect to FIG. The user interface may include a coordinate system 7030 and at least one audio signal indicator 7046a-b, which may be examples of the corresponding elements described with respect to one or more of FIGS. 62 and 66. In some configurations, the user interface 7028 may include at least one icon 7048a-b corresponding to the type of audio signal indicator 7046a-b displayed. For example, user interface 7028 may display a triangle icon 7048a next to a first audio signal indicator 7046a that corresponds to a target audio signal (eg, speaker voice or user voice). Similarly, user interface 7028 may display a diamond icon 7048b next to a second audio signal indicator 7046b corresponding to interference (eg, interfering audio signal or noise).
[00424] FIG. 71 shows an example of the sector selection function 6232 of the user interface 7128. In some implementations, user interface 7128 may be an example of user interface 6228 described with respect to FIG. User interface 7128 may include coordinate system 7130 and / or audio signal indicator 7146, which may be examples of the corresponding elements described with respect to one or more of FIGS. 62 and 66. As described above, the user interface 7128 may include a sector selection function 6232 that allows selection of at least one sector, for example by touch input. In Figure 71, the selected sector 7150 is indicated by a dashed line. In some implementations, the angular range of the selected sector 7150 (eg, from about 225 degrees to about 315 degrees, as shown in Figure 71) may also be displayed. As described above, in some implementations, the electronic device 6202 may pass the audio signal indicated within the selected sector 7150 (eg, represented by the audio signal indicator 7146). In this example, the audio signal source is on the side of the phone (approximately 270 degrees). In some configurations, other sectors outside the selected sector 7150 may be noise-suppressed and / or attenuated.
[00425] In the example shown in FIG. 71, user interface 7128 includes a target sector indicator. The target sector indicator, in this case, indicates the selected sector between 225 and 315 degrees. Note that sectors can be displayed using other parameters in other configurations. For example, the target sector indicator can radiate selected sectors according to sector number or the like.
[00426] FIG. 72 shows another example of the sector selection function 6232 of the user interface 7228. In some implementations, user interface 7228 may be an example of user interface 6228 described with respect to FIG. User interface 7228 may be an example of a corresponding element described with respect to at least one of FIGS. 62, 66, and 71, a coordinate system 7230, an audio signal indicator 7246, and at least one selected. It may include sectors 7250a ~ b. As described above, the sector selection function 6232 may allow simultaneous selection of multiple sectors. In Figure 72, two sectors 7250a-b are selected (for example, as indicated by the dashed line). In this example, the audio signal is at approximately 270 degrees. Other sectors outside the selected sectors 7250a-b may be noise-suppressed and / or attenuated. Thereby, the systems and methods described herein may allow one-time selection of two or more sectors 7250.
[00427] FIG. 73 shows another example of the sector selection function 6232 of the user interface 7328. In some implementations, user interface 7328 may be an example of user interface 6228 described with respect to FIG. User interface 7328 includes a coordinate system 7330 and at least one audio signal indicator 7346a-b, which may be an example of the corresponding elements described for at least one of FIGS. 62, 66, and 71. It may include at least one selected sector 7350a ~ b. In FIG. 73, two sectors 7350a-b are selected (for example, as indicated by the dashed line). In this example, the speaker is on the side of the electronic device 6202. Other sectors outside the selected sectors 7250a-b may be noise-suppressed and / or attenuated.
[00428] FIG. 74 shows more examples of the sector selection function 6232 of the user interfaces 7428a-f. In some implementations, user interfaces 7428a-f may be examples of user interface 6228 described with respect to FIG. The user interfaces 7428a-f may be examples of the corresponding elements described for at least one of FIGS. 62, 66, and 71, the coordinate systems 7430a-f and at least one audio signal indicator 7446a. It may contain ~ f and at least one selected sector 7450a ~ c. In this example, the selected sectors 7450a-c can be determined based on the touch input 7452. For example, sector and / or sector angles can be selected based on a finger swipe. For example, the user can enter a circular touch input 7452. The selected sector 7150b can then be determined based on the circular touch input 7452. In other words, the user can narrow the sector by drawing the area instead of making manual adjustments (eg, based on touchpoints or "handles"). In some implementations, if multiple sectors are selected based on touch input 7452, the "best" sector 7450c may be selected and readjusted to match that region. In some implementations, the term "best" may refer to a sector with at least one strongest audio signal. This can be one user-friendly way to select and narrow sectors. Note that multiple fingers (eg, two or more) can be used simultaneously on or at the top of the screen to zoom in or out on a sector. Other examples of touch input 7452 may include tap input from the user. In this example, the user can tap part of the coordinate system and select a sector in the center of the tap position (or matching a preset range of angles). In this example, it will be explained below.
[00429] FIG. 75 shows more examples of the sector selection function 6232 of the user interfaces 7528a-f. In some implementations, user interfaces 7528a-f may be examples of user interface 6228 described with respect to FIG. The user interfaces 7528a-f may be examples of the corresponding elements described for at least one of FIGS. 62, 66, and 71, the coordinate system 7530a-f and at least one audio signal indicator 7546a. It may contain ~ f and at least one selected sector 7550a ~ c. In this example, the selected sectors 7550a-c can be determined based on the touch input 7552. For example, sector and / or sector angles can be selected based on a finger swipe. For example, the user may enter swipe touch input 7552. In other words, the user can narrow the sector by drawing the area instead of adjusting it manually (eg, based on a touchpoint or "handle"). In this example, sectors can be selected and / or adjusted based solely on swipe touch input 7552 (instead of drawing a circle, for example). The selected sector 7150b can then be determined based on the swipe touch input 7552. In some implementations, if multiple sectors are selected based on touch input 7552, the "best" sector 7550c may be selected and readjusted to match that region. In some implementations, the term "best" can refer to a sector that has at least one of the strongest audio signals. This can be one user-friendly way to select and narrow sectors. Note that multiple fingers (eg, two or more) can be used simultaneously on or at the top of the screen to zoom in or out on a sector. A single finger or multiple fingers can be perceived according to any of the sector selection and / or adjustment techniques described herein.
[00430] FIG. 76 is a flow diagram showing one configuration of the method 7600 for editing sectors. Method 7600 can be performed by electronic device 6202. An electronic device 6202 (eg, display 6264) may display at least one point (eg, touchpoint) that corresponds to at least one sector (7602). In some implementations, at least one touchpoint may be performed by the sector editing function 6436 to allow editing of at least one sector. For example, user interface 6228 may include at least one touchpoint that allows the user to adjust (eg, zoom in or out) the size of selected sectors. Touchpoints may appear around sector boundaries.
[00431] An electronic device 6202 (eg, a touch sensor) may receive a touch input corresponding to at least one point (eg, a touch point) (7604). For example, the electronic device 6202 may receive a touch input that edits a sector (eg, adjusts its size and / or shape). For example, the user may select at least one touchpoint by touching the touchpoint. In this example, the user may move the touchpoint displayed on the user interface 6228. In this implementation, receiving a touch input (7604) may include adjusting the touch point based on the touch input. For example, as the user moves the touch point through the touch sensor 6438, the electronic device 6202 may move the touch point accordingly.
[00432] Electronic device 6202 (eg, user interface 6228) may edit at least one sector based on touch input (7606). For example, electronic device 6202 may adjust sector size and / or shape based on single or multiple touch inputs. Similarly, electronic device 6202 may reposition sectors with respect to coordinate system 6230 based on touch input.
[00433] FIG. 77 shows an example of the sector editing function 6436 of the user interfaces 7728a to 7728. In some implementations, user interfaces 7728a-b may be examples of user interface 6228 described with respect to FIG. User interfaces 7728a-b may include coordinate systems 7730a-b, which may be examples of the corresponding elements described with respect to FIG. 62. User interfaces 7728a-b may include at least one touchpoint 7754a-h. As described above, touchpoints 7754a-h can be handles that allow editing of at least one sector. Touch points 7754a-h can be located at the vertices of the sector. In some implementations, sector editing can be done independently of sector selection. Therefore, in some configurations, unselected sectors may be adjusted.
[00434] In some implementations, user interfaces 7728a-b may provide interactive control that allows fixed and edit modes for user interfaces 7728a-b. For example, the user interfaces 7728a-b may each include an activate / deactivate button 7756a-b that controls whether the user interfaces 7728a-b are operational. The activate / deactivate buttons 7756a-b can toggle the activated / deactivated state for the user interface 7728a-b. When in editable mode, user interfaces 7728a-b may display at least one touchpoint 7754a-f (eg, a handle) (eg, a circle at the edge of a sector) that corresponds to at least one sector.
[00435] FIG. 78 shows more examples of the sector editing function 6436 of the user interfaces 7828a-c. In some implementations, user interfaces 7828a-c may be examples of user interface 6228 described with respect to FIG. The user interfaces 7828a-c may be examples of the corresponding elements described for at least one of FIGS. 62, 66, and 71, the coordinate systems 7830a-c and at least one audio signal indicator 7846a. It may contain ~ b, at least one selected sector 7850a ~ e, and at least one touchpoint 7854a ~ l. In Figure 78, at least one sector is selected (for example, as indicated by the dashed line). As shown in FIG. 78, the selected sectors 7850a-e can be narrowed for greater accuracy. For example, the user can use touchpoints 7854a-l to adjust (eg, zoom in or out) selected sectors 7850a-e. Other sectors outside the selected sector 7850a-b may be noise-suppressed and / or attenuated.
[00436] FIG. 79 shows more examples of the sector editing function 6436 of the user interfaces 7928a-b. In some implementations, user interfaces 7928a-b may be examples of user interface 6228 described with respect to FIG. User interfaces 7928a-b may be examples of the corresponding elements described for at least one of FIGS. 62, 66, and 71, the coordinate system 7930a-b and at least one audio signal indicator 7946a. It may include ~ b, at least one selected sector 7950a ~ b, and at least one touchpoint 7954a ~ h. In FIG. 79, the electronic device 6202 (eg, telephone) may be in the palm of the user. For example, the electronic device 6202 can be tilted upwards. In this example, part of the user interface 7928a-b (eg, coordinate system 7930a-b) can be aligned with the horizontal reference plane as described above. Therefore, the coordinate systems 7930a-b appear in three-dimensional perspective extending within the user interface 7928a-b. The audio signal of FIG. 79 can come from a user holding the electronic device 6202 in his hand and speaking in front of it (eg, at approximately 180 degrees). FIG. 79 also illustrates that at least one sector can be narrowed or widened in real time. For example, selected sectors 7950a-b may be adjusted during an ongoing conversation or telephone call.
[00437] FIG. 80 shows more examples of the sector editing function 6436 of the user interfaces 8028a-c. In some implementations, user interfaces 8028a-c may be examples of user interface 6228 described with respect to FIG. User interfaces 8028a-c may be examples of the corresponding elements described for at least one of FIGS. 62, 66, and 71, the coordinate system 8030a-c and at least one audio signal indicator 8046a. It may include ~ c, at least one selected sector 8050a ~ b, and at least one touchpoint 8054a ~ b. The first figure shows an audio signal indicator 8046a showing the presentation of an audio signal at approximately 270 degrees. The middle figure shows the user interface 8028b with the selected sector 8050a. The figure on the right shows an example of editing the selected sector 8050b. In this case, the selected sector 8050b is narrowed. In this example, the electronic device 6202 can pass an audio signal with an arrival direction associated with the selected sector 8050b and attenuate other audio signals with an arrival direction associated with the outside of the selected sector 8050b. it can.
[00438] Figure 81 shows more examples of the sector editing function 6436 of the user interfaces 8128a-d. In some implementations, user interfaces 8128a-d may be examples of user interface 6228 described with respect to FIG. The user interfaces 8128a-d may be examples of the corresponding elements described for at least one of FIGS. 62, 66, and 71, the coordinate systems 8130a-d and at least one audio signal indicator 8146a. It may include ~ d, at least one selected sector 8150a ~ c, and at least one touchpoint 8154a ~ h. The first figure shows an audio signal indicator 8146a showing the presentation of an audio signal at approximately 270 degrees. The second figure shows a user interface 8128b with selected sectors 8150a. The third figure shows at least one touchpoint 8154a ~ d used to edit a sector. The fourth figure shows an example of editing the selected sector 8150d. In this case, the selected sector 8150d is narrowed. In this example, the electronic device 6202 has an arrival direction associated with the outside of the selected sector 8150d, passing an audio signal having an arrival direction associated with the selected sector 8150d (eg, obtained based on user input). Other audio signals can be attenuated.
[00439] FIG. 82 shows an example of a user interface 8228 having a coordinate system 8230 oriented independently of the orientation of electronic device 6202. In some implementations, user interface 8228 may be an example of user interface 6228 described with respect to FIG. The user interface may include a coordinate system 8230 and an audio signal indicator 8246, which may be examples of the corresponding elements described with respect to at least one of FIGS. 62 and 66. In FIG. 82, the electronic device 6202 (eg, telephone) is tilted upward (eg, in the palm of the user). The coordinate system 8230 (eg, polar graph) of the user interface 8228 indicates or displays the audio signal source position. In this example, part of the user interface 8228 is aligned with the horizontal datum, as described above. The audio signal in Figure 82 originates from source 8215 at approximately 180 degrees. As explained above, the source 8215 is capable of generating a user, speaker, or audio signal (ie, for example, holding an electronic device 6202 in its hand and speaking in front of it). Can include something.
[00440] FIG. 83 shows another example of a user interface 8328 having a coordinate system 8330 oriented independently of the electronic device 6202 direction. In some implementations, the user interface 8328 may be an example of the user interface 6228 described with respect to FIG. The user interface 8328 may include a coordinate system 8330 and an audio signal indicator 8346, which may be examples of the corresponding elements described for at least one of FIGS. 62 and 66. In Figure 83, the electronic device 6202 (for example, a telephone) gradually rises from the bottom of the electronic device 6202 (towards sound source 8315) to the top of the electronic device 6202 (for example, in the palm of the user) at an angle or tilt. It is in the direction of The coordinate system 8330 (eg, polar graph) of the user interface 8328 displays the audio signal source position. In this example, part of the user interface 8328 is aligned with the horizontal datum, as described above. The audio signal of FIG. 83 originates from the source 8315 towards the back of the electronic device 6202 (eg, telephone). FIG. 83 illustrates that the reference plane of user interface 8328 is aligned with a physical plane (eg, horizontal) in the 3D world. Note in FIG. 83 that the user interface 8328 plane is in the screen state, even though the electronic device 6202 is held semi-vertically. Thus, the coordinate system 8330 plane of the user interface 8328 is at 0 degrees to the physical plane of the floor, even though the electronic device 6202 is at approximately 45 degrees to the physical plane of the floor. For example, the reference plane on the user interface 8328 corresponds to the reference plane in the physical coordinate system.
[00441] FIG. 84 shows another example of a user interface 8428 with a coordinate system 8430 oriented independently of the electronic device 6202 orientation. In some implementations, user interface 8428 may be an example of user interface 6228 described with respect to FIG. 62. The user interface 8428 may include a coordinate system 8430 and an audio signal indicator 8446, which may be examples of the corresponding elements described for at least one of FIGS. 62 and 66. In FIG. 84, the electronic device 6202 (eg, telephone) is vertical (eg, in the palm of the user). The coordinate system 8430 (eg, polar graph) of the user interface 8428 displays the audio signal source position. In this example, part of the user interface 8428 is aligned with the horizontal datum, as described above. The audio signal of FIG. 84 originates from a source 8415 directed to the back left (eg, back) of an electronic device 6202 (eg, telephone).
[00442] FIG. 85 shows another example of a user interface 8528 with a coordinate system 8530 oriented independently of the electronic device 6202 orientation. In some implementations, the user interface 8528 may be an example of the user interface 6228 described with respect to FIG. The user interface 8528 may include a coordinate system 8530 and an audio signal indicator 8546, which may be examples of the corresponding elements described for at least one of FIGS. 62 and 66. In FIG. 85, the electronic device 6202 (eg, telephone) is in a table-up horizontal orientation (eg, tabletop mode). The coordinate system 8530 (eg, polar graph) of the user interface 8528 displays the audio signal source position. The audio signal of FIG. 85 originates from a source 8515 heading to the upper left of the electronic device 6202 (eg, a telephone). In some examples, the audio signal source is tracked. For example, when noise suppression is enabled, the electronic device 6202 can track the loudest speaker or sound source. For example, an electronic device 6202 (eg, a telephone) can suppress other sounds (eg, noise) from other areas (eg, zones or sectors) while at the same time tracking the loudest speaker movements. ..
[00443] FIG. 86 shows more examples of user interfaces 8628a-c with coordinate systems 8630a-c oriented independently of the electronic device 6202 orientation. In other words, the coordinate systems 8630a-c and / or the audio signal indicators 8646a-c stay in the same direction with respect to physical space, regardless of how the electronic device 6202 rotates. In some implementations, user interfaces 8628a-c may be examples of user interface 6228 described with respect to FIG. 62. The user interface 8628a-c may include a coordinate system 8630a-c and an audio signal indicator 8646a-c, which may be examples of the corresponding elements described with respect to at least one of FIGS. 62 and 66. Without a compass, the sector selection function 6232 may not be relevant to the physical coordinate system of the real world (eg, north, south, east, west, etc.). Therefore, if the electronic device 6202 (for example, a telephone) is in the vertical direction facing the user (for example, browse talk mode), the top of the electronic device 6202 can be specified as "0 degrees" and along the vertical axis. move on. When the electronic device 6202 is rotated, for example, 90 degrees clockwise, "0 degrees" is located on the horizontal axis. Therefore, when a sector is selected, the rotation of the electronic device 6202 affects the selected sector. By adding another component that can detect direction, for example a compass, the sector selection function 6232 of user interfaces 8628a-c can be relative to physical space rather than telephone. In other words, by adding a compass, when the phone is selected from a vertical upright position to a horizontal position, the top side of the phone facing the user remains at "0 degrees". For example, in the first image of FIG. 86, the user interface 8628a is illustrated without tilt (ie, eg, at 0 degree tilt). For example, the coordinate system 8630a is a user interface. Consistent with face 8628a and / or electronic device 6202. By comparison, in the second image of FIG. 86, the user interface 8628b and / or the electronic device 6202 is tilted to the left. However, the coordinate system 8630b (and the mapping between the real world and the electronic device 6202) can be maintained. This can be done, for example, based on tilt sensor data 5608. In the third image of FIG. 86, the user interface 8628c and / or the electronic device 6202 is tilted to the right. However, the coordinate system 8630c (and the mapping between the real world and the electronic device 6202) can be maintained.
[00444] It should be noted that as used herein, the term "physical coordinates" may or may not refer to geographic coordinates. In some configurations, for example, when the electronic device 6202 does not include a compass, the electronic device 6202 may still map coordinates from the multi-microphone configuration to physical coordinates based on sensor data 5608. In this case, the mapping 5612 may be relative to the electronic device 6202 and may not directly correspond to Earth coordinates (eg, north, south, east, west). Nevertheless, the electronic device 6202 may be able to distinguish the direction of sound in physical space with respect to the electronic device 6202. However, in some configurations, the electronic device 6202 may include a compass (or other means of navigation). In this case, the electronic device 6202 can map the coordinates from the multi-microphone configuration to the physical coordinates corresponding to the earth coordinates (eg, north, south, east, west). Different types of coordinate systems 6230 may be utilized according to the systems and methods disclosed herein.
[00445] FIG. 87 shows another example of a user interface 8728 with a coordinate system 8730 oriented independently of the electronic device 6202 direction. In some implementations, the user interface 8728 may be an example of the user interface 6228 described with respect to FIG. The user interface 8728 may include a coordinate system 8730 and an audio signal indicator 8746, which may be examples of the corresponding elements described for at least one of FIGS. 62 and 66. In some implementations, the user interface 8728 also includes a compass 8756 along with a coordinate system 8730 (as described above). In this implementation, the compass 8756 can detect direction. The compass 8756 portion can display the electronic device 6202 direction with respect to real world world coordinates. Through the compass 8756, the sector selection function 6232 on the user interface 8728 can be relative to the physical space rather than the electronic device 6202. In other words, by adding the compass 8756, when the electronic device 6202 is selected from the vertical position to the horizontal position, the vicinity of the top surface of the electronic device 6202 facing the user remains "0 degrees". Note that determining the physical electronic device 6202 direction can be done using the compass 8756. However, if the compass 8756 is not present, it can instead be determined based on GPS and / or gyro sensors. Thus, instead of or in addition to the compass 8756, any sensor 5604 or system that can be used to determine the physical orientation of the electronic device 6202 may be used. Thus, the compass 8756 can be replaced with another sensor 5604 or system in any of the configurations described herein. Therefore, it is possible to provide the user with a screenshot in which the orientation remains fixed.
[00446] If a GPS receiver is included within the electronic device 6202, GPS data can be used to provide additional functionality (in addition to being merely a sensor). In some configurations, for example, an electronic device 6202 (eg, a mobile device) may include GPS functionality with map software. On the other hand, in the equation, the coordinate system 8730 can be aligned so that the zero degree always points to the street, for example. With the compass 8756, for example, an electronic device 6202 (eg, coordinate system 8730) can be oriented according to physical north and / or south, but because GPS functionality offers more options. Can be used for.
[00447] FIG. 88 is a block diagram showing another configuration of the user interface 8828 in which a system and method for displaying the user interface 8828 on the electronic device 8802 may be implemented. The user interface 8828 may be an example of the user interface 6228 described with respect to FIG. 62. In some implementations, the user interface 8828 may be presented on display 8864 of electronic device 8802, which may be an example of the corresponding elements described with respect to FIG. 62. The user interface 8828 may include a coordinate system 8830 and / or sector signal indicator 8832, which may be examples of the corresponding elements described for at least one of FIGS. 62 and 66. The user interface 8828 may be coupled to at least one microphone 8806 and / or motion block / module 8814, which may be an example of the corresponding elements described with respect to at least one of FIGS. 56 and 66.
[00448] In some implementations, the user interface 8828 can be included within the electronic device 8802 and / or can be coupled to the database 8858 which can be coupled to the electronic device 8802. For example, database 8858 may be stored in memory located on electronic device 8802. Database 8858 may contain one or more audio signatures. For example, database 8858 may contain one or more audio signatures for one or more audio signal sources (eg, individual users). Database 8858 may also contain information based on audio signatures. For example, database 8858 may contain identifying information about the user corresponding to the audio signature. The identifying information may include an image of the audio signal source (eg, an image of the person corresponding to the audio signature) and / or contact information such as name, email address, telephone number.
[00449] In some implementations, the user interface 8828 may include an audio signature recognition block / module 8860. The audio signature recognition block / module 8860 may recognize the audio signature received by at least one microphone 8806. For example, the microphone 8806 may receive an audio signal. The audio signature recognition block / module 8860 can take an audio signal and compare it with the audio signature contained in database 8858. In this example, the audio signature recognition block / module 8860 may retrieve the audio signature and / or identification information about the audio signature from database 8858 and pass the identification information to the display 8864.
[00450] FIG. 89 is a flow diagram illustrating another configuration of method 8900 for displaying the user interface 8828 on the electronic device 8802. Method 8900 can be performed by electronic device 8802. The electronic device 8802 may acquire a coordinate system 8830 corresponding to physical coordinates (8902). In some implementations, this can be done as described with respect to FIG. 63.
[00451] Electronic device 8802 may present a user interface 8828 that may include a coordinate system 8830 (8904). In some implementations, this can be done as described with respect to FIG. 63.
[00452] The electronic device 8802 may recognize the audio signature (8906). Audio signatures can be characterizations that correspond to a particular audio signal source. For example, an individual user may have an audio signature that corresponds to an individual voice. Examples of audio signatures include speech recognition parameters, audio signal components, audio signal samples, and / or other information for functionalizing audio signals. In some implementations, the electronic device 8802 may receive an audio signal from at least one microphone 8806. The electronic device 8802 can then recognize the audio signature by, for example, determining whether the audio signal is from an audio source, such as an individual user, compared to a noise signal (8906). This can be done by measuring at least one characteristic of the audio signal (eg, harmony, pitch, etc.). In some implementations, recognizing an audio signature (8906) may include identifying an audio signal coming from a particular audio source.
[00453] Electronic device 8802 may look up the audio signature in database 8858 (8908). For example, electronic device 8802 can look up audio signatures in database 8858 in audio signatures. The electronic device 8802 can obtain the identification information corresponding to the audio signature (8910). As explained above, database 8858 may also contain information based on audio signatures. For example, database 8858 may contain identifying information about the user corresponding to the audio signature. The identifying information may include an image of an audio signal source (eg, a user) and / or contact information such as a name, email address, telephone number, and the like. After acquiring the identification information (eg, an image) corresponding to the audio signature (8910), the electronic device 8802 may display the identification information on the user interface 8828 (8912). For example, the electronic device 8802 may display the user's image next to the audio signal indicator 6646 on the display 6264 (8912). In other implementations, the electronic device 8802 can display at least one identification information as part of the identification display (8912). For example, part of the user interface 8828 may include identifying information about the audio signature (eg, image, name, email address, etc.).
[00454] The electronic device 8802 can provide a sector selection function 6232 that allows selection of at least one sector in the coordinate system 8830 (8914). In some implementations, this can be done as described with respect to FIG. 63.
[00455] Figure 90 shows an example of a user interface 9028 coupled to database 9058. In some implementations, user interface 9028 may be an example of user interface 6228 described with respect to FIG. The user interface 9028 may include a coordinate system 9030 and an audio signal indicator 9046, which may be examples of the corresponding elements described for at least one of FIGS. 62 and 66. In some implementations, as described above, the user interface 9028 can be an example of the corresponding element described for at least one of FIGS. 88 and 89, at least one audio signature 9064 and / Or can be combined with database 9058 containing identification information 9062a corresponding to audio signature 9064. In some configurations, the electronic device 6202 may recognize the audio signature 9064 and look up the audio signature 9064 in the database 9058. The electronic device 6202 can then obtain (eg, retrieve) the corresponding identification information 9062a corresponding to the audio signature 9064 recognized by the electronic device 6202. For example, the electronic device 6202 may take a picture of the speaker or person and display the picture of the speaker or person (and other identification information 9062b) by the audio signal indicator 9046. In this way, the user can easily identify the source of the audio signal. Note that the database 9058 can be local or remote (on a server over a network, for example LAN or the Internet). In addition, or instead, the electronic device 6202 may transmit identification information 9062 to another device. For example, the electronic device 6202 may be on another device (for example, a smartphone, server, network, computer, etc.) that presents identification information 9062 so that a far-off user evaluates the current speaker. You can send one or more usernames (and / or images, identifiers, etc.). This can be useful, for example, when there are multiple users talking on the speakerphone.
[00456] Optionally, in some implementations, the user interface 9028 may display the identification information 9062 separately from the coordinate system 9030. For example, user interface 9028 may display identification information 9062c under coordinate system 9030.
[00457] FIG. 91 is a flow diagram illustrating another configuration of method 9100 for displaying user interface 6428 on electronic device 6402. Method 9100 can be performed by electronic device 6402. The electronic device 6402 can acquire the coordinate system 6430 corresponding to the physical coordinates (9102). In some implementations, this can be done as described with respect to FIG. 63.
[00458] Electronic device 6402 may present a user interface 6428 that may include coordinate system 6430 (9104). In some implementations, this can be done as described with respect to FIG. 63.
[00459] The electronic device 6402 may provide a sector selection function 6432 that allows selection of at least one sector in the coordinate system 6430 (9106). In some implementations, this can be done as described with respect to FIG. 63.
[00460] Electronic device 6402 may indicate image data from at least one sector (9108). As described above, the electronic device 6402 may include at least one image sensor 6434. For example, several image sensors 6434 that collect data about electronic device 6402 may be included on electronic device 6402. More specifically, at least one image sensor 6434 may collect image data. For example, a camera (eg, an image sensor 6434) can generate an image. In some implementations, at least one image sensor 6434 may provide image data to the user interface 6428. In some implementations, the electronic device 6402 may represent image data from at least one image sensor 6434 (9108). In other words, the electronic device 6402 may display image data (eg, still image or video) from at least one image sensor 6434 on the display 6464.
[00461] In some implementations, the electronic device 6402 may pass image data based on at least one sector (9110). For example, electronic device 6402 may pass the image data shown in the selected sector (9110). In other words, at least one of the techniques described herein with respect to the user interface 6428 may be applied to image data in place of or in addition to the audio signal.
[00462] FIG. 92 is a block diagram showing a configuration of a wireless communication device 9266 in which a system and method for mapping source locations may be implemented. The wireless communication device 9266 illustrated in FIG. 92 can be an example of at least one of the electronic devices described herein. The wireless communication device 9266 may include an application processor 9278. The application processor 9278 generally processes instructions (for example, to execute a program) to perform a function on the wireless communication device 9266. The application processor 9278 can be coupled to an audio coder / decoder (codec) 9276.
[00463] The audio codec 9276 can be an electronic device (eg, an integrated circuit) used to code and / or decode an audio signal. The audio codec 9276 may be coupled to at least one speaker 9268, earpiece 9270, output jack 9272, and / or at least one microphone 9206. Speaker 9268 may include one or more electroacoustic transducers that convert electrical or electronic signals into acoustic signals. For example, speaker 9268 can be used to play music, output speakerphone conversations, and so on. The earpiece 9270 can be another speaker or electroacoustic transducer that can be used to output an acoustic signal (eg, an audio signal) to the user. For example, the earpiece 9270 can be used to ensure that only the user can hear the acoustic signal. The output jack 9272 can be used to connect other devices, such as headphones, to the wireless communication device 9266 for outputting audio. Speakers 9268, earpieces 9270 and / or output jack 9272 can generally be used to output audio signals from the audio codec 9276. The at least one microphone 9206 can be an acoustic electric transducer that converts an acoustic signal (such as a user's voice) into an electric or electronic signal provided in the audio codec 9276.
[00464] The coordinate mapping block / module 9217a can optionally be implemented as part of the audio codec 9276. For example, the coordinate mapping block / module 9217a may be implemented according to one or more of the functions and / or structures described herein. For example, the coordinate mapping block / module 9217a may be implemented according to one or more of the functions and / or structures described with respect to FIGS. 57, 59, 60, and 61.
[00465] In addition, or instead, the coordinate mapping block / module 9217b may be implemented within the application processor 9278. For example, the coordinate mapping block / module 9217b may be implemented according to one or more of the functions and / or structures described herein. For example, the coordinate mapping block / module 9217b may be implemented according to one or more of the functions and / or structures described with respect to FIGS. 57, 59, 60, and 61.
[00466] The application processor 9278 may also be coupled to the power management circuit 9280. An example of a power management circuit 9280 is a power management integrated circuit (PMIC) that can be used to manage the power consumption of a wireless communication device 9266. The power management circuit 9280 can be coupled to the battery 9282. The battery 9292 can generally power the wireless communication device 9266. For example, the battery 9292 and / or the power management circuit 9280 may be coupled to one or more of the elements contained within the wireless communication device 9266.
[00467] The application processor 9278 may be coupled to one or more input devices 9286 for receiving inputs. Examples of input devices 9286 include infrared sensors, image sensors, accelerometers, touch sensors, keypads and the like. The input device 9286 may allow user interaction with the wireless communication device 9266. The application processor 9278 may also be coupled to one or more output devices 9284. Examples of output devices 9284 include printers, projectors, screens, and tactile devices. The output device 9284 may allow the wireless communication device 9266 to produce an output that can be received by the user.
[00468] Application processor 9278 may be coupled to application memory 9288. The application memory 9288 can be any electronic device capable of storing electronic information. Examples of application memory 9288 include double data rate synchronous dynamic random access memory (DDRAM), synchronous dynamic random access memory (SDRAM), and flash memory. Application memory 9288 can provide storage for application processor 9278. For example, application memory 9288 may store data and / or instructions for the functionality of a program running on application processor 9278.
[00469] The application processor 9278 can be coupled to the display controller 9290, and the display controller 9290 can be coupled to the display 9292. The display controller 9290 can be a hardware block used to generate an image on the display 9292. For example, the display controller 9290 may convert instructions and / or data from the application processor 9278 into an image that can be presented on the display 9292. Examples of displays 9292 include liquid crystal display (LCD) panels, light emitting diode (LED) panels, cathode ray tube (CRT) displays, plasma displays and the like.
[00470] Application processor 9278 may be coupled to baseband processor 9294. The baseband processor 9294 generally processes communication signals. For example, the baseband processor 9294 may demodulate and / or decode the received signal. In addition, or alternative, the baseband processor 9294 may encode and / or modulate the signal in preparation for transmission.
[00471] Baseband processor 9294 may be coupled to baseband memory 9296. The baseband memory 9296 can be any electronic device capable of storing electronic information, such as SDRAM, DDRAM, flash memory, and the like. The baseband processor 9294 can read information (eg, instructions and / or data) from baseband memory 9296 and / or write information to baseband memory 9296. In addition, or instead, the baseband processor 9294 may use the instructions and / or data stored in the baseband memory 9296 to perform communication operations.
[00472] Baseband processor 9294 may be coupled to radio frequency (RF) transmitter 9298. The RF transmitter / receiver 9298 may be coupled to a power amplifier 9201 and one or more antennas 9203. The RF transmitter / receiver 9298 can transmit and / or receive radio frequency signals. For example, the RF transmitter / receiver 9298 can transmit RF signals using a power amplifier 9201 and at least one antenna 9203. The RF transmitter / receiver 9298 may also use one or more antennas 9203 to receive RF signals.
[00473] Figure 93 illustrates the various components that can be utilized within the electronic device 9302. The illustrated components may be located within the same physical structure or within separate enclosures or structures. The electronic device 9302 described with reference to FIG. 93 may be implemented according to at least one of the electronic devices and wireless communication devices described herein. Electronic device 9302 includes processor 9311. Processor 9311 can be a general purpose single or multichip microprocessor (eg ARM), a dedicated microprocessor (eg Digital Signal Processor (DSP)), a microcontroller, a programmable gate array, and the like. Processor 9311 may be referred to as a central processing unit (CPU). Only a single processor 9311 is shown in the electronic device 9302 of FIG. 93, but alternative configurations may use a combination of processors (eg ARM® and DSP).
[00474] Electronic device 9302 also includes memory 9305 that is in electronic communication with processor 9311. That is, processor 9311 may read information from memory 9305 and / or write information to memory 9305. The memory 9305 can be any electronic component capable of storing electronic information. Memory 9305 includes random access memory (RAM), read-only memory (ROM), magnetic disk storage medium, optical storage medium, flash memory device in RAM, onboard memory included with the processor, programmable read-only memory (PROM), It can be erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM®), registers, and combinations thereof.
[00475] Data 9309a and instruction 9307a may be stored in memory 9305. Instruction 9307a may include at least one program, routine, subroutine, function, procedure, and so on. Instruction 9307a may include a single computer-readable statement or many computer-readable statements. Instruction 9307a may be executable by processor 9311 to implement at least one of the methods described above. Executing instruction 9307a may include the use of data 9309a stored in memory 9305. FIG. 93 shows some instructions 9307b and data 9309b loaded in processor 9311 (which may come from instruction 9307a and data 9309a).
[00476] Electronic device 9302 may also include at least one communication interface 9313 for communicating with other electronic devices. The communication interface 9313 may be based on wired communication technology, wireless communication technology, or both. Examples of various types of communication interfaces 9313 include serial ports, parallel ports, universal serial bus (USB), Ethernet (registered trademark) adapters, IEEE1394 bus interface, small computer system interface (SCSI) bus interface, infrared (IR). There are communication ports, Bluetooth wireless communication adapters, etc.
[00477] Electronic device 9302 may also include at least one input device 9386 and at least one output device 9384. Examples of different types of input devices 9386 include keyboards, mice, microphones, remote control devices, buttons, joysticks, trackballs, touchpads, light pens, and more. For example, electronic device 9302 may include at least one microphone 9306 for capturing acoustic signals. In one configuration, the microphone 9306 can be a transducer that converts an acoustic signal (eg, voice, voice) into an electrical or electronic signal. Examples of various types of output devices 9384 include speakers, printers, and the like. For example, electronic device 9302 may include at least one speaker 9368. In one configuration, the speaker 9368 can be a transducer that converts an electrical or electronic signal into an acoustic signal. One particular type of output device that can typically be included within the electronic device 9302 is the display device 9392. The display device 9392 used with the configurations disclosed herein is any suitable display device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a light emitting diode (LED), a gas plasma, electroluminescence, and the like. Image projection technology can be used. The display controller 9390 may also be provided to (as appropriate) convert the data stored in memory 9305 into the text, graphics, and / or video shown on the display device 9392.
[00478] The various components of the electronic device 9302 may be coupled together by at least one bus, which may include a power bus, a control signal bus, a status signal bus, a data bus, and the like. For brevity, in Figure 93 the various buses are shown as the bus system 9315. Note that FIG. 93 shows only one possible configuration of electronic device 9302. Various other architectures and components may be utilized.
[00479] Some diagrams are given below that show examples of the features and / or user interfaces described herein. In some configurations, features and / or user interfaces may be referred to with respect to the phrases "Sound Focus and Source Tracking," "SoFAST," or "SFAST."
[00480] In the above description, reference numbers are sometimes used with various terms. When the term is used with a reference number, this can mean referring to a particular element shown in at least one of the figures. When a term is used without a reference number, this can generally mean referring to a term that is not limited to any particular figure.
[00481] The term "join" and its variants may indicate direct or indirect connections between elements. For example, the first element attached to the second element can be directly connected to the second element or indirectly connected to the second element through another element.
[00482] The term "processor" should be broadly interpreted to include general purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. is there. Under some circumstances, "processor" may refer to application specific integrated circuits (ASICs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and so on. The term "processor" refers to a combination of processing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, multiple microprocessors, at least one microprocessor working with a digital signal processor (DSP) core, or any combination. It may refer to other such configurations.
[00483] The term "memory" should be broadly interpreted to include any electronic component capable of storing electronic information. The terms memory are random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable. It may refer to various types of processor readable media such as PROM (EEPROM), flash memory, magnetic or optical data storage, and registers. If the processor can read information from memory and / or write information to memory, the memory is said to be communicating electronically with the processor. The memory integrated in the processor communicates electronically with the processor.
[00484] The terms "instruction" and "code" should be broadly construed to include any type of computer-readable statement. For example, the terms "instruction" and "code" may refer to at least one program, routine, subroutine, function, procedure, and so on. The "instruction" and "code" may comprise a single computer-readable statement or multiple computer-readable statements.
[00485] At least one of the features, functions, procedures, components, elements, structures, etc. described with respect to any one of the configurations described herein is compatible, herein. Note that it can be combined with at least one of the features, procedures, components, elements, structures, etc. described for any of the other configurations described. In other words, any compatible combination of features, procedures, components, elements, etc. described herein can be implemented according to the systems and methods disclosed herein.
[00486] The methods and devices disclosed herein may be applied in generally any transmit and receive and / or voice sensing applications, especially in mobile or other portable cases of such applications. For example, the scope of the configuration disclosed herein includes communication devices residing within a wireless telephony communication system configured to use a code division multiple access (CDMA) over-the-air interface. However, methods and devices with the functionality described herein are wired and / or wireless (eg, CDMA, Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), and / or Time Division Simultaneous Can reside in any of a variety of communication systems using a wide range of technologies known to those of skill in the art, such as systems that use voice over IP (VoIP) over a Code Division Multiple Access (TDSCDMA) transmission channel. However, it will be understood by those in the art.
[00487] The communication devices disclosed herein are network-switched networks (eg, wired and / or wireless networks configured to carry audio transmissions according to protocols such as VoIP) and / or circuit-switched networks. It is expressly contemplated that it may be adapted for use in a network that is disclosed herein. Also, the communication devices disclosed herein are used in narrowband coding systems (eg, systems that encode an audible frequency range of about 4 or 5 kilohertz) and / or fullband wideband coding systems and splitband wideband. It is expressly contemplated and disclosed herein that it may be adapted for use in wideband coding systems, including coding systems (eg, systems that encode audible frequencies above 5 kilohertz).
[00488] An example of a codec that can be used with or adapted to be used with the transmitters and / or receivers of the communication devices described herein is "Enhanced Variable Rate Codec, Speech Service Options 3". , 68, and 70 for Wideband Spread Spectrum Digital Systems, described in Third Generation Partnership Project 2 (3GPP2) document C.S0014-C, v1.0, February 2007 (available online at www.3gpp.org). Enhanced Variable Rate Codec, "Selectable Mode Vocoder (SMV) Service Option for Wideband Spread Spectrum Communication Systems, 3GPP2 document C.S0030-0, v3.0, January 2004 (available online at www.3gpp.org) Selectable Mode Vocoder audio codec, document ETSI TS126 092 V6.0.0 ( Adaptive Multi Rate (AMR) audio codecs listed in European Telecommunications Standards Institute (ETSI), Sophia Antipolis Cedex, FR, December 2004, and document ETSI TS 126 192 V 6.0.0. (ETSI, December 2004). ) Has the AMR Wideband audio codec described in. Such codecs can be used, for example, to recover a reproduced audio signal from a received wireless communication signal.
[00489] The presentation of the configurations described is provided to allow those skilled in the art to manufacture or use the methods and other structures disclosed herein. The flowcharts, block diagrams, and other structures illustrated and described herein are examples only, and other variants of these structures are also within the scope of the present disclosure. Various modifications to these configurations are possible, and the general principles presented herein may apply to other configurations as well. Accordingly, this disclosure is not intended to be limited to the configurations shown above, and is disclosed in any manner herein, including the scope of the appended claims that are part of the original disclosure. The widest range should be given that is consistent with the principles and new features.
Those skilled in the art will appreciate that information and signals can be represented using any of a wide variety of techniques and techniques. For example, data, instructions, commands, information, signals, bits, and symbols that may be mentioned throughout the above description are by voltage, current, electromagnetic waves, magnetic or magnetic particles, light fields or optical particles, or any combination thereof. Can be represented.
An important design requirement for the implementation of the configurations disclosed herein is that the compressed audio or audiovisual information (eg, one of the examples identified herein, etc., is coded according to the compression format. Computation-intensive applications such as playback of files or streams to be converted, or applications of wideband communication (for example, voice communication at sampling rates higher than 8 kHz, such as 12, 16, 32, 44.1, 48, or 192 kHz). In particular, it may include minimizing processing delay and / or computational complexity (generally measured in millions of instructions per second or MIPS).
[00492] The devices disclosed herein (eg, any device configured to perform the techniques described herein) are considered suitable for the intended application, with software, and /. Alternatively, it can be implemented in any combination of hardware with firmware. For example, the elements of such a device can be made, for example, as electronic and / or optical devices residing on the same chip or between two or more chips in a chipset. An example of such a device is a fixed array or programmable array of logic elements such as transistors or logic gates, any of which can be implemented as one or more such arrays. Any two or more, or even all, of these elements can be implemented in the same array. Such one or more arrays may be mounted within one or more chips (eg, within a chipset containing two or more chips).
[00493] One or more elements of the various implementations of the devices disclosed herein, in whole or in part, are microprocessors, embedded processors, intellectual property (IP) cores, digital signal processors, FPGAs. Instructions configured to run on one or more fixed or programmable arrays of logic elements such as (field programmable gate arrays), ASSPs (application specific integrated circuits), and ASICs (application specific integrated circuits). Can be implemented as one or more sets of. Any of the various elements of the device implementations disclosed herein are programmed to execute one or more sets or sequences of instructions, also referred to as "processors". It can also be implemented as a machine containing one or more arrays), any two or more of these elements, and even all within the same such one or more computers. Can be implemented in.
[00494] The processors or other means for processing disclosed herein include, for example, one or more electronic devices and / or one or more electronic devices residing on the same chip or between two or more chips in a chipset. Alternatively, it can be manufactured as an optical device. An example of such a device is a fixed array or programmable array of logic elements such as transistors or logic gates, any of which can be implemented as one or more such arrays. Such one or more arrays may be mounted within one or more chips (eg, within a chipset containing two or more chips). Examples of such arrays are fixed or programmable arrays of logical elements such as microprocessors, embedded processors, IP cores, DSPs, FPGAs, ASSPs, and ASICs. The processors or other means for processing disclosed herein are one or more computers (eg, one or more computers programmed to execute one or more sets or sequences of instructions. It can also be implemented as a machine containing an array of) or other processors. The processor described herein is directly related to the procedures of the implementation of the methods disclosed herein, such as tasks related to another operation of the device or system in which the processor is embedded (eg, an audio sensing device). Can be used to perform tasks that do not, or to perform other sets of instructions. Also, some of the methods disclosed herein can be performed by the processor of the audio sensing device, and another part of the method can be performed under the control of one or more other processors. ..
Various exemplary modules, logic blocks, circuits, and tests and other operations described with respect to the configurations disclosed herein can be implemented as electronic hardware, computer software, or a combination thereof. The person skilled in the art will understand. Such modules, logic blocks, circuits, and operations are general purpose processors, digital signal processors (DSPs), ASICs or ASSPs, FPGAs or other programmable logic designed to generate the configurations disclosed herein. It can be implemented or implemented using devices, individual gate or transistor logic, individual hardware components, or any combination thereof. For example, such a configuration may be, at least in part, as a hard-wired circuit, as a circuit configuration made into a purpose-built integrated circuit, or a firmware program loaded into a non-volatile storage device, or a general purpose processor or other. It can be implemented as a software program that is loaded from or loaded into a data storage medium as a machine-readable code that is an instruction that can be executed by an array of logical elements such as a digital signal processing unit. The general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. Processors can be implemented in a combination of computing devices, such as a combination of DSPs and microprocessors, multiple microprocessors, one or more microprocessors associated with a DSP core, or any other such configuration. Software modules include RAM (random access memory), ROM (read-only memory), non-volatile RAM (NVRAM) such as flash RAM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, and hard disks. , Removable disk, or CD -Can reside in non-temporary storage media such as ROM, or in any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write the information to the storage medium. Alternatively, the storage medium may be integrated with the processor. Processors and storage media can reside within the ASIC. The ASIC can reside in the user terminal. Alternatively, the processor and storage medium may reside as separate components in the user terminal. The term "computer program product" refers to a computing device or processor combined with code or instructions (eg, "program") that can be executed, processed or calculated by the computing device or processor.
[00496] The various methods disclosed herein can be performed by an array of logical elements such as processors, and the various elements of the equipment described herein are designed to run on such an array. Note that it can be implemented as a module. As used herein, the term "module" or "submodule" refers to any method, device, device, unit or computer-readable data that includes computer instructions (eg, logical expressions) in the form of software, hardware or firmware. Can point to a storage medium. It should be understood that multiple modules or systems can be combined into one module or system, and one module or system can be separated into multiple modules or systems that perform the same function. When implemented in software or other computer-executable instructions, the elements of a process are essentially code segments that perform related tasks, such as using routines, programs, objects, components, data structures, and so on. The term "software" refers to one or more sets or sequences of instructions that can be executed by an array of source code, assembly language code, machine code, binary code, firmware, macrocode, microcode, logical elements, and so on. It should be understood that any combination of examples is included. The program or code segment may be stored on a processor-readable medium or transmitted over a transmission medium or communication link by a computer data signal embedded in a carrier wave.
[00497] The implementations of the methods, methods, and techniques disclosed herein are arrays of logical elements (eg, in the tangible computer-readable capabilities of one or more computer-readable storage media listed herein). It can also be tangibly implemented as one or more sets of instructions that can be executed by a machine, including (eg, a processor, microprocessor, microcontroller, or other finite state machine). The term "computer-readable medium" can include any medium capable of storing or transferring information, including volatile, non-volatile, removable, and non-removable storage media. Examples of computer-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memory, erasable ROMs (EROMs), floppy (registered trademark) diskettes or other magnetic storage, CD-ROM / DVD or other optical storage, A hard disk or any other medium that can be used to store the desired information, an optical fiber medium, a radio frequency (RF) link, or any other medium that can be used and accessed to carry the desired information. There is a medium. Computer data signals can include any signal that can be propagated through transmission media such as electronic network channels, optical fibers, wireless links, electromagnetic links, RF links, and the like. Code segments can be downloaded via a computer network such as the Internet or an intranet. In any case, the scope of this disclosure should not be construed as limited by such embodiments. Each of the tasks of the methods described herein can be performed directly in hardware, in a software module executed by a processor, or in combination of the two. In a typical application of an implementation of the method disclosed herein, an array of logic elements (eg, a logic gate) should perform one, more, or even all of the various tasks of this method. It is composed of. One or more of the tasks A computer (in some cases all) that is readable and / or executable by a machine (eg, a computer) that contains an array of logical elements (eg, a processor, microprocessor, microcontroller, or other finite state machine). As code (eg, one or more sets of instructions) implemented on a program product (for example, one or more data storage media such as disks, flash memory cards or other non-volatile memory cards, semiconductor memory chips, etc.) Can also be implemented. The tasks of the implementations of the methods disclosed herein can also be performed by two or more such arrays or machines. In these or other implementations, the task may be performed within a device for wireless communication, such as a cellular phone, or within another device that has such communication capabilities. Such devices may be configured to communicate with circuit-switched and / or packet-switched networks (using one or more protocols, such as VoIP). For example, such a device may include an RF circuit configured to receive and / or transmit coded frames. , (Using one or more protocols, such as VoIP), may be configured to communicate with circuit-switched and / or packet-switched networks. For example, such a device may include an RF circuit configured to receive and / or transmit coded frames. , (Using one or more protocols, such as VoIP), may be configured to communicate with circuit-switched and / or packet-switched networks. For example, such a device may include an RF circuit configured to receive and / or transmit coded frames.
[00498] The various methods disclosed herein may be performed by a portable communication device such as a handset, headset, or personal digital assistant (PDA), such as the various devices described herein. It is clearly disclosed that it can be contained in a device. A typical real-time (eg, online) application is a telephone conversation made using such a mobile device.
[00499] In one or more exemplary embodiments, the operations described herein may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, such behavior can be stored on a computer-readable medium as one or more instructions or codes, or transmitted via a computer-readable medium. The term "computer-readable medium" includes both computer-readable storage media and communication (eg, transmission) media. As an example, but not a limitation, a computer-readable storage medium may be a semiconductor memory (including, but not limited to, dynamic or static RAM, ROM, EEPROM, and / or flash RAM), or a dielectric memory, magnetoresistive memory, e. It can include an array of storage elements such as bonic memory, polymer memory, or phase change memory, CD-ROM or other optical disk storage, and / or magnetic disk storage or other magnetic storage device. Such storage media may store information in the form of instructions or data structures that can be accessed by a computer. Communication media can be used to carry the desired program code in the form of instructions or data structures, including any medium that facilitates the transfer of computer programs from one location to another, and is accessed by a computer. It can be equipped with any medium that can be used. Similarly, any connection is properly referred to as a computer-readable medium. For example, the software uses coaxial cable, fiber optic cable, twist pair, digital subscriber line (DSL), or wireless technology such as infrared, wireless, and / or microwave to a website, server, or other remote source. When transmitted from, wireless technologies such as coaxial cable, fiber optic cable, twisted pair, DSL, or infrared, wireless, and / or microwave are included in the definition of medium. Disks used herein ( disk and disc are compact disc (CD), laser disc (registered trademark) (disc), optical disc (disc), digital versatile disc (disc) (DVD), floppy disc (disk) and Including Blu-Ray Disc Association (Universal City, CA), discs typically reproduce data magnetically, and discs optics data with lasers. Play. This combination of the above should also be included within the scope of computer readable media.
[00500] The acoustic signal processing apparatus described herein can benefit from accepting voice inputs to control some operation or separating the desired noise from background noise. It can be incorporated into electronic devices such as communication devices. In many applications, one can benefit from emphasizing or separating a clear desired sound from background sounds generated from multiple directions. Such applications may include human-machine interfaces in electronic or computing devices that incorporate features such as speech recognition and detection, speech enhancement and isolation, and speech activation control. It may be desirable to implement such an acoustic signal processing device suitable for devices that provide only limited processing functionality.
[00501] The elements of various implementations of modules, elements, and devices described herein are, for example, electronic devices and / or light resident on the same chip or between two or more chips in a chipset. It can be made as a device. An example of such a device is a fixed or programmable array of logic elements, such as transistors or gates. One or more elements of the various implementations of the devices described herein, in whole or in part, are logical elements such as microprocessors, embedded processors, IP cores, digital signal processors, FPGAs, ASSPs, and ASICs. It can also be implemented as one or more sets of instructions configured to run on one or more fixed arrays or programmable arrays of.
[00502] One or more elements of an implementation of a device described herein are tasks that are not directly related to the operation of the device, such as tasks related to the operation of the device in which the device is embedded or another operation of the system. Can be used to perform or execute another set of instructions that are not directly related to the operation of the device. Also, one or more elements of such a device implementation correspond to a common structure (eg, a processor used to execute parts of code corresponding to different elements at different times, different elements). It may have a set of instructions that are executed to perform a task at different times, or an electronic and / or optical device configuration that performs operations for different elements at different times.
[00503] It should be understood that the claims are not limited to the exact components and components illustrated above. Various modifications, changes and modifications may be made in the configurations, operations and details of the systems, methods and devices described herein without departing from the claims.<u style="single"> The inventions described in the claims of the original application of the present application are described below.</u><u style="single"> [C1] A method for displaying a user interface on an electronic device, the method comprising:</u><u style="single"> Presenting a user interface, wherein the user interface comprises a coordinate system, wherein the coordinate system corresponds to physical coordinates based on sensor data.</u><u style="single"> To provide a sector selection function that enables selection of at least one sector in the coordinate system, and</u><u style="single"> To provide a sector editing function that enables editing of at least one sector.</u><u style="single"> [C2] The method according to [C1], further comprising displaying the direction of at least one audio signal captured by at least one microphone.</u><u style="single"> [C3] the at least one audio aux signal comprises a speech signal, the method according to [C2].</u><u style="single"> [C4] The method according to [C2], further comprising displaying an icon corresponding to the at least one audio signal.</u><u style="single"> The method according to [C4], wherein displaying the [C5] icon further comprises displaying at least one of an icon relating to the target audio signal and an icon relating to the interfering audio signal.</u><u style="single"> [C6] The method according to [C1], further comprising passing the audio signal shown in at least one sector.</u><u style="single"> [C7] The method according to [C1], further comprising attenuating an audio signal not shown in at least one sector.</u><u style="single"> [C8] The method according to [C1], further comprising showing image data from one or more image sensors.</u><u style="single"> [C9] The method according to [C1], further comprising passing image data based on the one or more sectors.</u><u style="single"> [C10] The method according to [C1], wherein at least one of the sector selection function and the sector editing function operates based on at least one of a group consisting of a single touch input and a multi-touch input.</u><u style="single"> [C11] Displaying at least one touch point corresponding to at least one sector, and</u><u style="single"> Receiving the touch input corresponding to at least one of the touch points</u><u style="single"> Editing at least one sector based on the touch input</u><u style="single"> The method described in [C1], further comprising.</u><u style="single"> [C12] The method according to [C1], further comprising aligning at least a portion of the user interface with a reference plane.</u><u style="single"> [C13] The method according to [C12], wherein the reference plane is horizontal.</u><u style="single"> [C14] The method of [C12], wherein aligning at least a portion of the user interface further comprises mapping a 2D polar plot to a 3D display space.</u><u style="single"> [C15] The method according to [C1], wherein the physical coordinates are earth coordinates.</u><u style="single"> [C16] The method according to [C1], wherein the physical coordinates represent a physical space unrelated to the earth coordinates.</u><u style="single"> [C17] The method according to [C1], wherein the coordinate system maintains a direction independent of the direction of the electronic device.</u><u style="single"> [C18] Recognizing audio signatures</u><u style="single"> Look up the audio signature in the database and</u><u style="single"> Acquiring the identification information corresponding to the audio signature</u><u style="single"> To display the identification information on the user interface</u><u style="single"> The method described in [C1], further comprising.</u><u style="single"> [C19] The method according to [C18], wherein the identification information is an image of a person corresponding to the audio signature.</u><u style="single"> [C20] The method described in [C1], further comprising providing a fixed mode and an editable mode.</u><u style="single"> [C21] The method described in [C1], further comprising padding the selected sector.</u><u style="single"> [C22] The method according to [C1], wherein the sector selection function enables simultaneous selection of a plurality of sectors.</u><u style="single"> [C23] The method according to [C1], wherein the sector editing function enables adjustment of the sector based on single-touch input or multi-touch input.</u><u style="single"> [C24] The method according to [C1], wherein the sector selection function is based on one or more swipe inputs.</u><u style="single"> [C25] The method according to [C24], wherein the one or more swipe inputs indicate a circular area.</u><u style="single"> [C26] The method according to [C24], wherein the one or more swipe inputs are a single swipe.</u><u style="single"> [C27] An electronic device with a display that displays a user interface.</u><u style="single">Here, the user interface comprises a coordinate system, wherein the coordinate system corresponds to physical coordinates based on sensor data, and the display allows selection of at least one sector of the coordinate system. The function provides a sector editing function that allows the display to edit the at least one sector.</u><u style="single"> Electronic device.</u><u style="single"> [C28] The electronic device according to [C27], wherein the display shows the direction of at least one audio signal captured by at least one microphone.</u><u style="single"> [C29] The electronic device according to [C28], wherein the at least one audio signal comprises an audio signal.</u><u style="single"> [C30] The electronic device according to [C28], wherein the display displays an icon corresponding to the at least one audio signal.</u><u style="single"> The electronic device according to [C30], wherein displaying the [C31] icon further comprises displaying at least one of an icon relating to the target audio signal and an icon relating to the interfering audio signal.</u><u style="single"> [C32] The electronic device according to [C27], further comprising an operating circuit coupled to the display, wherein the operating circuit passes an audio signal shown within the at least one sector.</u><u style="single"> [C33] The electronic device according to [C27], further comprising an operating circuit coupled to the display, wherein the operating circuit attenuates an audio signal not shown in at least one sector.</u><u style="single"> [C34] The electronic device according to [C27], wherein the display shows image data from one or more image sensors.</u><u style="single"> [C35] The electronic device according to [C27], further comprising an operating circuit coupled to the display, wherein the operating circuit passes image data based on the one or more sectors.</u><u style="single"> [C36] The electronic device according to [C27], wherein at least one of the sector selection function and the sector editing function operates based on at least one of a group consisting of a single touch input and a multi-touch input. ..</u><u style="single"> [C37] The display further comprises a touch sensor that displays at least one touch point corresponding to the at least one sector and the electronic device receives a touch input corresponding to the at least one touch point. The electronic device according to [C27], wherein the user interface edits the at least one sector based on the touch input.</u><u style="single"> [C38] The electronic device according to [C27], wherein the user interface aligns at least a part of the user interface with a reference plane.</u><u style="single"> [C39] The electronic device according to [C38], wherein the reference plane is horizontal.</u><u style="single"> [C40] The electronic device according to [C38], wherein aligning at least a portion of the user interface further comprises mapping a 2D polar coordinate plot to a 3D display space.</u><u style="single"> [C41] The electronic device according to [C27], wherein the physical coordinates are earth coordinates.</u><u style="single"> [C42] The electronic device according to [C27], wherein the physical coordinates represent a physical space unrelated to the earth coordinates.</u><u style="single"> [C43] The electronic device according to [C27], wherein the coordinate system maintains a direction independent of the direction of the electronic device.</u><u style="single"> [C44] Further comprising an audio signature recognition circuit that recognizes the audio signature, examines the audio signature in the database, acquires the identification information corresponding to the audio signature, and passes the identification information to the display [C44]. The electronic device described in C27].</u><u style="single"> [C45] The electronic device according to [C44], wherein the identification information is an image of a person corresponding to the audio signature.</u><u style="single"> [C46] The electronic device according to [C27], wherein the user interface provides a fixed mode and an editable mode.</u><u style="single"> [C47] The electronic device according to [C27], wherein the user interface pads selected sectors.</u><u style="single"> [C48] The electronic device according to [C27], wherein the sector selection function enables simultaneous selection of a plurality of sectors.</u><u style="single"> [C49] The electronic device according to [C27], wherein the sector editing function enables adjustment of the sector based on a single touch input or a multi-touch input.</u><u style="single"> [C50] The electronic device according to [C27], wherein the sector selection function is based on one or more swipe inputs.</u><u style="single"> [C51] The electronic device according to [C50], wherein the one or more swipe inputs indicate a circular area.</u><u style="single"> [C52] The electronic device according to [C50], wherein the one or more swipe inputs are a single swipe.</u><u style="single"> [C53] A computer program product for displaying a user interface, comprising a non-temporary tangible computer readable medium having an instruction on it, said instruction comprising:</u><u style="single"> A code for causing an electronic device to present a user interface, wherein the user interface comprises a coordinate system and the coordinate system corresponds to physical coordinates based on sensor data.</u><u style="single"> A code for causing the electronic device to provide a sector selection function that enables selection of at least one sector in the coordinate system.</u><u style="single"> A code for causing the electronic device to provide a sector editing function that enables editing of the at least one sector.</u><u style="single"> [C54] The computer program product according to [C53], wherein the instruction further comprises a code for causing the electronic device to display the direction of at least one audio signal captured by at least one microphone.</u><u style="single"> [C55] The computer program product according to [C53], wherein the instruction further comprises a code for causing the electronic device to pass an audio signal shown in the at least one sector.</u><u style="single"> [C56] The computer program product according to [C53], wherein the instruction further comprises a code in the electronic device for attenuating an audio signal not shown in the at least one sector.</u><u style="single"> [C57] The computer program according to [C53], wherein at least one of the sector selection function and the sector editing function operates based on at least one of a group consisting of single-touch input and multi-touch input. Product.</u><u style="single"> [C58] The computer program product according to [C53], wherein the sector selection function enables simultaneous selection of a plurality of sectors.</u><u style="single"> [C59] The computer program product according to [C53], wherein the sector selection function is based on one or more swipe inputs.</u><u style="single"> [C60] A device for displaying a user interface, the device comprising:</u><u style="single"> A means for presenting a user interface, wherein the user interface comprises a coordinate system, which corresponds to physical coordinates based on sensor data.</u><u style="single"> A means for providing a sector selection function that enables selection of at least one sector in the coordinate system, and</u><u style="single"> A means for providing a sector editing function that enables editing of at least one sector.</u><u style="single"> [C61] The device according to [C60], further comprising means for displaying the direction of at least one audio signal captured by at least one microphone.</u><u style="single"> [C62] The device according to [C60], further comprising means for passing the audio signal shown in at least one sector.</u><u style="single"> [C63] The device according to [C60], further comprising means for attenuating an audio signal not shown in at least one sector.</u><u style="single"> [C64] The device according to [C60], wherein at least one of the sector selection function and the sector editing function operates based on at least one of a group consisting of a single touch input and a multi-touch input.</u><u style="single"> [C65] The device according to [C60], wherein the sector selection function enables simultaneous selection of a plurality of sectors.</u><u style="single"> [C66] The device according to [C60], wherein the sector selection function is based on one or more swipe inputs.</u>
242 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23 Sheet 24 Sheet 25 Sheet 26 Sheet 27 Sheet 28 Sheet 29 Sheet 30 Sheet 31 Sheet 32 Sheet 33 Sheet 34 Sheet 35 Sheet 36 Sheet 37 Sheet 38 Sheet 39 Sheet 40 Sheet 41 Sheet 42 Sheet 43 Sheet 44 Sheet 45 Sheet 46 Sheet 47 Sheet 48 Sheet 49 Sheet 50 Sheet 51 Sheet 52 Sheet 53 Sheet 54 Sheet 55 Sheet 56 Sheet 57 Sheet 58 Sheet 59 Sheet 60 Sheet 61 Sheet 62 Sheet 63 Sheet 64 Sheet 65 Sheet 66 Sheet 67 Sheet 68 Sheet 69 Sheet 70 Sheet 71 Sheet 72 Sheet 73 Sheet 74 Sheet 75 Sheet 76 Sheet 77 Sheet 78 Sheet 79 Sheet 80 Sheet 81 Sheet 82 Sheet 83 Sheet 84 Sheet 85 Sheet 86 Sheet 87 Sheet 88 Sheet 89 Sheet 90 Sheet 91 Sheet 92 Sheet 93 Sheet 94 Sheet 95 Sheet 96 Sheet 97 Sheet 98 Sheet 99 Sheet 100 Sheet 101 Sheet 102 Sheet 103 Sheet 104 Sheet 105 Sheet 106 Sheet 107 Sheet 108 Sheet 109 Sheet 110 Sheet 111 Sheet 112 Sheet 113 Sheet 114 Sheet 115 Sheet 116 Sheet 117 Sheet 118 Sheet 119 Sheet 120 Sheet 121 Sheet 122 Sheet 123 Sheet 124 Sheet 125 Sheet 126 Sheet 127 Sheet 128 Sheet 129 Sheet 130 Sheet 131 Sheet 132 Sheet 133 Sheet 134 Sheet 135 Sheet 136 Sheet 137 Sheet 138 Sheet 139 Sheet 140 Sheet 141 Sheet 142 Sheet 143 Sheet 144 Sheet 145 Sheet 146 Sheet 147 Sheet 148 Sheet 149 Sheet 150 Sheet 151 Sheet 152 Sheet 153 Sheet 154 Sheet 155 Sheet 156 Sheet 157 Sheet 158 Sheet 159 Sheet 160 Sheet 161 Sheet 162 Sheet 163 Sheet 164 Sheet 165 Sheet 166 Sheet 167 Sheet 168 Sheet 169 Sheet 170 Sheet 171 Sheet 172 Sheet 173 Sheet 174 Sheet 175 Sheet 176 Sheet 177 Sheet 178 Sheet 179 Sheet 180 Sheet 181 Sheet 182 Sheet 183 Sheet 184 Sheet 185 Sheet 186 Sheet 187 Sheet 188 Sheet 189 Sheet 190 Sheet 191 Sheet 192 Sheet 193 Sheet 194 Sheet 195 Sheet 196 Sheet 197 Sheet 198 Sheet 199 Sheet 200 Sheet 201 Sheet 202 Sheet 203 Sheet 204 Sheet 205 Sheet 206 Sheet 207 Sheet 208 Sheet 209 Sheet 210 Sheet 211 Sheet 212 Sheet 213 Sheet 214 Sheet 215 Sheet 216 Sheet 217 Sheet 218 Sheet 219 Sheet 220 Sheet 221 Sheet 222 Sheet 223 Sheet 224 Sheet 225 Sheet 226 Sheet 227 Sheet 228 Sheet 229 Sheet 230 Sheet 231 Sheet 232 Sheet 233 Sheet 234 Sheet 235 Sheet 236 Sheet 237 Sheet 238 Sheet 239 Sheet 240 Sheet 241 Sheet 242
Every citation, both ways
| Document | Relation | Office | Cited during |
|---|---|---|---|
| TWI755941B | Cited by | Taiwan Province of China | Examiner |
| JP2006261900A | Cites | Japan | – |
| WO2011076286A1 | Cites | World Intellectual Property Organization (WIPO) | – |
| US20100303247A1 | Cites | United States of America | – |
| US20080101624A1 | Cites | United States of America | – |
| JP05333988A | Cites | Japan | – |
| US20150139426A1 | Cites | United States of America | – |
| JP2013522938A | Cites | Japan | – |
| 吉田雅敏ほか6名,“音を視覚化する録音再生システム”,情報処理学会 第69回(平成19年)全国大会講演論文集(2) 人工知能と認知科学,日本,社団法人情報処理学会,2007年 3月 6日,pp. 2-577~2-578 | Non-patent | – | – |
| 久保田祐史ほか4名,“顔追跡による音環境可視化システムのアウエアネスの改善”,情報処理学会 第71回(平成21年)全国大会講演論文集(4) インタフェース コンピュータと人間社会,日本,社団法人情報処理学会,2009年 3月10日,pp. 4-181~4-182 | Non-patent | – | – |
33 members in 6 offices
Priority claims34
| Document | Office | Kind | Date |
|---|---|---|---|
| 201261624181 | United States of America | P | |
| 201261624181 | United States of America | P | |
| 61624181 | United States of America | – | |
| 201261642954 | United States of America | P | |
| 201261642954 | United States of America | P | |
| 61642954 | United States of America | – | |
| 201261713447 | United States of America | P | |
| 201261713447 | United States of America | P | |
| 61713447 | United States of America | – | |
| 201261714212 | United States of America | P | |
| 201261714212 | United States of America | P | |
| 61714212 | United States of America | – | |
| 201261726336 | United States of America | P | |
| 201261726336 | United States of America | P | |
| 61726336 | United States of America | – | |
| 13835881 | United States of America | – | |
| 201313835881 | United States of America | A | |
| 201313835881 | United States of America | A | |
| 2013035926 | United States of America | W | |
| 2013035926 | United States of America | W | |
| 13835881 | – | – | – |
| 61624181 | – | – | – |
| 61642954 | – | – | – |
| 61713447 | – | – | – |
| 61714212 | – | – | – |
| 61726336 | – | – | – |
| US201261624181P | – | – | – |
| US201261642954P | – | – | – |
| US201261713447P | – | – | – |
| US201261714212P | – | – | – |
| US201261726336P | – | – | – |
| US2013035926 | – | – | – |
| US201313835881 | – | – | – |
| WO2013US35926 | – | – | – |
Members33
| Document | Office | Kind | |
|---|---|---|---|
| US2013272097A1 | United States of America | A1 | |
| US2013272538A1 | United States of America | A1 | |
| US2013272539A1 | United States of America | A1 | |
| US2013275077A1 | United States of America | A1 | |
| US2013275872A1 | United States of America | A1 | |
| US2013275873A1 | United States of America | A1 | |
| WO2013154790A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2013154791A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2013154792A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2013155148A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2013155154A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2013155251A1 | World Intellectual Property Organization (WIPO) | A1 | |
| CN104220896A | China | A | |
| CN104246531A | China | A | |
| CN104272137A | China | A | |
| EP2836851A1 | European Patent Office (EPO) | A1 | |
| EP2836852A1 | European Patent Office (EPO) | A1 | |
| EP2836996A1 | European Patent Office (EPO) | A1 | |
| JP2015520884A | Japan | A | |
| IN2283MUN2014A | India | A | |
| IN2195MUN2014A | India | A | |
| US9291697B2 | United States of America | B2 | |
| US9354295B2 | United States of America | B2 | |
| US9360546B2 | United States of America | B2 | |
| CN104220896B | China | B | |
| CN104272137B | China | B | |
| CN104246531B | China | B | |
| US9857451B2 | United States of America | B2 | |
| JP6400566B2This record | Japan | B2 | |
| US10107887B2 | United States of America | B2 | |
| US2019139552A1 | United States of America | A1 | |
| EP2836852B1 | European Patent Office (EPO) | B1 | |
| US10909988B2 | United States of America | B2 |
16 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Cancellation because of no payment of annual feesLAPS | LAPS | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Written request for extension of timeJAPANESE INTERMEDIATE CODE: A601A601 | A601 | |
| Written request for extension of timeJAPANESE INTERMEDIATE CODE: A601A601 | A601 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Report on retrievalJAPANESE INTERMEDIATE CODE: A971007A977 | A977 | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Written request for application examinationJAPANESE INTERMEDIATE CODE: A621A621 | A621 |
Numbers
- Publication
- 6400566
- Publication, DOCDB
- 6400566
- Publication, EPODOC
- JP6400566B
- Application
- 2015505869
- Application, DOCDB
- 2015505869
- Application, EPODOC
- JP20150505869
Titles2
- Japanese
- ユーザインターフェースを表示するためのシステムおよび方法
- English
- Systems and methods for displaying the user interface
Classification
- CPC, 21
- G01S3/80
- G01S15/876
- G10L17/00
- G01S3/8006
- G01S5/18
- G01S5/186
- G01S15/87
- G06F1/1633
- G10L2021/02166
- H04R3/005
- G01S3/8083
- G01S15/86
- H04R1/08
- H04R3/00
- G01B21/00
- G06F3/0484
- G06F3/167
- G06F16/433
- G06F3/04817
- G06F3/04883
- H04S7/40
- IPC, 7
- G06F3 16
- G06F3 0484
- G10L21 0316
- G10L25 51
- H04M1 00
- H04R1 40
- H04R3 00
