System and method for guiding card positioning using phone sensors
12 claims: 8 independent, 4 dependent
- 1短距離通信のための非接触カードを位置合わせさせる方法であって、 コンピューティング デバイスで、非接触カードを前記 コンピューティング デバイスと通信可能に結合するためのプロンプトを生成することと、前記 コンピューティング デバイスの近接センサが、前記非接触カードが前記 コンピューティング デバイスに近接していることを検出することと、前記 コンピューティング デバイスのカメラが、前記非接触カードが前記 コンピューティング デバイスに近づくと、一連の画像をキャプチャすることと、前記非接触カードが前記 コンピューティング デバイスに近づくときに、前記一連の画像を処理して前記非接触カードの位置および軌道を決定 し、前記コンピューティングデバイスに対する前記非接触カードの投影位置を予測することであって、前記一連の画像を処理して前記非接触カードの前記位置および前記軌道を決定することは、一連の画像を処理して前記非接触カードの1つまたは複数の特徴を検出し、前記検出された1つまたは複数の特徴に応答して、前記非接触カードの前記位置および前記軌道を識別するようにトレーニングされた特徴抽出機械学習モデルに前記一連の画像を転送することを含む、 ことと、前記一連の画像に基づいて、前記非接触カードが前記 コンピューティング デバイスに対して目標位置にあることを検出することと、前記 コンピューティング デバイスのインターフェースをトリガして前記非接触カードとの短距離通信を実行することと、を含む方法。
- 2前記一連の画像をキャプチャすることは、前記 コンピューティング デバイスの前記カメラ、前記 コンピューティング デバイスの赤外線センサ、または前記 コンピューティング デバイスのドットプロジェクタのうちの1つまたは複数によって実行され、前記一連の画像は、前記 コンピューティング デバイスで測定された赤外線エネルギおよび可視光エネルギのうちの1つまたは複数に関連する2次元画像情報および3次元画像情報の1つまたは両方を備える、請求項1に記載の方法。
- 3前記方法は、前記非接触カードを配置するという過去の試みに基づいて、前記一連の画像を、前記位置および前記軌道を予測するようにトレーニングされた第2の機械学習モデルに転送することを含む、請求項 1 に記載の方法。
- 4前記第2の機械学習モデルのトレーニングに使用された前記過去の試みは、前記 コンピューティング デバイスのユーザに合わせてカスタマイズされる、請求項 3 に記載の方法。
- 5前記プロンプトは、可視プロンプト、可聴プロンプト、またはそれらの組み合わせを含む、請求項1に記載の方法。
- 6プロセッサ回路と、前記プロセッサ回路に結合されたメモリと、を備えるコンピューティングデバイスであって、前記メモリは、命令を格納するように構成されており、前記命令は、前記プロセッサ回路によって実行されると、前記プロセッサ回路に、非接触カードを前記コンピューティングデバイスと通信可能に結合するためのプロンプトを生成することと、近接センサを介して、前記非接触カードが前記コンピューティングデバイスに近接していることを検出することと、カメラが、前記非接触カードが前記コンピューティングデバイスに近づくときに、一連の画像をキャプチャすることと、前記非接触カードが前記コンピューティングデバイスに近づくときに、前記一連の画像を処理して前記非接触カードの位置および軌道を決定 し、前記コンピューティングデバイスに対する前記非接触カードの投影位置を予測することであって、前記一連の画像を処理して前記非接触カードの前記位置および前記軌道を決定することは、一連の画像を処理して前記非接触カードの1つまたは複数の特徴を検出し、前記検出された1つまたは複数の特徴に応答して、前記非接触カードの前記位置および前記軌道を識別するようにトレーニングされた特徴抽出機械学習モデルに前記一連の画像を転送することを含む、 ことと、前記一連の画像に基づいて、前記非接触カードが前記コンピューティングデバイスに対して目標位置にあることを検出することと、前記 コンピューティング デバイスのインターフェースをトリガして、前記非接触カードとの短距離通信を実行することと、を実行させる、コンピューティングデバイス。
- 7前記一連の画像をキャプチャすることは、前記コンピューティングデバイスのカメラ、前記コンピューティングデバイスの赤外線センサ、または前記コンピューティングデバイスのドットプロジェクタのうちの1つまたは複数によって実行され、前記一連の画像は、前記コンピューティングデバイスで測定された赤外線エネルギおよび可視光エネルギのうちの1つまたは複数に関連する2次元画像情報および3次元画像情報の一方または両方を備える、請求項 6 に記載のコンピューティングデバイス。
- 8前記プロセッサ回路は、前記非接触カードを配置するという過去の試みに基づいて、前記一連の画像を、前記位置および前記軌道を予測するようにトレーニングされた第2の機械学習モデルに転送する、請求項 6 に記載のコンピューティングデバイス。
- 9前記第2の機械学習モデルのトレーニングに使用された前記過去の試みは、前記コンピューティングデバイスのユーザに合わせてカスタマイズされる、請求項 8 に記載のコンピューティングデバイス。
- 10前記プロンプトは、可視プロンプト、可聴プロンプト、またはそれらの組み合わせを含む、請求項 6 に記載のコンピューティングデバイス。
- 11命令を備えるコンピュータ可読媒体であって、前記命令は、処理回路によって実行されると、前記処理回路に、非接触カードをコンピューティングデバイスと通信可能に結合するためのプロンプトを生成することと、近接センサを介して、前記非接触カードが前記コンピューティングデバイスに近接していることを検出することと、カメラが、前記非接触カードが前記コンピューティングデバイスに近づくときに、一連の画像をキャプチャすることと、前記非接触カードが前記コンピューティングデバイスに近づくときに、前記一連の画像を処理して前記非接触カードの位置および軌道を決定 し、前記コンピューティングデバイスに対する前記非接触カードの投影位置を予測することであって、前記一連の画像を処理して前記非接触カードの前記位置および前記軌道を決定することは、一連の画像を処理して前記非接触カードの1つまたは複数の特徴を検出し、前記検出された1つまたは複数の特徴に応答して、前記非接触カードの前記位置および前記軌道を識別するようにトレーニングされた特徴抽出機械学習モデルに前記一連の画像を転送することを含む、 ことと、前記一連の画像に基づいて、前記非接触カードが前記コンピューティングデバイスに対して目標位置にあることを検出することと、前記コンピューティングデバイスのインターフェースをトリガして、前記非接触カードとの短距離通信を実行することと、を実行させる、コンピュータ可読媒体。
- 12前記一連の画像をキャプチャすることは、前記コンピューティングデバイスのカメラ、前記コンピューティングデバイスの赤外線センサ、または前記コンピューティングデバイスのドットプロジェクタのうちの1つまたは複数によって実行され、前記一連の画像は、前記コンピューティングデバイスで測定された赤外線エネルギおよび可視光エネルギのうちの1つまたは複数に関連する2次元画像情報および3次元画像情報の一方または両方を備える、請求項 11 に記載のコンピュータ可読媒体。
Independent claims12
130 paragraphs, as filed
RELATED APPLICATIONS This application receives priority from U.S. patent application Ser. claim. The contents of the aforementioned applications are hereby incorporated by reference in their entirety.
Near field communication (NFC) includes a set of communication protocols that allow electronic devices, such as mobile devices and contactless cards, to communicate information wirelessly. NFC devices may be used in contactless payment systems, similar to those used in contactless credit cards and electronic ticket smart cards. In addition to payment systems, NFC-enabled devices can function as electronic ID documents and key cards, for example.
Contactless devices (e.g. cards, tags, transaction cards, etc.) are bidirectional or unidirectional contactless devices based, for example, on the use of Radio Frequency Identification (RFID) standards, EMV standards, or NFC Data Exchange Format (NDEF) tags. NFC technology may be used for short range communication. Communication uses magnetic field induction between powered electronic devices, including mobile wireless communication devices, and unpowered or passively powered devices, such as transaction cards. may enable communication. In some applications, high frequency wireless communication technology allows the exchange of data between devices over short distances, such as just a few centimeters, and two devices may operate most efficiently in certain configurations.
While the benefits of using NFC communication channels for contactless card transactions are many, including ease of setup and low complexity, one of the challenges faced by NFC data exchange is that contactless cards equipped with small antennas such as It can be difficult to transmit signals between connected devices. Movement of the contactless card relative to the device during an NFC exchange can undesirably affect the NFC signal strength received at the device and interrupt the exchange. Furthermore, features of cards, such as metal cards, for example, can cause noise, signal reception attenuation, or other reflections that can cause NFC read transactions to be erroneously triggered. For systems that use contactless cards for authentication and transaction purposes, delays and interruptions can result in lost transactions and customer dissatisfaction.
A system of one or more computers may be configured to perform certain operations or actions by installing software, firmware, hardware, or a combination thereof on the system that causes or causes the system to perform an action during operation. It can be configured as follows. One or more computer programs may be configured to perform particular operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the action.
According to one general aspect, a method for guiding the positioning of a card relative to a device to a target location includes: a proximity sensor detecting the proximity of the card to the device; In response to being in close proximity to the device, the device captures a series of images of the three-dimensional volume proximate to the device and processes the series of images to identify the cards in the three-dimensional volume proximate to the device. determining a position and trajectory; and predicting a projected position of the card relative to the device based on the card position and card trajectory; and at least one trajectory adjustment that is predicted to reduce the one or more dispersion. identifying one or more variances between the projected position and the target position, including identifying one or more prompts predicted to achieve trajectory adjustment; displaying a prompt for and capturing a series of images, determining a position and trajectory of the card, predicting a projected position of the card, one or more dispersions, at least one trajectory adjustment, and one repeating the steps of identifying one or more prompts and displaying one or more prompts until the variance of one or more falls within a predetermined threshold; and triggering an event on the device. retrieving data from the card in response to the one or more variances being within a predetermined threshold. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the operations of the method.
Implementations may include one or more of the following features. processing the series of images to determine the position and trajectory of the card within a three-dimensional volume proximate to the device using at least one of a machine learning model or a simultaneous localization and mapping (SLAM) process; . The method includes, during an event, capturing a series of images, determining the position and trajectory of the card, predicting the projected position of the card, one or more dispersions, at least one trajectory adjustment, and one or more Identifying multiple prompts and displaying one or more prompts to ensure that the variance remains within predetermined thresholds and allow the device to read data from the card. Including the step of repeating the step. The method in which the step of triggering the event includes initiating a data exchange between the card and the device, the data exchange relating to at least one of a financial transaction and an authorization transaction. Capturing a series of images is performed by one or more of a camera on the device, an infrared sensor on the device, or a dot projector on the device, and the series of images captures infrared energy and visible light energy measured at the device. including one or both of two-dimensional image information and three-dimensional image information related to one or more of. The method includes generating a volume map of a three-dimensional volume proximate to the device using a series of images obtained from one or more of a camera, an infrared sensor, and a dot projector, the volume map comprising: , containing pixel data for multiple pixel locations within a three-dimensional volume proximate to the device. Processing the series of images to determine the position and trajectory of the card includes processing the series of images, processing the volume map to detect one or more features of the card, and responding to the one or more features. features trained to identify the location and trajectory of the card in the volume map. A method that includes transferring the extraction to a machine learning model. Predicting the projected position of the card relative to the device includes transferring the position and trajectory of the card to a second machine learning model trained to predict the projected position based on past attempts to place the card. method including. A second method in which the past attempts used to train the machine learning model are customized to the user of the device. The one or more prompts include at least one of a visual prompt, an audible prompt, or a combination of visual and audible prompts. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium.
According to one general aspect, a device includes a proximity sensor configured to detect whether a card is in proximity to the device, and a series of three-dimensional volumes coupled to the proximity sensor and in proximity to the device. an image capture device configured to capture an image; a processor coupled to the proximity sensor and the image capture device; a display interface coupled to the processor; a card reader interface coupled to the processor; a non-transitory medium storing alignment program code configured to guide to a target position. The alignment program code is operative when executed by the processor to monitor the proximity of the card to the device and cause the image capture device to capture a series of images of a three-dimensional volume in proximity to the device. process the set of images to determine the position and trajectory of the card within a three-dimensional volume in close proximity to the device, and based on the card position and card trajectory, determine the projected position of the card relative to the device. and identifying one or more variances between the projected position and the target position, including identifying one or more prompts to achieve the at least one trajectory adjustment and the at least one trajectory adjustment. the at least one trajectory adjustment is predicted to reduce the one or more variances, and one on the display interface before and during at least one of the card reading operations; or displaying a plurality of prompts and triggering a card reading operation by a card reader interface if the one or more variances are within a predetermined threshold. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the operations of the method.
Implementations may include one or more of the following features. Program code that is operable when executed to process a set of images to determine the position and trajectory of a card within a three-dimensional volume in close proximity to the device is a machine learning model or simultaneous localization and mapping (slam 12. The device of claim 11, using at least one of the following: Card reading operations are associated with one of the following: financial transactions and authorization transactions. The image capture device includes one or more of a camera, an infrared sensor, or a dot projector, and the series of images captures one or more of infrared energy and visible light energy measured by the device. . A series of images is a device that contains one or both of two-dimensional image information and three-dimensional image information. The alignment program code is further configured to generate a volume map of a three-dimensional volume proximate to the device using a series of images, an infrared sensor, and a dot projector, the volume map being a volume map of a three-dimensional volume proximate to the device. A device that contains pixel data for multiple pixel locations within a volume. The device further includes a feature extraction machine learning model trained to place the card within a three-dimensional volume proximate to the device and predict projection position using past attempts to place the card. Past Attempts is a user-specific past attempt at a device. The device in which the one or more prompts include at least one of a visual prompt, an audible prompt, or a combination of visual and audible prompts. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium.
According to one general aspect, a method for guiding a card to a target location relative to a device includes: the device detecting a request to perform a transaction; and using a proximity sensor of the device. , the device's camera and infrared to measure the card's proximity to the device and, when it is determined that the card is in proximity to the device, capture a series of images of a three-dimensional volume in proximity to the device. controlling at least one of the depth sensors and processing the series of images to determine a position and trajectory of the card within a three-dimensional volume proximate the device, the processing comprising: directing the card to a target position; performed by at least one of a machine learning model trained using past attempts to guide the card or a simultaneous localization and mapping (SLAM) process, and based on the position and trajectory of the card; predicting the projected position of the card relative to the device and identifying at least one trajectory adjustment selected to reduce one or more variances and identifying one or more prompts for achieving the trajectory adjustment; identifying one or more dispersions between the projection position and the target position, displaying one or more prompts on a display of the device, and capturing image information; determining the position and trajectory of the card, predicting the projected position of the card, one or more variances, at least one trajectory adjustment, and identifying one or more prompts; and one or more variances. repeating the steps of displaying one or more prompts until the variance is within a predetermined threshold; and triggering a swipe of the card by a card reader of the device if the variance is less than the predetermined threshold; , including the steps of. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the operations of the method.
<figref num="1A">FIG. 3 is a diagram provided to explain the interaction between a contactless card and a contactless card reading device.</figref><figref num="1B">FIG. 3 is a diagram provided to explain the interaction between a contactless card and a contactless card reading device.</figref><figref num="2">FIG. 2 is an illustration of an example operating volume of a near field wireless communication device.</figref><figref num="3">FIG. 3 is an illustration of a sensor bar of a mobile phone that may be configured to perform position alignment as disclosed herein.</figref><figref num="4">1 is a block diagram illustrating example components of one embodiment of a device configured as disclosed herein. FIG.</figref><figref num="5">5 is a flow diagram of example steps of a position alignment system and method that may be performed by the NFC transaction device of FIG. 4. FIG.</figref><figref num="6">FIG. 3 is a detailed flow diagram illustrating example steps that may be performed to align the position of a contactless card with respect to a device.</figref><figref num="7">FIG. 2 is a flow diagram illustrating example steps that may be performed to train a machine learning model as disclosed herein.</figref><figref num="8">FIG. 2 is a flow diagram illustrating example steps that may be performed in a simultaneous localization and mapping (SLAM) process that may be used as disclosed herein.</figref><figref num="9">FIG. 2 is a flow diagram illustrating example steps that may be performed to position a contactless card for NFC communication using a combination of a proximity sensor and an image capture device of a mobile phone device.</figref><figref num="10">FIG. 7 illustrates exemplary phone/card interaction and display during proximity sensing.</figref><figref num="11">FIG. 7 illustrates exemplary phone/card interactions and displays during alignment; FIG.</figref><figref num="12A">3 illustrates an exemplary mobile phone display that may be provided after successful alignment for NFC communication, including a prompt to adjust the position of the contactless card to maximize received signal strength by the mobile device. .</figref><figref num="12B">3 illustrates an exemplary mobile phone display that may be provided after successful alignment for NFC communication, including a prompt to adjust the position of the contactless card to maximize received signal strength by the mobile device. .</figref><figref num="12C">3 illustrates an exemplary mobile phone display that may be provided after successful alignment for NFC communication, including a prompt to adjust the position of the contactless card to maximize received signal strength by the mobile device. .</figref><figref num="13A">3 illustrates an example phone/card interaction as disclosed herein.</figref><figref num="13B">3 illustrates an example phone/card interaction as disclosed herein.</figref><figref num="13C">3 illustrates an example phone/card interaction as disclosed herein.</figref><figref num="14">FIG. 2 is a flow diagram of one embodiment of an example process for controlling a card reader interface of a device using captured image data as disclosed herein.</figref>
The alignment systems and methods disclosed herein facilitate positioning a contactless card relative to a device, such as positioning a contactless card proximate a target location within a three-dimensional target volume. In one embodiment, the alignment system uses a proximity sensor of the device to detect proximity of the contactless card. Upon detection of proximity, a series of images may be captured by one or more imaging elements of the device, including, for example, a camera of the device and/or an infrared sensor/dot projector of the device. The series of images may be processed to determine the position and trajectory of the card relative to the device. The position and trajectory information may be processed by a predictive model to identify trajectory adjustments to reach the target location and one or more prompts to effectuate the trajectory adjustments. Such a configuration uses the mobile device's existing imaging capabilities to provide real-time positioning assistance feedback to the user, thereby increasing the speed and accuracy of contactless card positioning and improving the received NFC signal strength. Maximize.
According to one aspect, the trigger system may automatically initiate near field communication between the device and the card to communicate cryptograms from an applet on the card to the device. The trigger system may operate in response to a darkness level or a change in darkness level of a series of images captured by the device. The trigger system may operate in response to a complexity level or a change in complexity level of a series of images. The trigger system may automatically trigger operations controlled by the device's user interface, such as, for example, automatically triggering a card swipe. The trigger system may be used alone or with the aid of one or more aspects of the position alignment system disclosed herein.
These and other features of the invention will be described with reference to the figures, and like reference numerals are used to refer to like elements throughout. With general reference to the notation and nomenclature used herein, the following detailed description may be presented with respect to a program process running on a computer or a network of computers. These process descriptions and representations are used by those skilled in the art to most effectively convey the substance of their work to those skilled in the art.
A process is herein generally considered to be a self-consistent series of operations that lead to a desired result. The process may be implemented in hardware, software, or a combination thereof. These operations are operations that require physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, or optical signals that can be stored, transferred, combined, compared, and otherwise manipulated. It is sometimes convenient, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities.
Additionally, the operations performed are often referred to with terms such as addition or comparison, which are commonly associated with intellectual operations performed by human operators. In any of the operations described herein that form part of one or more embodiments, such capabilities of a human operator are not necessary or desirable in most cases. Rather, the operations are machine operations. Useful machines for performing the operations of the various embodiments include general purpose digital computers or similar devices.
Various embodiments also relate to apparatus or systems for performing these operations. The apparatus may be specially constructed for the required purpose or may include a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. The processes presented herein are not inherently related to any particular computer or other apparatus. Various general purpose machines may be used with programs written in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the necessary method steps. The required structure for a variety of these machines will appear from the description given.
In the following description, numerous specific details are set forth for purposes of explanation and to provide a thorough understanding thereof. However, it may be apparent that new embodiments may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate description thereof. The intent is to cover all modifications, equivalents, and alternatives consistent with the claimed subject matter.
FIGS. 1A and 1B show a mobile phone device 100 and a contactless card 150, respectively. Contactless card 150 may comprise a payment or transaction card (hereinafter transaction card), such as a credit card, debit card, or gift card, issued by a service provider. In some examples, contactless card 150 is not related to a transaction card and may include, but is not limited to, an identification card or a passport. In some examples, the transaction card may comprise a dual interface contactless transaction card. Contactless card 150 may include a substrate that includes a single layer or one or more stacked layers constructed of plastic, metal, and other materials.
In some examples, the contactless card 150 may have physical characteristics that comply with the ID-1 format of the ISO/IEC7810 standard; otherwise, the contactless card may comply with the ISO/IEC14443 standard. . However, it is understood that a contactless card 150 according to the present disclosure may have different characteristics, and the present disclosure does not require that a contactless card be implemented into a transaction card.
In some embodiments, a contactless card may include an embedded integrated circuit device that can store, process, and communicate data with other devices, such as terminals or mobile devices, via NFC. Common uses for contactless cards include transportation tickets, bank cards, passports, etc. Contactless card standards cover various types embodied in ISO/IEC10536 (tightly coupled cards), ISO/IEC14443 (proximity cards), and ISO/IEC15693 (proximity cards), each of which Incorporated herein by reference. Such contactless cards are intended for operation at very close, near and far distances, respectively, to the associated coupling device.
Exemplary proximity contactless cards and communication protocols that may benefit from the positioning assistance systems and methods disclosed herein include "Contactless Cards", filed Nov. 29, 2018 by Osborn et al. These include those described in US patent application Ser. No. 16/205,119 entitled "System and Method for Cryptographic Authentication." This is incorporated herein by reference (hereinafter the '119 application).
In one embodiment, the contactless card is configured for bidirectional or unidirectional contactless short-range communication based on, for example, radio frequency identification (RFID) standards, EMV standards, or the use of NDEF tags. Equipped with an NFC interface consisting of hardware and/or software. Communications may use magnetic field induction to enable communication between electronic devices, including mobile wireless communication devices. Short-range, high-frequency wireless communication technology allows devices to exchange data over short distances, such as just a few centimeters.
NFC employs electromagnetic induction between two loop antennas when NFC-enabled devices exchange information. ISO/IEC14443-2:2016 (incorporated herein by reference) specifies the characteristics of power and bidirectional communication between a proximity coupling device (PCD) and a proximity card or object (PICC). The PCD generates a high frequency alternating magnetic field. This field is inductively coupled to the PICC to transfer power and is modulated for communication at rates ranging from 106 to 424 kbit/s within the 13.56 MHz radio frequency ISM band over the ISO/IEC18000-3 air interface. Operate. As specified in the ISO standard, PCD transmissions range from at least 1.5A/m (rms) Hmin to 7.5A to support Class 1, Class 2, and/or Class 3 antenna designs for PICC devices. Generate a uniform electric field strength ("H") varying up to Hmax of /m(rms).
In FIGS. 1A and 1B, mobile phone 100 is a PCD device and contactless card 150 is a PICC device. During a typical contactless card communication exchange, as shown in FIG. 1A, the user engages the card with the mobile device via the cell phone 100, for example, by including a prompt 125 on the display 130 indicating the card placement location. You may be encouraged to agree. For purposes of this application, "engaging" a card to a device includes, but is not limited to, bringing the card into the spatial operating volume of the NFC reading device (i.e., mobile phone 100). The operating volume of the NFC reading device is such that the uniform field strength of the signal transmitted between the mobile device 100 and the card 150 is sufficient to support data exchange, the operating volume of the NFC reading device is close to, adjacent to, and / or including surrounding spatial volumes. In other words, a user may engage a contactless card with a mobile device by tapping the card on the front of the device or holding the card within a distance from the front of the device that enables NFC communication. In FIG. 1A, a prompt 125 provided on display 130 is provided to achieve this result. FIG. 1B shows a card placed in an operational volume for a transaction. As shown in FIG. 1B, a reminder prompt, such as prompt 135, may be displayed to the user during a transaction.
An exemplary exchange between phone 100 and card 150 includes activation of card 150 by the RF operating field of phone 100, phone 100 sending a command to card 150, and card 150 sending a response to phone 100. may include. Some transactions may use several such exchanges, and some transactions may be performed using a single read operation of a transaction card by a mobile device.
In one example, successful data transmission may be best achieved by maintaining magnetic field coupling to an extent at least equal to a minimum (1.5 A/m(rms)) magnetic field strength throughout the transaction, and that magnetic field coupling is a signal It can be understood that it is a function of the strength and the distance between the card 150 and the mobile phone 100. When testing the compliance of an NFC-enabled device, for example, determine whether the device's power requirements (determining operating volume), transmitting requirements, receiving requirements, and signal format (time/frequency/modulation characteristics) meet ISO standards. In order to do so, a series of test transmissions are performed at test points within the operating volume defined in the NFC Forum analog specification.
FIG. 2 shows an exemplary operating volume 200 identified by the NFC Analog Forum for use in testing NFC-enabled devices. The operating volume 200 defines a three-dimensional volume disposed around a contactless card reader device (e.g. a mobile phone device) and is preferred for near field communication exchange, e.g. for NFC reading of a card by the device. Can represent distance. To test an NFC device, the received signal may be measured at various test points, such as point 210, to verify that the uniform field strength is within the minimum and maximum range of the NFC antenna class.
Although the NFC standard specifies specific operating volumes and test methods, the principles described herein are not limited to operating volumes with specific dimensions; the method operates based on the signal strength of a specific protocol. It will be readily appreciated that it is not necessary to determine the volume. the power of the PCD device, the type of PICC device, the intended communication between the PCD and PICC device, the duration of communication between the PCD and PICC device, the imaging capabilities of the PCD device, the expected operating environment of the device, and the user's preference for the device. Design considerations, including but not limited to past behavior, may be used to determine the operating volume as used herein. Accordingly, the following description, in various embodiments, refers to a "target volume" that may comprise a working volume or a subset of a working volume.
Although the placement of the card 150 on the phone 100 may appear simple in FIGS. 1A and 1B, a failed transaction is typically the only feedback provided to the user when card alignment is not optimal. A contactless card EMV transaction can involve a series of data exchanges that require a connection for up to 2 seconds. During such transactions, users who carry cards, NFC reading devices, and goods have difficulty finding and maintaining the target position of the card relative to the phone in order to maintain the desired distance for a successful NFC exchange. It can be.
According to one aspect, to overcome these problems, a card registration system and method activates an imaging component of a mobile device to capture a series of images. Using a series of images, the location and trajectory of the card may be determined in real time and the card guided to a preferred distance and/or target location for the NFC exchange. The series of images may also be used to automatically trigger NFC exchanges or operations, for example by measuring the darkness level and/or complexity level of the series of captured images, or their patterns. .
For example, using this information, the alignment method may determine trajectory adjustments and identify prompts associated with the trajectory adjustments for directing the card to the target volume. A trajectory adjustment prompt may be presented to the user using the phone's audio and/or display components to guide the card to a target location within the target volume and/or initiate an NFC read. In various embodiments, a "target location" (or "target location") may be defined at various granularities. For example, the target location may comprise the entire target volume or a subset of the target volume. Alternatively, the target location may be associated with a specific location of the contactless card within the target volume and/or a space surrounding and containing the specific location.
FIG. 3 is a front-facing top 300 of one embodiment of a mobile phone that may be configured to support the alignment systems and methods disclosed herein. Although the phone is shown to include a sensor panel 320 located along the top edge of portion 300, many devices may include fewer or more sensors that may be arranged differently on their devices. , it is understood that the invention is not limited to any particular type, number, arrangement, location, or design of sensors. For example, most phones have front-facing and rear-facing cameras and/or other sensors, any of which may be used for the purposes described herein for alignment guidance.
Sensor panel 320 is shown to include an infrared camera 302, a flood illuminator 304, a proximity sensor 306, an ambient light sensor 308, a speaker 310, a microphone 312, a front camera 314, and a dot projector 316.
Infrared camera 302 may be used with dot projector 316 for depth imaging. The infrared emitter of dot projector 316 may project up to 30,000 dots in a known pattern onto an object, such as a user's face. The dots are photographed by a dedicated infrared camera 302 for depth analysis. Flood illuminator 304 is a light source. Proximity sensor 306 is a sensor that can detect the presence of nearby objects without physical contact.
Proximity sensors are commonly used in mobile devices and act to lock down UI input. For example, detecting (and skipping) when you accidentally tap the touchscreen when you hold your phone to your ear. Exemplary proximity sensors operate by emitting an electromagnetic field or beam of electromagnetic radiation (eg, infrared) at a target and measuring the reflected signal received from the target. The design of the proximity sensor may vary depending on the target configuration. Capacitive or optoelectronic sensors may be used to detect plastic targets, and inductive proximity sensors may be used to detect metal targets. It is understood that other methods of determining proximity are within the scope of this disclosure and that this disclosure is not limited to proximity sensors that operate by emitting electromagnetic fields.
The top part 300 of the phone is also shown to include an ambient light sensor 308 that is used, for example, to control the brightness of the phone's display. Speaker 310 and microphone 312 enable basic telephone functions. Front camera 314 may be used for two-dimensional and/or three-dimensional image capture, as described in more detail below.
FIG. 4 is a block diagram of representative components of a mobile phone or other NFC-enabled device incorporating elements that facilitate card position alignment as disclosed herein. The components include interface logic 440, one or more processors 410, memory 430, display control 435, network interface logic 440, and sensor control 450 coupled via system bus 420.
Each component uses hardware, software, or a combination thereof to perform a specific function. Processor 410 may include various hardware elements, software elements, or a combination of both. Examples of hardware elements are devices, logic devices, components, processors, microprocessors, circuits, processor circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, application specific integrated circuits (ASICs). , Programmable Logic Devices (PLDs), Digital Signal Processors (DSPs), Field Programmable Gate Arrays (FPGAs), Application Specific Standard Products (ASSPs), Systems on Chips (SOCs), Complex Programmable Logic Devices (CPLDs), Memory May include units, logic gates, registers, semiconductor devices, chips, microchips, chipsets, etc. Examples of software elements are software components, programs, applications, computer programs, application programs, system programs, software development programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, It may include a process, a software interface, an application program interface (API), a set of instructions, a computing code, a computer code, a code segment, a computer code segment, a word, a value, a symbol, or any combination thereof. The determination of whether an embodiment is implemented using hardware and/or software elements depends on the desired computational speed, power level, thermal tolerance, processing cycle budget, input data rate, output data rate, memory resources. may vary as necessary for a given implementation depending on any number of factors, such as data bus speed and other design or performance constraints.
Image processor 415 may be any processor or a specialized digital signal processor used for image processing of data received from camera 452, infrared sensor controller 455, proximity sensor controller 457, and dot projector controller 459. (DSP). Image processor 415 may also employ parallel computing using SIMD (Single Instruction Multiple Data) or MIMD (Multiple Instruction Multiple Data) techniques to improve speed and efficiency. In some embodiments, the image processor may comprise a system on a chip with a multi-core processor architecture that enables high-speed real-time image processing capabilities.
Memory 430 may include a computer readable storage medium for storing program codes (such as registration unit program code 432 and payment processing program code 433) and data 434. Memory 430 may also store user interface program code 436. User interface program code 436 may be configured to interpret user input received at user interface elements including physical elements such as a keyboard and touch screen 460. User interface program code 436 may also interpret user input received from graphical user interface elements such as buttons, menus, icons, tabs, windows, widgets, etc. that may be displayed on the user display under the control of display control 435. According to one aspect, and as described in more detail below, memory 430 may also store trigger program code 431. Trigger program code 431 triggers NFC communication between the device and the card, e.g., in response to a determined darkness level and/or complexity level of a series of images captured by camera 452 or other sensor device. Can be used to trigger automatically. In some embodiments, an automatically triggered operation may be an operation that is typically performed in response to user input, for example, activation of a user interface element such as a read button provided on a graphic user interface. Automatically trigger read operations that are typically initiated by Automatic triggering reduces delays and inaccuracies associated with controlling NFC communications using user interface elements.
Examples of computer-readable storage media are any tangible media capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. may include. Program code may be any suitable type of code, including source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, etc. may include executable computer program instructions implemented using. Embodiments may also be implemented at least in part as instructions contained in or on a non-transitory computer-readable medium, which are capable of performing the operations described herein. may be read and executed by one or more processors to enable.
Alignment unit program code 432 comprises program code as disclosed herein for alignment assistance for contactless card/telephone communications. Alignment unit program code 432 may be used by any service provided by the telephone that uses contactless card exchange for authentication or other purposes. For example, a service such as a payment processing service embodied in payment processing program code 433 may use contactless card exchange for authentication during the initial stages of a financial transaction.
System bus 420 provides an interface for system components including, but not limited to, memory 430 and processor 410. System bus 420 can be further interconnected to memory buses (with or without a memory controller), peripheral buses, and local buses using any of a variety of commercially available bus architectures. It can be either.
Network interface logic includes transmitters, receivers, and controllers configured to support various known protocols associated with various types of network communications. Exemplary network interfaces that may be included in a mobile phone implementing the methods disclosed herein include, but are not limited to, a WIFI interface 442, an NFC interface 444, a Bluetooth interface 446, and a cellular interface 448. Not limited.
Sensor control 450 includes a camera 452 (which may include camera technology for capturing two-dimensional and three-dimensional light-based or infrared images), an infrared sensor 454 and associated infrared sensor controller 455, and a proximity sensor 456 and associated proximity sensors. A controller 457 is provided, as well as a subset of sensors that may support the alignment methods disclosed herein, including a dot projector 458 and an associated dot projector controller 459.
Referring now to FIG. 5, a flow diagram of an example process 500 for positioning a contactless card using image information obtained in real time from a sensor of an NFC reading device is shown. The process includes detecting the proximity of a contactless card in step 510, upon detection triggering an image capture in step 515 using the device's imaging capabilities, and processing the captured image series in step 520. include. Processing of the image may be performed at least in part by the alignment unit program code and may include positioning the contactless card within the target volume proximate the device and determining the trajectory of the card at step 525. Processing the images also includes predicting trajectory adjustments to align the card to the target location within the target volume, identifying prompts to accomplish the trajectory adjustments, and displaying the prompts on the device at step 535. may include. Prompts may include one or more instructions (in text or symbolic form), images, including one or more captured images, colors, color patterns, sounds, and other mechanisms.
The process of capturing an image at 515 and processing the image at 520 continues until it is determined at step 540 that the contactless card is at its target location (and/or preferred distance from the device). The alignment process may then initiate or initiate a data exchange transaction/communication between the card and the device at step 545. For example, the alignment process may perform one or more of providing display prompts to a user to initiate a transaction. Alternatively, the alignment process may automatically initiate the data exchange process when alignment is detected at step 540. In embodiments using NFC interface technology, the alignment process may turn on the NFC interface to enable NFC communication, and at step 550, NFC communication may be performed.
FIG. 6 shows the first step of a registration process 600 that uses a machine learning predictive model to process captured images to extract features, position the card within a three-dimensional target volume, and determine the card trajectory. FIG. 3 is a flow diagram of an example embodiment. The system may also use a machine learning predictive model to identify trajectory adjustments to move the card to a target position within the target volume and identify prompts to accomplish the trajectory adjustment.
At step 605, the phone monitors the reflected energy emitted by the device and reflected back to the device, and detects that the card is in close proximity to the device when the reflected energy exceeds a threshold by the proximity sensor. include. In some phones, the proximity sensor may be implemented using an optical sensor chip. Common optical sensor chips include Intersil and Sharp's ISL29003/23 and GP2A, respectively. Both of these sensor chips are primarily active light sensors, providing ambient light intensity in the LUX unit. Such a sensor is implemented as a Boolean sensor. A Boolean sensor returns two values: "NEAR" and "FAR". Thresholding is based on LUX values. That is, the LUX value of the optical sensor is compared with a threshold value. A LUX value above the threshold means the proximity sensor returns "FAR". Anything less than the threshold, the sensor returns "NEAR". The actual value of the threshold is custom defined depending on the sensor chip in use and its optical response, the location and orientation of the chip on the smartphone body, the configuration and reflection response of the target contactless card, etc.
At step 610, in response to the card's proximity to the device, the device begins capturing an image. Image capture may include capturing two-dimensional images using one or more cameras accessible on the device. Two-dimensional images may be captured by a visible light camera, an infrared camera, or both. For example, some mobile devices may include a rear camera that can take high dynamic range (HDR) photos.
Certain mobile devices may include dual cameras that capture images along different imaging planes to create a depth of field effect. Some may further include a "selfie" infrared camera, or may include infrared emitter technology to project a known pattern of infrared dot matrices onto the target, for example. These dots can then be photographed and analyzed with an infrared camera.
The captured images from any one or more of the above sources, and/or subsets or various combinations of the captured images, are then used for image processing and determination of the position and trajectory of the contactless card. may be forwarded to steps 615 and 620 for locating the contactless card containing.
According to one aspect, the image processing includes constructing a volume map of a target volume proximate to the phone, including a region proximate to and/or including at least a portion of the operating volume of the NFC interface of the phone; A volume map is represented as a three-dimensional array of voxels that stores values related to the color and/or intensity of the voxels within the visible or infrared spectrum. In some embodiments, a voxel is a discrete element within an array of volume elements that constitute a conceptual three-dimensional space, such as each of the array of discrete elements into which a representation of a three-dimensional object is partitioned.
According to one aspect, registration involves processing voxels of the target volume to extract features of the contactless card, determining the position of the card within the target volume, and aligning voxels of the target volume constructed at different times. Comparisons include tracking the movement of cards over time and determining the trajectory of the cards. Location and trajectory may be tracked using a variety of processes, including the use of machine learning models and the use of SLAM techniques. Each will be explained in detail below.
Machine learning is a field of artificial intelligence related to mathematical models that can learn, classify, and make predictions from data. Such mathematical models, sometimes called machine learning models, can classify input data into two or more classes, cluster input data between two or more groups, and predict outcomes based on input data. identifying patterns or trends in the input data, identifying distributions of the input data in space, or any combination thereof. Examples of machine learning models are (i) neural networks, (ii) decision trees such as classification trees and regression trees, (iii) naive bias classifiers, logistic regression classifiers, ridge regression classifiers, random forest classifiers, and minimal Classifiers such as absolute shrinkage and selector (LASSO) classifiers, support vector machines, (iv) clusters such as k-means clusters, mean shift clusters, spectral clusters, (v) factorizers, principal component analyzers, kernel principal including factorization devices, such as component analyzers; (vi) ensembles or other combinations of machine learning models; In some examples, neural networks are deep neural networks, feedforward neural networks, recurrent neural networks, convolutional neural networks, radial base function (RBF) neural networks, echo state neural networks, long short-term memory neural networks, and bidirectional neural networks. May include a recurrent neural network, a gated neural network, a hierarchical recurrent neural network, a stochastic neural network, a modular neural network, a spiking neural network, a dynamic neural network, a cascaded neural network, a neurofuzzy neural network, or any combination thereof .
Different machine learning models can be used interchangeably to perform tasks. Examples of tasks that may be performed, at least in part, using machine learning models include various types of scoring, bioinformatics, chemoinformatics, software engineering, fraud detection, customer segmentation, online recommendation generation, adaptive Websites, determining customer lifetime value, search engines, advertising in real-time or near real-time, classifying DNA sequences, emotional computing, performing natural language processing and understanding, object recognition and computer vision, robotic locomotion, and gaming. including the play of optimization and metaheuristics, network intrusion detection, medical diagnosis and monitoring, or predicting when assets such as machinery require maintenance.
Machine learning models may be built through an at least partially automated (eg, with little or no human involvement) process called training. During training, input data may be repeatedly provided to the machine learning model, allowing the machine learning model to identify patterns associated with the input data or to identify relationships between the input data and output data. Training may convert a machine learning model from an untrained state to a trained state. Input data may be divided into one or more training sets and one or more validation sets, and the training process may be repeated multiple times. The split may follow a k-fold cross-validation rule, a leave-one-out rule, a leave-p-out rule, or a holdout rule.
According to one embodiment, a machine learning model identifies characteristics of a contactless card as it approaches an NFC reading device using image information captured by one or more imaging elements of the device. The feature information can be used to identify the position and trajectory of the card within the target volume.
A method 700 of training and using a machine learning model to identify locations and trajectories is now outlined below with reference to the flowchart of FIG. At block 704, training data may be received. In some examples, training data may be received from a remote or local database, constructed from various subsets of data, or entered by a user. Training data may be used in raw form to train machine learning models or may be preprocessed into other forms that may be used to train machine learning models. For example, the raw form of the training data may be smoothed, truncated, aggregated, clustered, or otherwise manipulated into other forms and used to train a machine learning model. In embodiments, the training data may include communication exchange information, historical communication exchange information, and/or information related to communication exchanges. The communication exchange information may be for the general public and/or specific to the user and user account within the financial institution's database system. For example, in the case of position registration, the training data includes processing image data comprising a contactless card from different orientations and different viewpoints to learn the voxel values of the features of the card at those orientations and viewpoints. obtain. For trajectory adjustments and rapid identification, such training data may include data relating to the impact of trajectory adjustments on cards when in different locations. A machine learning model may be trained to identify prompts by measuring the effectiveness of the prompts in achieving trajectory alignment, where effectiveness is measured in one embodiment by time to card alignment. obtain.
At block 706, a machine learning model may be trained using the training data. Machine learning models can be trained in a supervised, unsupervised, or semi-supervised manner. In supervised training, each input of training data may be correlated to a desired output. The required outputs may be scalars, vectors, or different types of data structures such as text or images. This may allow machine learning models to learn mappings between inputs and desired outputs. In unsupervised training, the training data contains the input but not the desired output, so the machine learning model must find the structure in the input on its own. In semi-supervised training, only some inputs in the training data are correlated to the desired output.
At block 708, the machine learning model may be evaluated. For example, the evaluation data set may be obtained via user input or from a database, for example. The evaluation data set may include inputs that correlate to desired outputs. Inputs may be provided to a machine learning model, and outputs from the machine learning model may be compared to desired outputs. The accuracy of a machine learning model can be high if the output from the machine learning model closely corresponds to the desired output. For example, the accuracy of a machine learning model may be high if more than 90% of the output from the machine learning model is the same as the desired output of the evaluation data set, such as current communication exchange information. Otherwise, the accuracy of the machine learning model may be low. The 90% figure is just an example. The percentage of accuracy that is realistic and desirable may depend on the problem and data.
In some examples, if the machine learning model has an insufficient degree of accuracy for a particular task, the process may return to block 706 where the machine learning model further uses additional training data. It can be trained or modified to improve accuracy. If the machine learning model has sufficient accuracy for the particular task, the process may continue to execute block 710.
At this point, the machine learning model has been trained using the training dataset and processes the captured images to determine the position and trajectory, and based on the current position and trajectory the projected position of the card relative to the device. predicting and identifying at least one trajectory adjustment and one or more prompts for achieving the trajectory adjustment.
At block 710, new data is received. For example, new data may be received during alignment for each contactless card communication exchange. At block 712, the trained machine learning model may be used to analyze new data and provide results. For example, new data may be provided as input to a trained machine learning model. As new data is received, the results of feature extraction predictions, position and trajectory predictions may be continuously adjusted to minimize the duration of the alignment process.
At block 714, the results may be post-processed. For example, the results may be added to, multiplied with, or otherwise combined with other data as part of the job. As another example, the results may be converted from a first format, such as a time series format, to another format, such as a count series format. Any number and combination of operations may be performed on the results during post-processing.
Simultaneous localization and mapping (SLAM) has become well defined in the robotics community for on-the-fly reconstruction of 3D image space. For example, "MonoSLAM: Real-Time Single CameraSLAM" by Davidson et al., IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29, No. 6, 2007 (incorporated herein by reference) focuses on localization. Together, we present a real-time algorithm that can recover the 3D trajectory of a monocular camera and quickly navigate previously unknown scenes. According to one aspect, it is understood that the techniques described by Davidson for camera tracking may be exploited for use in the position alignment systems and methods disclosed herein. Rather than tracking the card's advancement into the phone as described above, SLAM technology is used to position the card relative to the phone by tracking the advancement of the phone's camera to the card's detected features. A similar result can be achieved.
Referring now to FIG. 8, the following is a flow diagram illustrating exemplary steps of a mono-SLAM method 800 for contactless card location that may be used to perform the functions of steps 615 and 620 of FIG. Explain. The technique disclosed by Davidson builds a persistent map of scene landmarks that is referenced indefinitely in a state-based framework. Forming a persistent map can be advantageous when camera movement is limited, and SLAM techniques can therefore be beneficial for position alignment processes focused on specific objects such as contactless cards. It can be. Using persistent maps can limit the processing requirements of the algorithm and maintain continuous real-time operation.
SLAM can be used to probabilistically estimate the state of a moving camera and its map on the fly, and use the running estimates to constrain predictive searches and guide efficient processing.
At step 810, an initial probabilistic feature-based map may be generated that represents a snapshot of current estimates of the camera state and all features of interest, and the uncertainty of these estimates at any given time. Maps are initialized at system startup and persist until the end of operation, but can evolve continuously and dynamically as they are updated over time with new image information. Estimates of camera and feature probabilistic states are updated during relative camera/card movement and feature observation. As new features are observed, the map may be expanded with the new state, and features may also be removed if necessary. However, it will be appreciated that once the features of the contactless card are identified with high probabilistic certainty, further image processing can limit subsequent searches to the identified features.
The probabilistic character of the map lies in the propagation over time of a first-order uncertainty distribution that represents not only the average "best" estimates of the camera/card state, but also the size of possible deviations from these values. Mathematically, a map can be represented by a state vector and a covariance matrix P. The state vector x^ consists of the stacked state estimates of the camera and features, and P can be a square matrix of equal dimensions that can be divided into submatrix elements as shown in Equation I below.
Formula I<math num="1"><img file="JP7352002B2_D0001.tif" /></math>
The resulting probability distribution of all map parameters can be approximated as a single multivariate Gaussian distribution in a space of dimensions equal to the total size of the state vectors. Explicitly, the camera state vector xv is defined as the metric 3D position vector r with respect to a fixed world frame W and a "robot" frame R carried by the camera (13 parameters), as shown in Equation II below.<sup>W</sup>, oriented quaternion q<sup>RW</sup>, velocity vector v<sup>W</sup>, angular velocity vector ω<sup>R</sup>Equipped with
Formula II<math num="2"><img file="JP7352002B2_D0002.tif" /></math>
Here, the feature state yi is the 3D position vector of the position of the point feature. According to one aspect, the point features may include contactless card features. Map 825's role allows real-time localization to capture a sparse set of high-quality landmarks. Specifically, each landmark may be considered to correspond to a well-located point feature in 3D space. A camera can be modeled as a rigid body that requires translation and rotation parameters to describe its position. It also maintains estimates of linear velocity and angular velocity. According to one aspect, the camera modeling herein is translated relative to the extracted features (i.e., contactless card) to translate the card movement to maintain linear and angular card speeds for the phone. and rotation parameters may be defined.
In one embodiment, Davison uses relatively large (11x11 pixels) image patches to serve as long-term landmark features in step 830. Camera location information may be used to improve camera displacement and rotation matching. Salient image regions provide repeatable visual landmark detection, e.g., J. Shi and C. Tomasi, "Good Features to Track", Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 593-600. , 1994 (incorporated herein by reference), automatically (i.e., based on the attributes of the card). Once the 3D position of the feature, including its depth, is fully initialized, each feature can be stored as an oriented planar texture. When measuring a feature from a new (relative) camera position, the patch can be projected from 3D to the image plane to create a template for matching with the actual image. The saved feature templates are saved over time and the feature's position can be remeasured over any extended period of time to determine the feature's trajectory.
According to one embodiment, a constant velocity, constant angular velocity model may be used, which assumes that the camera always moves at a constant velocity with an undetermined acceleration occurring within a Gaussian profile. This model gives some smoothness to relative card/camera movements, but gives robustness to systems that use sparse visual measurements. In one embodiment, the predicted location of the image feature (ie, predicted card location) may be determined prior to searching for the feature in the SLAM map.
One aspect of Davison's approach involves predicting feature locations at 850 and limiting image review to the predicted feature locations. Feature matching between the image frames themselves may be performed using a simple normalized cross-correlation search of template patches projected onto the current camera estimate. The template can be scanned over the image, starting at the predicted location, and testing for a match until a peak is found. The assumption of sensible confidence bounds focuses image processing efforts and allows image processing to be performed in real time at high frame rates by restricting the search to a small search region of the input image using sparse maps. Do it like this.
In one embodiment, position prediction may be performed as follows. First, the estimated value of camera position x<sub>v</sub>and feature position y<sub>i</sub>Using , the position of the point feature with respect to the camera is expected to be as shown in Equation III:
Formula III<math num="3"><img file="JP7352002B2_D0003.tif" /></math>
Using a perspective camera, the location (u,v) at which the feature is expected to be found in the image is found using the standard pinhole model shown in Equation IV below.
<math num="4"><img file="JP7352002B2_D0004.tif" /></math>
Here, fk<sub>u</sub>,fk<sub>v</sub>, u<sub>0</sub>, and v<sub>0</sub>with standard camera calibration parameters. This method allows us to actively control the viewing direction towards informative measurements with innovative covariances and limit the maximum number of feature searches per frame to the most informative 10 or 12.
Accordingly, according to one aspect, the performance advantages associated with SLAM, including the ability to perform real-time localization of contactless cards while limiting external image processing, can be achieved by the registration system disclosed herein. It is understood that this is advantageous.
Returning to FIG. 6, according to one aspect, the position alignment system and method predicts trajectory adjustments once the position and trajectory information may be obtained either through machine learning models, SLAM techniques, or other methods. and associated prompts for guiding the card to the target location within the target volume. According to one aspect, the prediction is performed using a predictive model, such as a machine learning model trained and maintained using the machine learning principles described above, to predict the trajectory based on previous trajectory adjustments and the effectiveness of prompts. Identify adjustments and prompts that are customized by user behavior. Trajectory adjustments may be determined, for example, by identifying the variance between the target position and the predicted position and selecting adjustments to the current trajectory to minimize the variance. Effectiveness may be measured in a variety of ways including, but not limited to, the duration of the registration process. For example, in some embodiments, artificial intelligence, neural networks, or other aspects of machine learning models self-select prompts that are most effective in helping users achieve the final result of card alignment. It is possible.
In some embodiments, it is envisioned that a trajectory adjustment may be linked to a set of one or more prompts configured to effectuate a related trajectory adjustment. The set of one or more prompts may include audible and visual prompts, including instructions displayed by the device (in textual or symbolic form), images, including one or more of captured images, colors, color patterns, and sounds. , and one or more forms of other mechanisms. In some embodiments, an effectiveness value may be stored for each prompt, where the effectiveness value relates to past responses and effectiveness of displaying such prompts to achieve trajectory adjustment. The effectiveness value may be used by the machine learning model to select one or more of the trajectory adjustments and/or prompts to guide the card to the target location.
At step 630, a prompt may be displayed on the phone's display. In step 635, the process continues to capture image information, determine position and trajectory, and adjust trajectory until the variance between the target position and the predicted position is determined to be within a predetermined threshold. continue to identify and prompt. The predetermined threshold is a matter of design choice and may vary depending on the target volume or volumes, NFC antenna, etc.
If, in step 635, the variance is determined to be within the threshold, the card may be considered aligned and, in step 630, the NFC mobile device initiates a communication exchange with the card, triggered in step 640. can start.
According to one aspect, the data exchange may be a cryptographic data exchange as described in the '119 application. During the cryptogram exchange, after communication is established between the phone and the contactless card, the contactless card may generate a message authentication code (MAC) cryptogram according to the NFC data exchange format. In particular, this may occur during reading, such as NFC reading of Near Field Wireless Data Exchange (NDEF) tags, which may be created according to the NFC data exchange format. For example, an application running by device 100 (FIG. 1A) may send a message, such as an applet selection message, to contactless card 150 (FIG. 1A) along with an applet ID of an NDEF-generated applet, where the applet: It may be an applet that is stored in the memory of the contactless card and is operational when executed by processing components of the contactless card and generating NDEF tags. Once the selection is confirmed, a series of select file messages followed by read file messages may be sent. For example, a sequence may include "select function file," "read function file," and "select NDEF file." At this point, a counter value maintained by the contactless card is updated or incremented, which may be followed by a "Read NDEF File".
At this point, a message may be generated that may include a header and a shared secret. Next, a session key may be generated. A MAC ciphertext may be created from a message that may include a header and a shared secret. The MAC ciphertext may then be concatenated with one or more blocks of random data, and the MAC ciphertext and random number (RND) may be encrypted with the session key. The ciphertext and header can then be concatenated, encoded as ASCII hex, and returned in the NDEF message format (in response to a "Read NDEF File" message).
In some examples, the MAC ciphertext may be sent as an NDEF tag, and in other examples, the MAC ciphertext may be included in a uniform resource indicator (eg, as a formatted string).
In some examples, an application may be configured to send a request to a contactless card, the request comprising instructions to generate a MAC ciphertext, and the contactless card sends the MAC ciphertext to the application.
In some examples, transmission of the MAC ciphertext is performed via NFC, although the present disclosure is not limited thereto. In other examples, this communication may occur via Bluetooth, Wi-Fi, or other wireless data communication means.
In some examples, the MAC ciphertext may serve as a digital signature for verification purposes. For example, in one embodiment, the MAC ciphertext may be generated by a device configured to perform key diversification using a counter value. In such a system, the sending and receiving devices may be provisioned with the same master symmetric key. In some examples, the symmetric key may comprise a shared secret symmetric key that may be kept secret from all parties other than the sending and receiving devices involved in exchanging secure data. Furthermore, both the transmitting device and the receiving device may be provided with the same master symmetric key, and furthermore, at least a portion of the data exchanged between the transmitting device and the receiving device may be referred to as a counter value. It is understood that the The counter value may comprise a number that changes each time data is exchanged between the transmitting device and the receiving device. Further, the transmitting device and the receiving device may use suitable symmetric encryption algorithms, which may include at least one of a symmetric encryption algorithm, an HMAC algorithm, and a CMAC algorithm. In some examples, the symmetric algorithm used to process the diversified value is any symmetric encryption algorithm optionally used to generate a diversified symmetric key of the desired length. can be provided. Non-limiting examples of symmetric algorithms may include symmetric encryption algorithms such as 3DES or AES128, symmetric HMAC algorithms such as HMAC-SHA-256, symmetric CMAC algorithms such as AES-CMAC.
In some embodiments, the transmitting device may employ the selected encryption algorithm and use the master symmetric key to process the counter value. For example, the sender may select a symmetric encryption algorithm and use a counter that is updated for each conversation between the sending and receiving devices. The sending device may then use the master symmetric key to encrypt the counter value with the selected symmetric encryption algorithm to create a diversified symmetric key. Diversified symmetric keys may be used to process sensitive data before transmitting the results to the receiving device. The transmitting device may then transmit the protected encrypted data along with the counter value to the receiving device for processing.
The receiving device may first obtain the counter value and then perform the same symmetric encryption using the counter value as an input to the encryption and the master symmetric key as the key for the encryption. The output of encryption may be the same diversified symmetric key value created by the sender. The receiving device then retrieves the protected encrypted data and uses a symmetric decryption algorithm along with the diversified symmetric key to decrypt the protected encrypted data to restore the original confidential data. Data can be revealed. Then, when sensitive data needs to be transmitted from the sender to the receiver via the respective sending and receiving devices, different counter values may be selected to generate different diversified symmetric keys. By processing the counter value using the same symmetric encryption algorithm as the master symmetric key, both the sending device and the receiving device may independently generate the same diversified symmetric key. This diversified symmetric key, rather than the master symmetric key, may be used to protect sensitive data.
In some examples, the key diversification value may comprise a counter value. Other non-limiting examples of key diversification values are a random nonce that is generated each time a new diversified key is needed, a random nonce that is sent from a sending device to a receiving device, a sending device and a receiving device. The complete value of a counter value sent from a sending device and a portion of a counter value sent from a receiving device, a counter that is maintained independently by the sending and receiving device but not sent between the two devices, the sending A one-time passcode exchanged between the device and the receiving device, including a cryptographic hash of sensitive data. In some examples, one or more portions of the key diversification value may be used by a party to create multiple diversified keys. For example, a counter may be used as a key diversification value. Additionally, combinations of one or more of the above example key diversification values may be used.
FIG. 9 is a flow diagram 900 illustrating the use of the alignment system disclosed herein to align a contactless card with an NFC mobile device equipped with a proximity sensor and imaging hardware and software. At step 905, the alignment logic detects a request by the device to perform a communication exchange. At step 910, alignment logic uses a proximity sensor of the device to measure reflected energy emitted from and reflected back to the device such that the reflected energy exceeds a predetermined threshold indicative of proximity of the card to the device. This includes deciding when
FIG. 10 shows a contactless card 1030 approaching the operating volume 1020 of the proximity sensor 1015 of the phone 1010. When the phone enters the operating volume 1020, in one embodiment, the infrared beam emitted by the proximity sensor 1015 is reflected back to the proximity sensor 1015 as signal R1035. As the card approaches the phone's operating volume, the reflected signal strength increases until the trigger threshold is reached, at which point the proximity sensor indicates the card is "NEAR." In some embodiments, during a proximity search, the phone's display 1050 provides notification that the card is being searched as shown in FIG. 10, such as by providing a visual or audible indication. You can prompt the user by doing so.
In step 915 (FIG. 9), when the proximity sensor is triggered, the position registration logic causes the position registration logic to capture a series of images of the three-dimensional volume proximate to the device when the reflected energy exceeds a predetermined threshold. , controlling at least one of the device's camera and infrared depth sensor. It can be appreciated that depending on the location of the NFC reader and the location of the camera on the phone, the camera may be selected for image capture with a working volume that overlaps at least a portion of the working volume of the NFC interface of the phone.
At step 920, registration logic processes the captured images to determine the position and trajectory of the card within the three-dimensional volume proximate to the device. As mentioned above, the processing may be performed by one or both machine learning models trained using a simultaneous localization and mapping (SLAM) process with past attempts to guide the card to the goal location. At step 925, the registration process predicts a projected position of the card relative to the device based on the position and trajectory of the card, and at step 930 identifies one or more variances between the projected position and the target position. identifying at least one trajectory adjustment selected to reduce the one or more variances, and identifying one or more prompts for achieving the trajectory adjustment; The alignment process displays one or more prompts on the device's display.
FIG. 11 shows an example display 1105 of phone 1110 capturing image information related to card 1150 within target volume 1120. Display 1105 may include a number of prompts, such as a location prompt 1115 associated with the target location, an image prompt 1130, and an arrow prompt 1140 that may be displayed to the user to assist in guiding the card 1150 to the target location. Image prompt 1130 may include, for example, a portion of an image captured by an imaging component of phone 1110 during position alignment and may be beneficial to the user to assist in the user's understanding of their movement relative to the target. Arrows 1140 may provide directional assistance, such as moving the user to adjust the card upward for proper alignment, as shown in FIG. 11, for example. Other types of prompts may also be used. This may include text instructions, symbols and/or glyphs, audio instructions, and color-based guidance (i.e., displaying a first color (e.g., red) to the user when the card is relatively far from the goal; (transition of screen to green when aligned).
At step 940 (FIG. 9), the position alignment process captures image information, determines the position and trajectory of the card, predicts the projected position of the card, performs one or more dispersions, at least one trajectory adjustment, and identifying the one or more prompts and displaying the one or more prompts until the one or more variances are within a predetermined threshold. At step 945, the registration process may trigger reading of the card by the device's card reader when the variance is less than a predetermined threshold. In some embodiments, the alignment process operates during data exchange between the card and the mobile device, e.g., to provide a prompt to adjust the position of the card if the card moves during reading. You can continue to do so.
12A, 12B, and 12C are examples of display prompts that may be provided by the alignment process after alignment is detected. In FIG. 12A, a prompt 1220 may be provided to notify the user when the card is aligned to the target location. In some embodiments, the interface may provide a link, such as link 1225, to allow the user to initiate a card to be read by phone. In other embodiments, alignment may automatically trigger reading of the card.
In FIG. 12B, a prompt, eg, countdown prompt 1230, may be provided to the user during the card reading process. Additionally, additional prompts, e.g. arrows 1240, are provided to allow the user to correct any movements that may have occurred to the card during the swipe, ensure that the connection is not lost, and improve the success rate of NFC communication. It may be possible to improve. Following the reading, the display provides a notification 1250 to the user regarding the success or failure of the communication exchange, as shown in FIG. 12C.
Accordingly, alignment systems and methods have been shown and described that facilitate aligning a contactless card to a preferred location within a target volume with respect to a contactless card reading device. The alignment logic uses information captured from available imaging devices such as infrared proximity detectors, cameras, infrared sensors, dot projectors, etc. to guide the card to the target location. The captured image information may be processed to identify the card's location, trajectory, and predicted location using one or both of machine learning models and/or simultaneous localization and mapping logic. Trajectory adjustments and prompt identification can be intelligently controlled and customized using machine learning techniques to customize guidance based on user preferences and past behavior. The result is faster and more accurate contactless card alignment and maximizes received NFC signal strength, reducing missed transactions.
The above techniques discuss various methods for guiding the placement of a contactless card to a desired position relative to a device's card reader interface after the proximity of the card is first detected using a proximity sensor. Ta. However, it is understood that the principles disclosed herein can be extended to augment or completely replace proximity sensor information using captured image data to detect card proximity. be done. The captured image information is further processed to determine when the card is in a particular position relative to the card reader interface and perform operations associated with the user interface elements, e.g., by a mobile device without waiting for user input. It may be performed automatically, such as automatically triggering an NFC read operation or other functions. Such a configuration may automatically trigger functions and control operations without requiring user input. For example, the need for human interaction with user interface elements of the device may be avoided.
According to one aspect, image processing logic 415 (FIG. 4) may be expanded to include program code for determining image parameters that may indicate proximity of a card to a card reader. For example, the image parameters may relate to proximity features of the image, ie, features that indicate that the object may be in close proximity to the camera. In some embodiments, the card reader may be placed on the same surface as the camera of the device used to capture the image, and thus the image information may further indicate the proximity of the card to the card reader. In various embodiments, the card reader/camera may be placed on the front or back of the device.
In some embodiments, the image parameter comprises one or more of the darkness level and/or complexity level of the image. For example, referring now briefly to FIGS. 13A and 13B, device 1310 may be configured as described above to obtain the MAC ciphertext from contactless card 1320, e.g., when card 1320 is brought into proximity to device 1310. The device may have a contactless card reading interface configured to. For example, the device may send an applet selection message using the applet ID of the NDEF-generated applet. The applet may be an applet stored in the memory of the contactless card and operable to generate an NDEF tag when executed by a processing component of the contactless card. According to one aspect, a series of images may be captured using the device's camera and analyzed for darkness level and/or complexity level to transfer the NFC read operation from the contactless card's NDEF generation applet. may be determined to automatically trigger when the card is at a preferred distance from the device.
In FIGS. 13A and 13B, for illustrative purposes only, an image 1320 is shown on a display 1340 of a device 1310, a captured image used as disclosed herein to determine card proximity. There is no need for the displayed image to be displayed on device 1310.
According to one embodiment, when a device initiates an NFC communication (e.g., by a user selecting an NFC read operation (such as button 1225) on the device's user interface, or when a device initiates an NFC communication with a card) The device may capture a series of images of a spatial volume proximate to the device, such as by receiving a request from a third party (such as a merchant application or a mobile communication device) to do so. The series of images may be processed to identify one or more image parameters of one or more of the series of images, including, but not limited to, darkness level or complexity level of the images. Complexity level and/or darkness level may be used to trigger NFC reading. Alternatively, or in combination, image processing may include identifying trends and/or patterns in the darkness level and/or complexity level of the series of images or portions of the series of images that are indicative of advancement of the card. Identification of trends and/or patterns within the series of images indicating that the card may be at a preferred distance to the device may be used to automatically trigger an NFC read.
For example, as shown in FIGS. 13A-13C, if the card is further away from the device, the captured image (represented here as image 1330A) will appear at a relatively later time as the card 1320 approaches the device. may be relatively brighter than image 1330B captured in . As the card approaches, as shown in FIG. 13B, the captured image (not visible in FIG. 13C) becomes darker and the card 1320 blocks light from appearing in the image, as shown in FIG. 13C. Until the image becomes darker. This may be because when the card approaches the device, the card (or hand) may block the ambient light received by the camera.
As mentioned above, the presence of the card at a preferred distance from the device is determined depending on the darkness level, the trend in darkness level, the complexity level, and/or the trend in complexity level of the series of captured images. can be done. In particular, the presence of a card may be determined by processing pixel values of a series of images and identifying the darkness level of each processed pixel. For example, assigning grayscale values to pixels. The darkness level of an image may be determined by averaging the darkness levels of image pixels. In some embodiments, the darkness level may be compared to a threshold corresponding to the darkness level when the card is at a preferred distance from the device, eg, such distance supports a successful NFC read operation. In some embodiments, the threshold may be an absolute threshold. For example, in a system where "0" indicates white and "1" indicates dark, if the darkness level is 0.8 or higher, the card is considered "present" and the card reader may be enabled. In other embodiments, the threshold may be a relative threshold that takes into account the ambient light of the environment in which the communication exchange takes place. In such embodiments, the captured first image may provide a baseline darkness level, and the threshold may relate to an amount above the threshold for triggering the NFC communication. For example, the threshold may be a relative threshold. For example, in a dark room where the initial darkness level is 0.8, it may be desirable to delay triggering the NFC communication until the darkness level is 0.95 or higher.
In addition to triggering NFC communications based on individually calculated darkness levels, the system further considers recognizing trends or patterns in darkness levels of images to trigger NFC readings. Recognizing a trend may include, for example, determining an average value across a set of images and triggering a reading when the average value across a set of images meets a threshold value. For example, while an individual image may exceed a threshold, the position of the card may not be stable enough to perform an NFC read; therefore, a predetermined number of images captured in succession will trigger a read. It may be desirable to indicate that a darkness threshold is exceeded before Additionally, or alternatively, sequentially processed images may be monitored to identify spikes and/or plateaus, i.e., sudden shifts in darkness level maintained between successive images indicative of activity in the card reader. It is possible.
In some embodiments, the darkness level of the entire image may be determined by averaging at least a subset of the calculated pixel darkness values. In some embodiments, particular darkness values may be weighted to increase their relevance in darkness level calculations. For example, portions of the image that are known to be close to the card reader or closer to recognized features may be weighted higher than portions that are farther from the card reader.
As mentioned above, a complexity level may be calculated for each captured image, and the complexity level is generally related to the frequency distribution of pixel values within the captured image. In one embodiment, a complexity value may be determined for each pixel by comparing the pixel value of each pixel to the pixel value of one or more neighboring pixels. As shown in FIG. 13B, when the card approaches the device, the background image may be obscured by the card if the card is properly placed. By default, as the card covers the image, the image becomes more uniform, and adjacent pixels usually have the same pixel value. In various embodiments, complexity may be determined for each pixel within the image or for a subset of pixels at previously identified locations within the image. The complexity of each pixel can be determined by examining neighboring pixel values. The complexity level of the entire image may be determined by averaging at least a subset of the calculated pixel complexity values. In some embodiments, particular complexity levels may be weighted to increase their relevance in complexity calculations. For example, those portions of the image that are known to be close to the card reader or identified features may be weighted higher than portions that are farther from the card reader or identified features.
In other embodiments, machine learning methods as disclosed herein may recognize, for example, patterns of pixel darkness/pixel complexity values in successive images indicative of known card activity in proximity to a card reader. can enhance image processing. Such a pattern may, for example, include pixel darkness/complexity levels that vary (ie, darken from top to bottom or bottom to top) in a known manner. The pattern may also include image elements (stripes, icons, printing, etc.) that aid card recognition and may be used as described above, particularly to provide prompts for proper placement of recognized cards. Over time, information related to successful and unsuccessful card reads may be used to determine appropriate image patterns that establish the presence of the card for a successful NFC card communication exchange.
FIG. 14 is a flow diagram of example steps that may be performed to trigger an NFC card read using one or both of the darkness and/or complexity image attributes described above. At step 1410, near field communication may be initiated by the device. Initiation of near field communication may occur through selection of a user interface element on the device, such as read button 1225 in FIG. 12A. Alternatively, or in combination, the initiation of near field communication may occur as a result of an action by an application running on the device, for example an application that utilizes the use of cryptograms from the card for authentication or other purposes. .
During initiation of NFC communication, at step 1420, a camera of the device, such as a front camera, may capture a series of images of a spatial volume in front of the device camera. In some embodiments, 60, 120, 240 or more images may be captured every second, although the present disclosure is not limited to capturing a particular number of images in a sequence. At step 1430, the image may be processed to identify one or more image parameters, such as a darkness level representative of the distance between the card and the device. At step 1440, the processed darkness level of the image is compared to a predetermined darkness level, eg, a darkness level associated with a preferred distance for near field communication operations. When it is determined in step 1450 that the darkness level corresponds to a preferred darkness level for an NFC read operation, an NFC read operation may be automatically triggered, e.g., to communicate ciphertext from an applet on the card. .
In some embodiments, automatic triggering of NFC read operations may bypass or replace triggers historically provided by user interface elements. For example, in some embodiments, a graphical user interface element such as a read button (1225) allows a user to activate NFC communication when the user determines that the card may be properly placed with respect to the device. may be provided on the device to do so. In some embodiments, user interface elements may be associated with functionality such as read operations. It will be appreciated that the techniques described herein may be used to trigger other user interface elements and may automatically trigger various corresponding related functions. The automatic triggers disclosed herein may reduce delays and inaccuracies historically associated with controlled user interface elements and improve NFC communication flows and success rates.
Accordingly, systems and methods have been shown and described for detecting the presence of a card and triggering an NFC read using captured image information. Such systems may utilize machine learning methods and/or SLAM methods described in more detail above to provide additional guidance before triggering a card swipe. Such a configuration may improve card placement and improve the success rate of NFC communication exchanges.
The terms "system," "component," and "unit" as used in this application refer to a computer-related entity that is either hardware, a combination of hardware and software, software, or running software. is intended, examples of which are provided herein. For example, a component may include a process running on a processor, a processor, a hard disk drive, a plurality of storage drives, a non-transitory computer-readable medium (either optical and/or magnetic storage medium), an object, an executable, It can be, but is not limited to, a thread of execution, a program, and/or a computer. As an illustration, both an application running on a server and a server can be a component. One or more components can exist within a process and/or thread of execution, and a component can be localized to one computer and/or distributed across two or more computers.
Additionally, the components may be communicatively coupled to each other through various types of communication media to coordinate operation. Coordination may include one-way or two-way information exchange. For example, components may communicate information in the form of signals communicated over a communication medium. Information may be implemented as signals assigned to various signal lines. In such an assignment, each message is a signal. However, further embodiments may alternatively use data messages. Such data messages may be sent via various connections. Exemplary connections include parallel interfaces, serial interfaces, and bus interfaces.
Some embodiments, along with derivatives thereof, may be described using the phrases "one embodiment" or "an embodiment". These terms mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in this specification are not necessarily all referring to the same embodiment. Furthermore, it is recognized that the features described above can be used together in any combination, unless stated otherwise. Accordingly, any features discussed separately may be used in combination with each other, unless it is noted that the features are incompatible with each other.
With general reference to the notation and nomenclature used herein, the detailed description herein is in terms of functional blocks or units that may be implemented as program procedures running on a computer or a network of computers. can be presented. These procedural descriptions and representations are used by those skilled in the art to most effectively convey the substance of their work to those skilled in the art.
A procedure is herein generally considered to be a self-consistent series of operations that lead to a desired result. These operations are operations that require physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It is sometimes convenient, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities.
Additionally, the operations performed are often referred to with terms such as addition or comparison, which are commonly associated with intellectual operations performed by human operators. In any of the operations described herein that form part of one or more embodiments, such capabilities of a human operator are not necessary or desirable in most cases. Rather, the operations are machine operations. Useful machines for performing the operations of the various embodiments include general purpose digital computers or similar devices.
Some embodiments may be described using the terms "coupled" and "connected," along with their derivatives. These terms are not necessarily intended as synonyms for each other. For example, some embodiments use the terms "connected" and/or "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. It can be explained as follows. However, the term "coupled" can mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.
It is emphasized that an Abstract of the Disclosure is provided to enable the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Moreover, in the foregoing detailed description, various features may be grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment. In the appended claims, the terms "comprising" and "in" are used as the plain English equivalents of the respective terms "comprising" and "herein," respectively. Furthermore, terms such as "first," "second," "third," etc. are used merely as labels and are not intended to impose numerical requirements on them.
What has been described above includes examples of the disclosed architecture. Of course, it is not possible to describe every possible combination of components and/or methodologies, but one skilled in the art will recognize that many more combinations and permutations are possible. Accordingly, the new architecture is intended to embrace all such changes, modifications, and variations within the spirit and scope of the appended claims.
23 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23
Every citation, both ways
| Document | Relation | Office |
|---|---|---|
| WO2018016203A1 | Cites | World Intellectual Property Organization (WIPO) |
| JP2016143982A | Cites | Japan |
32 members in 12 offices
Members32
| Document | Office | Kind | |
|---|---|---|---|
| US10498401B1 | United States of America | B1 | |
| CA3121974A1 | Canada | A1 | |
| US2021021305A1 | United States of America | A1 | |
| WO2021011384A1 | World Intellectual Property Organization (WIPO) | A1 | |
| US10903876B1 | United States of America | B1 | |
| US2021067202A1 | United States of America | A1 | |
| AU2020314553A1 | Australia | A1 | |
| SG11202105749SA | Singapore | A | |
| CN113557551A | China | A | |
| KR20220035021A | Republic of Korea | A | |
| EP4000042A1 | European Patent Office (EPO) | A1 | |
| US11374622B2 | United States of America | B2 | |
| JP2022539601A | Japan | A | |
| US2022360297A1 | United States of America | A1 | |
| AU2020314553B2 | Australia | B2 | |
| AU2023202848A1 | Australia | A1 | |
| JP7352002B2This record | Japan | B2 | |
| JP2023166561A | Japan | A | |
| AU2023202848B2 | Australia | B2 | |
| US12003285B2 | United States of America | B2 | |
| AU2024205485A1 | Australia | A1 | |
| NZ776491A | New Zealand | A | |
| EP4000042B1 | European Patent Office (EPO) | B1 | |
| EP4000042C0 | European Patent Office (EPO) | C0 | |
| ES3014412T3 | Spain | T3 | |
| EP4550263A2 | European Patent Office (EPO) | A2 | |
| JP7675142B2 | Japan | B2 | |
| EP4550263A3 | European Patent Office (EPO) | A3 | |
| PL4000042T3 | Poland | T3 | |
| CN113557551B | China | B | |
| JP2025111676A | Japan | A | |
| CN120707747A | China | A |
9 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Report on retrievalJAPANESE INTERMEDIATE CODE: A971007A977 | A977 | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Written request for application examinationJAPANESE INTERMEDIATE CODE: A621A621 | A621 |
Numbers
- Publication
- 7352002
- Application
- 2022500707
Titles2
- Japanese
- 電話センサを使用してカードの位置決めを案内するためのシステムおよび方法
- English
- System and method for guiding card positioning using phone sensors
Classification
- CPC, 30
- G06T17/00
- H04B5/77
- G06T7/73
- G06N20/00
- G01B11/00
- G06F16/5854
- H04M2250/52
- H04M1/0264
- H04M2250/04
- G06T2207/20084
- G06T2207/20081
- G06T7/246
- G06T2207/30241
- G06Q20/32
- G06Q20/3278
- G06Q20/341
- G06Q20/352
- G09G2354/00
- G06F3/147
- G06T7/74
- H04M1/72454
- H04N23/60
- G06T15/08
- G06Q20/3276
- G06F3/14
- G06T7/521
- G06T2207/10048
- G06T2207/10028
- G06F16/583
- H04N23/90
- IPC, 5
- G06T7 73
- G06K7 015
- G06K7 10
- G06V10 70
- H04N23 90
