Device, system and method for identification of object in an image, and a transponder
40 claims: 24 independent, 16 dependent
- 1画像登録装置で登録された画像内における少なくとも1つの物体の識別のための装置であって、前記少なくとも1つの物体のそれぞれに該物体に関連する識別情報が保存された無線タグが設けられ、 少なくとも1つの質問信号を送信するように動作可能な少なくとも1つの送信器と、 前記少なくとも1つの質問信号に応答した無線タグからの無線応答信号を検出するように動作可能な少なくとも2つのセンサとを備え、 無線タグからの前記無線応答信号がその物体に関連する識別情報を含み、 前記無線タグのそれぞれへの方向を、前記少なくとも2つのセンサにより検出された前記無線応答信号の位相差に基づき推定する、装置。
- 2前記少なくとも1つの質問信号が放射ヌルを含む、請求項1に記載の装置。
- 3前記少なくとも1つの送信器が、前記少なくとも2つのセンサのうち少なくとも1つによって設けられている、請求項1又は2に記載の装置。
- 4前記少なくとも1つの質問信号の送信後の時間窓内において前記少なくとも2つのセンサにより検出された前記無線応答信号のみ検出できるように動作可能な請求項1~3のいずれか一項に記載の装置。
- 5前記少なくとも2つのセンサ及び前記送信器が、電磁信号を受信するように動作可能なアンテナである、請求項1~4のいずれか一項に記載の装置。
- 6前記送信器が無線周波数送信器であり前記少なくとも2つのセンサが無線周波数受信器である 、請 求項1~5のいずれか一項に記載の装置。
- 7前記無線タグからの 前記 識別情報及び前記無線タグの 前記推定された 方向を、前記画像登録装置で登録された 前記 画像の画像データと一緒に処理するのに適した処理装置を更に備える、請求項1~6のいずれか一項に記載の装置。
- 8前記処理装置がリアルタイムで処理を行うのに適している、請求項7に記載の装置。
- 9前記無線タグがRFIDタグであり、前記 少なくとも2つの センサが、少なくとも2つのアンテナを有するRFIDリーダによって設けられる、請求項1~8の少なくとも一項に記載の装置。
- 10RFIDアンテナのマトリクスが設けられた、請求項7に記載の装置。
- 11前記RFIDタグが、アクティブRFIDタグ又はパッシブRFIDタグである、請求項 9 に記載の装置。
- 12前記無線タグが、端末内、端末のカバー内、又は人物により装着されるアイテム(例えば、時計内、ジュエリー、ブレスレット、IDカード内)、又は建物若しくは商品に配置されるユニット内に配置されている、請求項1~11の少なくとも一項に記載の装置。
- 13前記画像登録装置がカメラ又はフィルム/ビデオカメラである、請求項1~12の少なくとも一項に記載の装置。
- 14前記装置が端末に組み込まれており、前記端末が、携帯電話、カメラ、フィルム/ビデオカメラ、PC又はタブレットのうちの1つである、請求項1~10の少なくとも一項に記載の装置。
- 15前記画像登録装置で登録された画像を表示するためのディスプレイ装置を備える、請求項1~14の少なくとも一項に記載の装置。
- 16前記無線 応答 信号を登録するための前記 少なくとも2つの センサが、前記画像登録装置の画像登録センサに配置されている、請求項1~15の少なくとも一項に記載の装置。
- 17携帯電話、カメラ、フィルム/ビデオカメラ、PC又はタブレットのうちの少なくとも1つに装着するように構成された、請求項1~10の少なくとも一項に記載の装置。
- 18前記少なくとも1つの物体が、建物、アイテム/物品、人物又は動物のうち少なくとも1つである、請求項1~15の少なくとも一項に記載の装置。
- 19前記 識別 情報が、テキスト、オーディオ、動画、名前、住所、eメール、インターネットに保存された情報へのリンク、又は、インターネットに保存された情報への参照のうち少なくとも1つである、請求項1~18の少なくとも一項に記載の装置。
- 20前記識別された少なくとも1つの物体の前記 識別 情報及び位置を前記少なくとも1つの物体の前記画像と共に保存するように動作可能なメモリユニットを更に備える、請求項1~19の少なくとも一項に記載の装置。
- 21請求項1~20のいずれか一項に記載の、画像内において少なくとも1つの物体を識別する装置を備える撮像装置。
- 22画像内における少なくとも1つの物体の識別のための方法であって、前記少なくとも1つの物体のそれぞれに該物体に関連する識別情報が保存された無線タグが設けられ、 少なくとも1つの質問信号を送信することと、 前記少なくとも1つの質問信号に応答した無線タグからの無線 応答 信号であって、前記物体の 前記 識 別情 報を含む無線 応答 信号を少なくとも2つのセンサにより検出することと、 前記少なくとも1つの質問信号に応答した前記無線タグのそれぞれの少なくとも1つの方向を、前記 少なくとも2のセンサにより 検出された 前記 無線 応答 信号の位相差に基づき推定することとを含む方法。
- 23前記少なくとも1つの質問信号が放射ヌルを含む、請求項22に記載の方法。
- 24前記少なくとも1つの質問信号の送信後の時間窓内に到達した前記無線応答信号のみを検出することを更に含む、請求項22又は23に記載の方法。
- 25前記質問信号が、強度最大の第1の放射パターンを含む、請求項22~24のいずれか一項に記載の方法。
- 26前記質問信号が、放射ヌルを含む第2の放射パターンを更に含む、請求項22~25のいずれか一項に記載の方法。
- 27放射ヌルを含む放射パターンを用いることによって撮像装置の視野内の角度範囲を走査することを更に含む、請求項22~26のいずれか一項に記載の方法。
- 28前記少なくとも2つのセンサにより検出された前記無線応答信号を処理することによって角度範囲を走査することを更に含む、請求項22~25のいずれか一項に記載の方法。
- 29前記無線タグの位置判定を更に含む、請求項22~28のいずれか一項に記載の方法。
- 30前記少なくとも1つの質問信号及び前記無線 応答 信号が電磁信 号で ある、請求項22~29の少なくとも一項に記載の方法。
- 31前記電磁信号が電 波で ある、請求項30の少なくとも一項に記載の方法。
- 32前記画像を画像登録装置で登録することを更に含む、請求項22~31の少なくとも一項に記載の方法。
- 33前記少なくとも1つの無線タグからの前 記識 別情報、及び、前記少なくとも1つの無線タグの 前記 推定された方向を、前記画像登録装置からの画像データと共に処理し、前記画像を前記少なくとも1つの物体のそれぞれに割り当てられた情報と共にディスプレイ装置上に表示することを更に含む、請求項 32 に記載の方法。
- 34登録、処理、表示のうち少なくとも1つをリアルタイムで行う、請求項22~33の少なくとも一項に記載の方法。
- 35前記画像を、前記少なくとも1つの物体のそれぞれに割り当てられた情報と共に記憶装置に保存することを更に含む、請求項22~34の少なくとも一項に記載の方法。
- 36請求項22~35のいずれか一項に記載の、少なくとも1つの物体の識別のための方法を実行するためのコンピュータ読取可能プログラムコードを備えた、コンピュータ読取可能媒体に保存されたコンピュータプログラム。
- 37画像内における少なくとも1つの物体の識別のためシステム であって 、 物体に割り当てられ、前記物体のための無線タグを備えることに適した少なくとも1つの無線トランスポンダ装置 であって、 前記物体の識別のための情報を保存する記憶装置と、 質問信号に対する応答である無線信号を送信する送信器であって、前記無線信号が前記物体の識別のための 前記 保存された情報を含む 、 送信器と を備える無線トランスポンダ装置、及び 画像登録装置で登録された画像内における少なくとも1つの物体の識別のための、請求項1~21のいずれか一項に記載の装 置 を備え るシ ステム。
- 38前記無線タグからの前 記識 別情報及び前記無線タグの 前記推定された 方向を、前記画像登録装置で登録された前記画像の画像データと共に処理するのに適した処理装置を更に備える、請求項37に記載のシステム。
- 39画像登録装置で登録された画像内における少なくとも1つの物体の識別のための装置であって、前記少なくとも1つの物体のそれぞれに無線タグが設けられ、前記装置が、前記無線タグからの無線 応答 信号を 検出する少なくとも2つのセンサであって、前記無線タグへの方向が、前記少なくとも2つのセンサにより検出された前記無線応答信号の位相差に基づき推定される、 少なくとも 2 つのセンサを備え、前記無線 応答 信号が、前記物体の識別のための情報を含む、装置。
- 40請求項2~20の少なくとも一項に記載の装置を更に備える、請求項39に記載の装置。
Independent claims40
69 paragraphs, as filed
0001The present invention relates to devices, systems, and methods for identifying objects in digital images. The present invention also relates to a wireless transponder device.
0002In recent years, social media on the Internet has contributed to changes in the way people communicate. The ever-increasing capacity and existence of wireless networks has also contributed to this, and it is now possible to communicate through social media from almost anywhere there are people. This has led people to use social networks more often to convey and exchange information about what to do or what to participate in. It is exchanged in any form of text, images, moving images, or a combination thereof. This method of communication is spreading throughout society and is extremely prevalent around the world.
0003One particular use of this technology is to find yourself or others in images shared with friends on social networks. By visualizing oneself in this way, one indicates the need or desire to show that the person is participating in something, alone or with others. Currently, it is necessary to identify oneself or another person in an image manually or by image recognition. Tags are used for this. Meaning In addition to electronic labels, text tags are also used on Internet social networks to find or publish information in lists / images. It is most often used on Facebook® for the act of identifying or referencing a person in an image.
0004Patent Document 1 describes a solution for providing an RFID reader in a camera or a mobile phone with a digital camera / video camera. Place the RFID transponder tag at the target point, eg, a historical monument. The tag stores data about the historical monument and sends that data to the camera's RFID reader. The camera user can then take an image of the monument and receive information about the monument. The multimedia file is then tagged so that the improved multimedia file can contain information about the memorial read by the RFID reader along with the image. Later users can enjoy them in their local environment. Tags can be supplemented with time / date and GPS location generated from your phone.
0005Patent Document 2 relates to a method of tracking an object in an image, and determines the position of the object in 3D space. RF tags (eg RFID tags) are attached to the object. This system uses three-sided surveying and requires an external positioning device to be placed near the object in order to receive the signal from the RF tag. Three-range multilateration requires three (or more) transmitter / receiver units to be placed around the ID tag to be positioned. By measuring the time it takes for a signal to reach three different points in space from an ID tag, the point at which the signal originated in space is determined by trilateration, thereby determining the position of the ID tag. It becomes possible to do.
0006Non-Patent Document 1 relates to a system for rediscovering passive RFID tags along a passage in a warehouse. The digital camera is placed in the middle of the antenna placement configuration. A camera with an antenna is placed in a fixed position in a warehouse with known dimensions. The location of the passive RFID tag is registered and the location can be marked on the image. This system is not designed to solve the needs of the present invention because it is based on a dedicated reader that is neither owned by the general public nor of practical size. The system is also being developed to rediscover items in situations where you know what you are looking for. This system verifies that an item exists and where it is.
0007An alternative wireless location technology is GPS. GPS can be used to provide information about where the image was taken (shooting position), but cannot alone provide information about what or who is in the image.
0008Patent Document 3 describes a system for personal identification of an imaged product. Combine the information from the tagged item with the image of the item itself. This tag is in the form of an RF tag. Animate is needed to establish an estimate of the orientation to the tagged object for the imager. Video allows individual RF tags to be tracked by an imaging device that uses a highly directional antenna beam at its maximum peak. The information obtained from the tagged objects becomes information in a dedicated database, where it is used to find compatibility between similar products imaged. However, when the size of the antenna is limited, it is not possible to generate a highly directional radiation pattern to locate the RF tag, as in the case of the antenna of an imaging device as a mobile phone. There is. Also, multipath propagation produces pseudo signals, but this problem is unsolved in the prior art.
<p num="0009"><patcit num="1"><text>U.S. Patent Application Publication No. 2007/028499</text></patcit><patcit num="2"><text>U.S. Patent Application Publication No. 2010/0103173</text></patcit><patcit num="3"><text>U.S. Patent Application Publication No. 2005/0104956</text></patcit></p>
<p num="0010"><nplcit num="1"><text>Hislop, D.Lekime, M.Drouguet, and C.Craeye, "A prototype 2D direction finding system with passive RFID tag,", Proc.European Conference on antennas and Propagation, April 2010, Barcelona, Spain</text></nplcit><nplcit num="2"><text>Shubair et al. "A setup for the Evaluation of MUSIC and LMS Algorithms for a Smart Antenna System"; Journal of Communications, Vol.2, No.4, June 2007</text></nplcit></p>
<p num="0011"> An object of the present invention is to find a solution to the above-mentioned problem. The present invention is defined in the appended claims.</p>
<p num="0012"> The present invention provides a concept for identifying at least one object in an image registered with an image registration device. At least one object is provided with a radio tag having identification information associated with that object. The device comprises at least one sensor that registers the radio signal from the radio tag, which contains information for identifying at least one object. The radio signal is also used to register at least one direction to the radio tag of at least one object.</p><p num="0013"> Therefore, in the image, at least one object is identified and its position with respect to at least one object is determined.</p><p num="0014"> The present invention is a device for identifying at least one object in an image registered by an image registration device, and each of the at least one object is provided with a wireless tag in which identification information related to the object is stored. With at least one transmitter capable of operating to transmit at least one question signal and at least two sensors capable of detecting a radio response signal from a radio tag that responded to at least one question signal. The device comprises which the radio response signal from the radio tag contains identification information associated with the object and estimates the direction of each of the radio tags based on the phase difference of the radio response signals detected by at least two sensors. I will provide a.</p><p num="0015"> At least one interrogation signal may contain a radiating null. At least one transmitter may be provided by at least one of at least two sensors. The device for identification may be operable so that only the radio response signal detected by at least two sensors can be detected within the time window after transmission of at least one interrogation signal. At least two sensors and transmitters may be antennas that can operate to receive electromagnetic signals. The transmitter may be a radio frequency transmitter and at least two sensors may be radio frequency receivers, or the transmitter may be an acoustic transmitter and at least two sensors may be acoustic receivers, or The transmitter may be an infrared transmitter and at least two sensors may be infrared receivers, or the transmitter can operate to transmit visible light so that at least two sensors receive visible light. It may be operational.</p><p num="0016"> The device may further include a processing device suitable for processing the registered identification information from the wireless ID tag and the registration direction of the wireless ID tag together with the image data of the image registered by the image registration device. ..</p><p num="0017"> Registration and processing may be done in real time. The sensor for registering the wireless signal may be arranged in the image registration sensor (CMOS chip) of the image registration device of the terminal. The device may further comprise a memory unit capable of operating to store the information and location of at least one identified object along with an image of that at least one object.</p><p num="0018"> In one embodiment, the radio ID tag may be an RFID tag, and the sensor for registering the radio signal may be an RFID reader having at least two antennas. The device may be provided with a matrix of RFID antennas. The RFID tag may be an active RFID tag or a passive RFID tag. The wireless ID tag may be placed in the terminal, in the cover of the terminal, or in an item worn by a person, such as in a watch, jewelry, bracelet, ID card, or in a unit placed in a building or merchandise. ..</p><p num="0019"> The device may be built into the terminal. The device may include a display device for displaying an image registered by the image registration device. The terminal may be one of a mobile phone, a camera, a film / video camera, a PC or a tablet. Alternatively, the device may be configured to be attached to at least one of a mobile phone, a camera, a film / video camera, a PC or a tablet. The device may be designed as a cover.</p><p num="0020"> The image registration device may be a camera or a film / video camera. At least one object may be a building, item / article, person or animal. The information may be text, audio, video, name, address, email, or a reference to information stored on the Internet.</p><p num="0021"> In a further aspect, the invention is a method for identifying at least one object in an image, each of which is provided with a radio tag in which identification information related to that object is stored. Sending at least one question signal, and detecting a radio signal from a radio tag that responds to at least one question signal, which contains information for identifying an object, by at least two sensors. , A method comprising estimating at least one direction of each of the radio tags in response to at least one interrogation signal based on the phase difference of the detected radio signals.</p><p num="0022"> At least one interrogation signal may contain a radiating null. The method may further include detecting only the radio response signal within the time window after transmission of at least one interrogation signal. The interrogation signal may include a first radiation pattern with maximum intensity. The method may include scanning the angular range by processing the response signals detected by at least two sensors.</p><p num="0023"> The interrogation signal may further include a second radiation pattern that includes radiation nulls. The method may further include scanning an angular range within the field of view of the imaging device by using a radiation pattern that includes radiation nulls.</p><p num="0024"> The at least one interrogation signal and radio signal may be an electromagnetic signal or an acoustic signal. The electromagnetic signal may be radio waves, infrared light or visible light.</p><p num="0025"> Registering a radio signal from a radio ID tag may further include detecting visible or IR light with an image chip or main sensor in a digital image registration device. Registering a radio signal from a radio ID tag may further include detecting acoustic waves received by at least two sensors. Registration and processing may be performed in real time. The processing of the identification and the direction information may further include the position determination of the wireless ID tag.</p><p num="0026"> The method may further include registering the image with an image registration device. The method may further include processing the registered identification information from the wireless ID tag and the registration direction of the wireless ID tag together with the image data from the image registration device. The image may be displayed on the display device with information assigned to each of at least one object. The display may be performed in real time.</p><p num="0027"> The image may be stored in storage with information assigned to each of at least one object.</p><p num="0028"> In a further aspect, the invention provides a computer program stored on a computer-readable medium, comprising computer-readable program code for performing the method for identifying at least one object according to the method described above. To do.</p><p num="0029"> In a further aspect, the invention is a radio transponder device suitable for identifying at least one object in an image, wherein the radio transponder device is assigned to at least one object to identify at least one object and the radio transponder. Estimating at least one direction to the device is possible by registering at least one radio signal from the radio transponder device, where the at least one radio signal contains information for identifying at least one object and is radio. The registration of the signal and the estimation of at least one direction to the radio transponder device are performed by the method described above, and a storage device that stores information for identifying at least one object and a transmitter that transmits the radio signal are used. Provided is a transponder device in which a radio signal contains stored information for identification of at least one object.</p><p num="0030"> The transponder device may be an RFID tag.</p><p num="0031"> In a further aspect, the invention is a system for identifying at least one object in an image, the at least one wireless transponder device described above and at least one object in an image registered with the image registration device. Provided is a system including the above-mentioned device for identification of. The system may be provided with a processing device suitable for processing the registered identification information from the wireless ID tag and the registration direction of the wireless ID tag together with the image data of the image registered by the image registration device.</p>
<p num="0032"> The present invention may be used to obtain information about an object in an image and where the object is located in the image. This information may be obtained automatically and in real time.</p><p num="0033"> RFID technology allows objects to be registered even if the reader does not have a free view of the wireless tag. RFID technology has traditionally been used only to detect the presence or absence of an object, not where the object is located considering the position of the reader.</p><p num="0034"> To estimate the direct direction of the signal received by the tag (arrival direction), as in Patent Document 2, it is necessary to place an external transmitter / receiver unit around the wireless tag in order to find the position of the tag. There is no. Also, the use of at least two sensors and the estimation of the phase difference between the radio signals received by these two sensors also have an external transmitter / receiver unit around it to locate the radio tag. No need to place. In the present invention, all transmitter / receiver equipment can be placed in the same unit placed on the camera itself. According to the present invention, a user of a camera / film camera or a terminal with a camera (the terminal can be, for example, a mobile phone, a camera, a film / video camera, a PC or a tablet) can take a digital photographic image in real time at any place. Then, information for individually identifying each object provided with the wireless ID tag in the image can be acquired directly and in real time in the image. The present invention makes interactive imaging personalization more accessible to the general public.</p><p num="0035"> Currently known solutions require that the information to be introduced in the image is known in advance. According to the present invention, it is the wireless ID tag itself that provides information in an image in an instant.</p><p num="0036"> According to the present invention, a mobile phone can be used to generate an interactive image in real time. The direction to the tagged object can be estimated from the portable device without the need for video. By using a plurality of antennas, the present invention provides a solution for locating the presence of a tagged object within the same area as the person viewed by the optical lens.</p><p num="0037"> The present invention provides wireless, real-time, tagged object identification and interactive image generation using a portable imaging device such as a smartphone or electronic tablet. The image pickup apparatus of the present invention may wirelessly acquire information from a tagged object, and then obtains the acquired information in the vicinity of or near the position of the actual tag in the captured image, or in a virtual tag. It is incorporated and displayed in the form of. The virtual tag may be a clickable hyperlink that provides information about the captured (tagged) object. In addition, by incorporating it in an image, when the image is shared on a social network or posted as a multimedia message from a smartphone, a virtual tag may be sent together with the image as an integrated part. This allows the recipient of the image to view and interactively access the information provided by all tagged objects in the image. Therefore, the present invention provides a technique that enables a smartphone to generate an interactive image in real time.</p><p num="0038"> Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings.</p>
0039<figref num="1">Indicates a sensor and a terminal with a wireless tag for reading.</figref><figref num="2">A wireless tag reader and a terminal with a camera are shown. (A) shows an example in which two sensors included in the reader are arranged horizontally, and (b) shows an example in which two sensors provided in the reader are arranged vertically.</figref><figref num="3">A wireless tag reader and a camera-equipped terminal are shown, and an example in which the reader is equipped with three sensors is shown.</figref><figref num="4">A wireless tag reader and a camera-equipped terminal are shown, and an example is shown in which the reader is provided with four sensors distributed as a 2 × 2 matrix.</figref><figref num="5">A wireless tag reader and a camera-equipped terminal are shown, and an example is shown in which the reader is equipped with eight sensors distributed as a 2 × 4 matrix.</figref><figref num="6">An embodiment in which the present invention is attached as a cover of a mobile terminal is shown.</figref><figref num="7">Illustrate the use of a terminal to identify a person.</figref><figref num="8">Illustrate the use of terminals to identify products.</figref><figref num="9">Illustrate another use of the terminal to identify the product.</figref><figref num="10">Illustrate the use of terminals to identify buildings.</figref><figref num="11">This is an example of using a terminal.</figref><figref num="12">The power half width (HPBW) according to the aperture length is illustrated.</figref><figref num="13">It is a flowchart of the procedure for identifying the RF tag in front of an image pickup apparatus.</figref><figref num="14">The maximum radiation pattern (Σ beam) in the broadside direction and the radiation null radiation pattern (Δ beam) in the broadside direction are illustrated.</figref><figref num="15">It is a flowchart of the procedure for finding a direction to a radio tag using a phase scan by a Δbeam.</figref><figref num="16">An embodiment in which the switch is released after a predetermined time frame is shown, (a) is a diagram in which the switch is first closed and the device is passed directly through the signal, and (b) is a diagram in which the switch is released after a predetermined time frame and the signal is released. Shows a diagram that prevents the device from passing through.</figref>
0040The present invention relates to the identification of a marked object (person or product) in an image. The present invention provides information about what or who is present in an image and where the marked object is located in the image. The location of the object can be specified. Images can be displayed on the display of the terminal, and information can be provided in real time. The image can be a digital image.
0041FIG. 1 shows a terminal 3 having an image registration device 1 and a sensor 2 for reading a wireless tag. The image registration device can be a camera, a film camera or a film / video camera. The terminal can be, for example, a mobile terminal, a handheld terminal, a fixed terminal, a mobile phone, a camera, a film / video camera, a PC or a tablet. The image can be a photograph or a film. The image registration device can be a digital device.
0042The object to be detected is marked with an identification tag (wireless tag or wireless ID tag). Such wireless ID tags are commonly referred to as "tags". The wireless ID tag ("tag") can wirelessly communicate with the sensor 2 of the terminal 3 capable of taking an image. The radio ID tag transmits a radio signal containing information for identifying an object to which the radio ID tag is attached. The radio signal is also used by the sensor for registration of at least one direction of the radio ID tag. When registering a location-specified object in the field of view of the image registration device 1 using a terminal, the sensor 2 reads a wireless signal from the wireless ID tag. The information contained in the radio signal from each radio ID tag is registered and further processed in the processing apparatus to combine the identification and direction information from the radio ID tag with the image data from the digital camera. The image is displayed on the display device of the terminal, along with information associated with each of the tagged objects in the image. The combination of the image data and the information from the registered radio signal and the display of the information obtained thereby and the information associated with the marked object may be performed in real time.
0043Radio ID tags are based on the transmission of electromagnetic signals, visible or infrared light (IR), or sound waves. Sensors can be arranged to receive these signals. Therefore, the sensor can be an RDIF reader, or can be configured with an image chip or main sensor in the camera / digital camera, or a microphone located at the terminal.
0044An embodiment having a radio ID tag that transmits continuously, or at intervals or intermittently, or a radio ID tag that requires a trigger of a question pulse can be envisioned.
0045In an example of an embodiment of the invention that registers an ID tag that transmits continuously or at intervals, when an image is taken using the camera of a mobile phone / camera / tablet, the sensor will come from different radio signals. Register from which direction the signal is coming. The information stored in the wireless ID tag is also registered by the sensor. Then, the information and the position from each wireless ID tag whose location is specified in the area of the camera lens are combined and presented simultaneously in the photographic image. As a result, each object in the image provided with the wireless ID tag is identified in the photographic image. This can be done automatically with the camera, by launching an application, or by transferring the image to a PC.
0046In a further embodiment of registering an ID tag arranged to be triggered by a question pulse, when registering an image using the camera of such a mobile phone / camera / tablet embodiment, the mobile terminal The transmitter may transmit a signal that can be registered by a location-specified wireless ID tag within an area corresponding to the area exposed to the optical lens of the camera. A wireless ID tag within the reach of the transmitter's transmission responds by returning the information stored in the wireless ID tag. This information is registered in the camera. The sensor registers in which direction the signals from different wireless ID tags are coming from. Information and location from each radio ID tag located within the area of the camera lens are combined and presented simultaneously in the photographic image. As a result, each object in the image provided with the wireless ID tag is identified in the photographic image. This can be done automatically with the camera, by launching an application, or by transferring the image to a PC.
0047An embodiment of the present invention having both a transmitter and a receiver in a terminal may be arranged to register all types of wireless ID tags. When taking an image using a mobile phone / camera / tablet, the transmitter and sensor may be activated to register a location-specific wireless ID tag within the area of the camera lens.
0048This allows the invention to register what or who is present in the image and where different wireless ID tags are located in the image. With respect to position, in this context it is possible to find the position of an identified object in the image, the location in this case is within the frame of the image rather than the geographical position (as in GPS coordinates). The relative position of the object. To find the location of the ID tag, you may use direct direction estimation for the signal received from the ID tag.
0049In the case of a person, the wireless ID tag may include, for example, the person's name and email address, and in the case of a product, the wireless ID tag may include product information and the internet address of the manufacturer of the product. As described above, the terminal can be, for example, a mobile terminal, a handheld terminal, a fixed terminal, a mobile phone, a digital camera, a digital film / video camera, a PC or a tablet. In the following description of exemplary embodiments of the invention, the terminals in the drawings are exemplified as those in the form of a mobile phone, but the following examples are not intended to limit the scope of the invention. Absent.
0050(Example-RFID technology) An example is 1. Radio Position Technology-Wireless ID Tag Radio Detection (RFID (Radio Frequency IDentification) Technology) and 2. Digital Photography Technology-Presentation of wireless ID tag location and information in digital images It has a combination of.
0051RFID technology is based on wireless electromagnetic transmission of data. A wireless ID tag that is detachably attached to an object is called an RFID tag (or simply a tag). On the other hand, a transmitter / receiver unit that reads RFID tag information is called an RFID reader. A camera-equipped mobile terminal used to detect an RFID tag in an image is provided with an RFID reader. Traditionally, RFID readers use only one antenna. This is because it is only desirable (necessary) to detect the presence or absence of RFID tags. However, if you want to know the direction of the signal from the RFID tag, you need to use at least two antennas to receive the signal. Then, the signal from the RFID tag is received by different antennas of the reader at different times, and the time difference is recalculated as the phase difference of the frequency. This also makes it possible to estimate the angle of the signal with respect to the antenna when the frequency of the signal is known. In this way, it is possible to detect not only whether or not the RFID tag exists, but also where the tag is located with respect to the RFID reader. The more antennas used in an RFID reader, the more accurate angular resolution can be achieved. Therefore, this concept can be realized by arranging an antenna matrix on a camera (or a camera-equipped mobile phone / tablet).
0052FIG. 2 shows an embodiment of a terminal 3 having a camera 1 and a wireless ID tag reader 2. Here, the reader comprises two sensors distributed either (a) horizontally or (b) vertically. The reader's two sensors provide an opportunity to determine the direction of the radio signal from the radio ID tag in the plane in which they are distributed. Thereby, the embodiment of FIG. 2 allows the estimation of the direction of the radio ID tag in the horizontal plane (a) or the vertical plane (b) (the angle with respect to the radio ID tag in the vertical plane is distributed horizontally by the sensor). If so, it will be zero. The angle with respect to the radio ID tag in the horizontal plane will be zero if the sensors are distributed vertically). The angle with respect to the radio ID tag not located in the predetermined plane is equal to the projection angle of the radio ID tag 3 lowered to either the horizontal plane or the vertical plane. Two or more sensors are placed in both the horizontal and vertical directions so that the direction with respect to the wireless ID tag can be estimated in both the horizontal and vertical directions.
0053FIG. 3 shows an embodiment of a terminal 3 having a camera 1 and a wireless ID tag reader 2. Here, the reader includes three sensors. These three sensors are arranged to provide an opportunity to determine the direction of the radio signal from the radio ID tag in both the horizontal and vertical directions.
0054FIG. 4 shows an embodiment of a terminal 3 having a camera 1 and a wireless ID tag reader. Here, the reader comprises four sensors distributed as a 2x2 matrix. These four sensors are arranged so as to enable the direction determination of the radio signal from the radio ID tag in both the horizontal direction and the vertical direction.
0055FIG. 5 shows an embodiment of a terminal 3 having a camera 1 and a wireless ID tag reader 2. Here, the reader comprises eight sensors distributed as a 2x4 matrix. These four sensors are arranged so as to enable the direction determination of the radio signal from the radio ID tag in both the horizontal direction and the vertical direction.
0056In FIG. 2, the terminal 3 is rotated so that the antenna is positioned in the desired orientation plane. In Figure 2 (a), the antenna is positioned in a horizontal plane. If you wish to use the antenna of FIG. 2 (a) to estimate the orientation in the vertical plane, rotate the terminal 3 of FIG. 2 (a) 90 degrees so that the antenna is positioned in the horizontal plane. In Figure 2 (b), the antenna is positioned in a vertical plane. If you wish to use the antenna of FIG. 2 (b) to estimate the orientation in the horizontal plane, rotate the terminal 3 of FIG. 2 (b) 90 degrees so that the antenna is positioned in the horizontal plane. In the embodiments of FIGS. 3 and 4, no matter how the terminal 3 is held, at least two antennas are arranged in either the horizontal plane or the vertical plane, so that the terminal does not need to be rotated 90 degrees. Can be realized. The only difference between FIG. 3 and FIG. 4 is how to arrange the antennas when they have 3 or 4 antennas.
0057How many sensors are placed on the terminal depends on the size of the terminal. The typical size of current cameras indicates that there will be space for a simple 5.8GHz resonant patch antenna. When using a 5.8GHz resonant patch antenna, up to eight antennas can be mounted on the back of current mobile phones. When such an antenna is placed on a current tablet, it is possible to place up to 54 antennas. A sensor for registering a wireless signal may also be arranged in the image registration sensor of the image registration device of the terminal. The camera may incorporate the sensor in its own CMOS chip that registers the image.
0058FIG. 6 shows an embodiment in which the present invention is in the form of a separate cover 5 including an antenna and additional possible electronic devices. This also makes it possible to make the present invention an external unit that can be an external device that can be easily attached to the camera-equipped mobile phone 3. The description of the different configurations of the antennad reader described above in the context of FIGS. 1-5 is also a possible embodiment for the antenna of the external unit.
0059(RFID tag) The RFID tag may be either passive or active. The difference between a passive tag and an active tag is that the active tag has an internal battery, whereas the passive RFID tag does not have an internal battery to transmit information. Passive tags draw energy from RFID readers and send information back to the reader by modulating its own radar cross section. Therefore, the range of passive tags is relatively narrow. However, the active tag does not rely on energy from the reader to send the division. In this case, the energy from the reader acts as a signal instructing the RFID tag to respond by activating from power saving mode and sending back information. The active tag uses its own internal battery to do this. This makes it possible to achieve a much wider range than when using passive tags. In addition, it is possible to turn RFID tags on and off. This means, for example, that it can be turned off (deactivated if it is not desired to be identified) to protect the privacy of the individual. RFID tags can be cell phone covers, watches, bracelets, and ID cards. In some cases, it may be desirable to use RFID tags as visible symbols, such as Lance Armstrong's yellow bracelet, which visualizes support for cancer eradication efforts.
0060(Determining the direction of wireless RFID tag) (Example 1) The directional directivity of an antenna is proportional to its effective area A. This means that the physical size of the antenna affects the width of the main lobe / beam. Therefore, the angular resolution can be increased by increasing the physical size of the antenna. However, doing this is sometimes impractical or undesirable. An example is an antenna used in a mobile terminal. These are typically limited in size due to the relatively small size of the terminal, and it is less desirable to increase the physical size of the antenna, even if it contributes to improved performance. However, small antennas require a wide main lobe, which accommodates reduced or low directional directivity. In such a case, if multiple signals are received within the main lobe of the antenna, it may be impossible to distinguish the direction of the signals. However, it is possible to achieve higher angular resolution by using group antennas (assembly of several single antennas) and to estimate the direction of the received signal by using signal processing based on linear algebra. .. This signal processing technique is referred to as a "subspace" because it is based on generating a correlation matrix of received signals and dividing this matrix into two subspaces called a signal room and a noise room. Estimating the direction of each single signal is done by finding the direction in the space where the signal room is orthogonal to the noise room. The direction in space where this is done corresponds to the estimated direction with respect to incidental signals. This signal processing technique involves estimation based on phase differences.
0061Directional estimation is performed in one embodiment by an algorithm called MUSIC. This algorithm is described in Non-Patent Document 2 and is incorporated herein by reference in its entirety.
0062The normalized angle spectrum for the MUSIC algorithm is defined as follows.<maths num="1"><img id="000002" he="30" wi="85" file="JP6411348B2_D0001.tif" img-format="tif" img-content="drawing" /></maths> In the formula, A is A = [a (θ)<sub>1</sub>), A (θ<sub>2</sub>), ..., a (θ)<sub>M</sub>)] Is a signal matrix defined as (.)<sup>H</sup>Is the Hermitian operator (transposed / complex conjugate) and a (θ)<sub>m</sub>) Is the direction θ<sub>m</sub>It is a control vector representing the responsiveness of the group antenna in. The control vector is a (θ<sub>m</sub>) = [e<sup>j (n-1) βm</sup>]<sup>T</sup>(However, it is defined as 1 n N), and in the formula, (.)<sup>T</sup>Is, and βm represents the electrical phase shift between two antennas located on the side of the group antenna among the group antennas. V<sub>n</sub>Is a matrix of noise eigenvectors, ideally should be orthogonal to vector A in the direction of the received signal. P (θ) is V<sub>n</sub>This consists of the product of A and the denominator A, so this is A and V<sub>n</sub>Means that P (θ) is maximized when and are orthogonal to each other. The received signal arrives from the direction in the space where it occurs.
0063(Example 2) (A. Detection of tags in front of the camera) In the prior art of RF tag detection described above, the purpose is generally to detect the RF tag no matter where it is. That is, the orientation of the RF tag with respect to the interrogator is not important. However, in the present invention, estimating the direction of tags in the vicinity is a basic part of the invention. The prior art solution described in Patent Document 3 introduces moving images so that individual RF tags can be tracked by an imaging device that uses a highly directional antenna beam at its maximum peak. However, due to the small size of current mobile phones, physical or electromagnetic theory has shown that it is not possible to generate highly directional radiation patterns using antennas of limited size. In fact, the physical (and electrical) size of the antenna aperture contributes to quantifying the height of directivity achievable from the antenna, so it is an important figure of merit in antenna theory. merit). Using a rectangular aperture length as an example, in FIG. 12, as the aperture length increases, the full width at half maximum (HPBW) of the main lobe from such an aperture simply decreases (ie, as the aperture length increases, it is oriented. (Higher sex) has been observed. To further illustrate this, the dimensions of the back opening of the iPhone® 5 are used as an example. The size of this opening is about 5 x 10 cm. Table 1 shows the electrical magnitude of this aperture with respect to the free space wavelengths of three commonly used RFID frequencies. As can be seen from Table 1, the openings have the highest electrical magnitude at the highest frequencies.
0064<tables num="1"><img id="000003" he="40" wi="134" file="JP6411348B2_D0001.tif" img-format="tif" img-content="drawing" /></tables>
0065According to the antenna theory, the power half width θ from the rectangular opening<sub>HPBW</sub>Is expressed as follows.<maths num="2"><img id="000004" he="17" wi="92" file="JP6411348B2_D0001.tif" img-format="tif" img-content="drawing" /></maths> In the equation, l is the aperture length and λ is the wavelength. The length of the rectangular opening is either its height or its width, but it is common to choose the length in the plane in which the electric field is oriented. From Eq. (1), the feasible HPBW can be derived from the size of the openings listed in Table 1. They are shown in Table 2.
0066<tables num="2"><img id="000005" he="38" wi="112" file="JP6411348B2_D0001.tif" img-format="tif" img-content="drawing" /></tables>
0067Note that some of the values are displayed as N / A (not applicable), which is always the case when the argument of the arcsin (.) component in equation (1) is greater than 1 (l <0.443λ). It happens. For such a value, the equation (1) cannot generate a valid number. Physically, the HPBW of these values is greater than 180 °, which means that the directivity of the radiation pattern of the main lobe is very limited, resulting in approximately equal radiation across the field of view. As can be seen from Table 2, the minimum HPBW is 26.5 ° obtained at 5.8GHz. This is the minimum value, but it does not represent very high directivity and is not high enough to be practical enough to determine the position of multiple closely placed objects very accurately. Is certain. The only way to improve this value is to increase the antenna aperture, but it requires a large imaging device (assuming a fixed operating frequency). This is physically impossible with today's small handheld electronic devices such as mobile phones and tablets.
0068This embodiment of the present invention provides a solution for tracking the direction of an RF tag by a technique using a minimum value radiation pattern generally referred to as null instead of the maximum value in Patent Document 3. At least two antennas are needed to establish the direction to the tag in the hemiplane (vertical or horizontal). A minimum of three antennas are required to establish orientation to the tag in both the vertical and horizontal planes. If more antennas can be used in the camera-like space than the minimum required, this can contribute to improving the accuracy of direction estimation. The antenna array can further improve the accuracy of direction estimation. In addition, by using the antenna array, it is possible to estimate the direction to the tag using a single image, which eliminates the need to use moving images as in Patent Document 3.
0069To identify all RF tags in front of the imager, a signal is given that allows them to respond. However, as described in relation to the prior art, highly directional radiation patterns that are strictly responsive only to the RF tag in front of the imaging device cannot be generated from a very limited size antenna. .. Therefore, RF tags in large areas respond. RF tags typically respond using an omnidirectional antenna. This is because the RF tag has no way of knowing where the imaging device is located. As a result, the response from the RF tag is transmitted (and reflected) in all directions before some of it is received by the imaging device. In short, the present invention transmits at least one interrogation signal and detects by at least two sensors a radio signal from a radio tag that responds to the at least one interrogation signal and includes object identification information. And provide a procedure that includes estimating at least one direction of each of the radio tags in response to at least one interrogation signal based on the phase difference in the registered radio signal.
0070In the present invention, the image pickup apparatus can distinguish between the signal response generated by the tag placed in front of the camera (from the viewpoint of the optical lens) and the response generated by the tag elsewhere. .. The present invention addresses this issue by introducing a procedure that utilizes the phase difference between two or more antennas. By utilizing the phase difference, both constructive and destructive interference can be created through a two-step procedure.
0071FIG. 13 is a flow chart of a two-step procedure for identifying the RF tag in front of the imaging device.
0072In the first part of this procedure (step 1 in FIG. 13), a question signal is transmitted. The interrogation signal can be supplied by exciting the two antennas in a phase where the combined radiation pattern produces the largest radiation pattern in the broadside direction. The broadside direction is in front of the antenna and therefore in front of the camera. Also, a single antenna may be used to create the highest intensity radiation pattern in the broadside direction. The solid line in FIG. 14 illustrates the radiation pattern with maximum intensity in the broadside direction. Such radiation patterns are shortly referred to as thumb beams or sigma beams. FIG. 14 illustrates the intensity according to the angle (degrees) of the signal in front of the antenna. Since the thumb beam (Σ beam) is radiated to almost the entire half-plane in front of the imaging device, all RF tags in the vicinity can respond to the interrogation signal. For example, a sensor in the form of a dedicated radio receiver detects all radio response signals from the RF tag. Radio response signals containing identification information about the object are stored for the second and final parts of this procedure. The identification information may include a tag identification number.
0073In the second part of this procedure (step 2 in FIG. 13) for identifying the RF tag in front of the imaging device, a radiation null question signal is transmitted in the broadside direction. The two antennas are excited out of phase (180 ° phase difference) to create this radiation pattern of radiation null in the Broadside direction. This is called the Δ beam. The dotted line in FIG. 14 illustrates the radiation pattern of radiation null in the broadside direction. The radio response signal from the RF tag that responded to the interrogation signal in this second part of the procedure is detected by the sensor. A radio response signal containing identification information about the object is stored.
0074The results obtained using two different beams (Σ beam and Δ beam) in these two steps of this procedure can be explained as follows. It can be assumed that the RF tags detected in both antenna patterns (Σ and Δ) are not immediately in front of the imaging device. However, the RF tag that was present in the Sam Σ beam but disappeared in the Δ beam should have been captured by the null of the Δ beam created just before the imaging device. Therefore, the RF tag that was present in the Sam Σ beam but disappeared in the Δ beam is an RF tag that is incorporated and retained in the image. This is because these RF tags are most likely to be found just before the imaging device (compared to all other tags).
0075(B.RF tag direction) In Patent Document 3, the direction to the RF tag is associated with the direction of the maximum electric field strength, and the maximum electric field strength is established by tracking the received signal via the moving image. Patent Document 3 assumes that the receiver has an antenna having sufficient directivity to ensure that the received signal can be measured relatively stronger in one direction than in the other direction. In fact, it is assumed to be directional enough to use this concept to identify several RF tags through multiple peaks in the received field strength. However, as already discussed, it is not possible to generate a highly directional beam from an antenna with a limited aperture size. Therefore, separating individual RF tags using only maximum field strength is not the optimal approach.
0076The above-mentioned problems of the prior art are solved by this exemplary embodiment of the invention by using the exact opposite, the radiation null. It is a well-known fact from antenna theory that the radiated null is much sharper than the maximum intensity beam. Therefore, by introducing a null-based concept, the orientation of any target can be established much more accurately. Sharp nulls can be generated by exciting two antennas with different phases as described above. As in the previous section, this may be referred to as the Δ beam. When the phase difference is 180 °, sharp nulls occur in the broadside direction. The direction of the null can be changed by changing the phase difference between the antennas (from 180 ° to different). Then, this can be used as a scanning procedure for detecting whether or not a tag exists in the current null direction. If the tag detected by the thumb sigma beam disappears in the delta beam, the direction to the tag should be the null direction created by the delta beam. Therefore, directions to individual tags can be identified by using the procedure of scanning a limited angular range in front of the imaging device with a single thumb Σ beam and scanning delta beam.
0077FIG. 15 shows a flowchart of an embodiment of a scanning procedure for finding a direction to a radio tag using phase scanning with a Δbeam. As already described for FIG. 13, step 1 identifies the wireless ID tag in front of the camera. The scanning procedure is performed in step 2. A question signal is transmitted by the Δ beam. The radio response signal from the radio ID tag that responds to the radio interrogation signal is detected and stored along with the identification information about the object. The identification information may include a tag identification number. If the tag detected by the thumb sigma beam disappears in the delta beam, the direction to the tag should be the null direction created by the delta beam. Then, the phase of the antenna is adjusted to install the Δbeam in different directions. Then, after transmitting a further question signal in these different directions, the radio response signal is detected as described above. Then, all tags in this different direction detected by the thumb Σ beam and disappeared by the Δ beam are identified as being in this different direction. Step 2 is repeated until the desired angular range in front of the camera is scanned.
0078The second part of this procedure may also be performed by processing the detection signal from the tag obtained in the first part of this procedure in FIG. This process can be done digitally. Therefore, this alternative method does not require the physical transmission of a second interrogation signal. Therefore, the scanning procedure for giving a radiation null may be performed digitally with respect to the signal data from step 1.
0079Therefore, the above procedure can estimate the direction to the wireless tag using a single image and does not require the moving motion required in the prior art.
0080(C. Reduction of interference from multipath) Multipath propagation is a general term used to describe the fact that radio signals are typically transmitted from transmitter to receiver by different paths. As a result, even if the signal originates from a single point in space, the signal can be received by the receiver from multiple directions.
0081Patent Document 3 claims that responses from a plurality of RF tags can be separated by using moving images. The video concept is needed to get the input signal tracking process, i.e. to identify the direction of maximum signal strength. However, Patent Document 3 does not introduce any technique for dealing with the problem of how to separate the correct direction from the wrong direction caused by multipath propagation. That is, it is essentially assumed that the signals from the individual RF tags are received only from the direction of maximum signal strength, and that the direction of maximum signal strength always represents the correct direction to the object.
0082Radio response from an RF tag that arrives within a given time window after the first request is transmitted in a sigma beam (question signal) to reduce the number of misdirection input signals caused by multipath propagation. Procedures may be provided that allow only signals. Thus, if the radio response signal reaches the sensor and is registered too late, it is interpreted as a multipath signal arriving from the wrong direction, regardless of its electric field strength. An example of carrying out such a procedure is illustrated in FIG. In FIG. 16, the switch provided in association with the sensor that detects the radio response signal is released after a predetermined time frame from the start of transmission of the question signal. First, the switch is closed to allow the detected response signal to pass directly through the device (a). When the switch is released after a predetermined time frame from the start of transmission of the interrogation signal, the detected response signal cannot pass through the device (b). Establishment of blocking of signals detected after a predetermined time window may be performed using a filter.
0083The terminals shown in FIGS. 1 (a) and 1 (b) and FIGS. 3-6 provide examples of sensor configurations that can be used to perform the phase difference based tracking described above. In the embodiment, two sensors and a separate transmitter that transmits a question signal are provided. However, when using a sensor in the form of an antenna, the antenna may also be configured to function as a transmitter. Therefore, when two antennas are provided, these two antennas may be configured to operate as both a sensor and a transmitter. The antenna may be controlled by a control unit. An estimation unit is provided that estimates the direction of each of the radio tags based on the phase difference of the radio response signals detected by at least two sensors.
0084In the prior art, RF tags are commonly used as tags for radio identification of objects. The present invention is not limited to this type of tag, but for simplification, RF tags have been used to illustrate exemplary embodiments of the invention.
0085The above example is also applicable to electromagnetic signals in the form of infrared and visible light. Acoustic signals may be used, in which case the transmitter, sensor and tag will be acoustic devices.
0086(Example-use) (Detection of the wireless ID tag associated with a person or product and its relative position in the image) FIG. 7 illustrates the identification of a person. Each person wears a wireless ID tag 4 on his arm. A person is displayed on the display device of terminal 3, and the person's name and email address automatically appear in association with each person as part of the image.
0087Figures 8, 9 and 10 illustrate the identification of products and buildings. The car is equipped with a wireless ID tag. The imaged vehicle is displayed on the display device of the terminal, and the vehicle manufacturer and the Internet address of the vehicle manufacturer automatically appear in relation to each vehicle. Similarly, the bicycle is equipped with a wireless ID tag, which is displayed along with the type and internet address. As shown in the museum in Fig. 10, wireless ID tags can also be installed in buildings. It is possible to automatically connect to the internet by clicking on an internet address and view more information about the associated marked object.
0088FIG. 11 illustrates the use of the present invention. The image was taken at Maihaugen in Lillehammer. The small rectangular field in the image represents the information received from the wireless ID tag that was active at the time the image was taken. The image provides information about what is in the image (the building associated with the internet address), who is in the image (the person associated with the name and email address), and where in the image the registration information is There is a pointer from the information to the point where the different wireless ID tags are located, not just related. The larger field below the smaller rectangular field is an example of supplementary information about the two buildings in the image. Internet links shown within a rectangular field can be clickable. Clicking on the link will bring up relevant information, as illustrated in the large field. In this way, more information can be attached to the image apart from simply pure visual information, thus providing added value to the image. As a result, the image in Figure 11 becomes a new marketing channel for Maihaugen. If the device that took the image is a mobile device with GPS, it will be possible to attach a geographical location to the image so that the destination Maihaugen can be easily found on the map. In the example of FIG. 11, two people are identified in the image along with their name and email address. Both of these people are sending images from the email address shown in the image. If you are interested, you can follow the links to find out more about this product.
0089(Installation of products related to sporting events) The organizer can register who is tagged, what is tagged with the product, and so on. After an event (soccer match / handball match / ice hockey match, etc.), the organizer / sponsor will send information about those gifts, including products and other information that they consider suitable for sharing. Can be provided. This may be a discount on a natural event or an offer of a new product of the same brand that you are already using.
0090(A. Example 1-Personal tag) Figure 7 shows an example of a personal tag. This figure shows a situation where a user is taking an image of two other people. It is assumed that both photographed persons have a portable device that also functions as a tag. In addition, both portable devices shall have access to installed applications that allow them to choose whether or not to allow the portable device to function as a tag. In this embodiment, it is assumed that both persons have selected to switch the tag to the on position. This means that when the user takes a picture, both tags respond with information. In this type of use, each person may individually choose what kind of information he or she wants to present. For example, it is natural to reveal the person's name, which is an internet site (Facebook®, and in some cases, a registered trademark) where the user can provide more information by clicking on the hyperlink. You can be directed to an advertisement for a cultural event that the person is paying attention to. The user can then share this image with a friend of that person on a social network. The recipient will also be able to obtain the same information as the original user who captured the image. Note that if the two people are sufficiently separated, the system can distinguish between the positions of the two tags so that each tag is placed on the correct person. Therefore, if the user does not know who, the image works for that person. Then, the virtual tag of each person is placed on the image according to the position of the actual physical tag. If one of the people intentionally switches the tag off, no information about this person will be available. The person is then presented in the image in the traditional form, i.e., in the absence of available names or information.
0091(B. Example 2-Product Tag) In this embodiment, all imageable items and products around us in our daily lives are tagged objects (furniture, clothing, cars, bicycles, buildings, etc.). Figure 8, 9 and 10 are examples of such product tags. One example is a situation where a user is taking a picture and the person being photographed happens to be wearing a nice (tagged) jacket. The jacket responds with information and makes the location and information of the jacket accessible via virtual tags within an interactive image. The present invention allows the user to access information about a particular jacket as soon as an interactive image is generated. For example, the user is directed to an online shop where the jacket can be purchased. Thus, the present invention can be used as a personal tool for obtaining information about items in the vicinity of a person. However, the present invention inherently has properties that go far beyond mere personal tools for individual individual users. This property means that all recipients of the image will also have access to the same information about the jacket. And the product tag and interactive image concept will be a powerful tool for the marketing industry as it will allow the interactive image of an individual to be used to market the product on social networks. And what makes this concept so powerful is the way images are circulated on social networks. To understand this, it is important to recognize that people typically select recipients of images (or messages) based on their content. This is because the poster (you) really knows why what you want to show (or convey) through the image is relevant to the people you have chosen as the recipient. Therefore, messages posted on social networks are often targeted at people selected for a particular reason. For example, the reason is that they have the same tastes such as humor, taste, music, and clothes. So if you decide to buy the jacket mentioned above online, you spread the image to some of your friends You can then let them know that this item is available for purchase and that you have purchased it. And since you know your friends and their tastes, you probably know in advance who of your friends you should send the image to. As a result, images of that particular jacket begin to circulate on social networks among friends (not just friends, but friends with similar tastes). Therefore, from a marketing point of view, the jacket is advertised exactly free of charge to the right target group through its own sustainable process. All the manufacturer has to do is put an information tag on their product so that the product has the potential to market itself each time the item is imaged. This special ability makes the invention a candidate for a new and powerful worldwide marketing channel, freeing tagged products present in personal images on social networks (or generally the Internet). Can be circulated and marketed at.
0092(C. Example 3-Assistance for people with poor eyesight) Another use of the present invention is an aid for people with poor eyesight. Instead of generating an image, the system can be designed to generate a voice message. Similar functionality already exists in the prior art and is being developed in popular tourist destinations and museums using dedicated audio terminals. Typically, they provide information of interest that can be found in a confined space. In the present invention, this type of system is extended to an open space, and everyone who needs it uses their own mobile terminal (smartphone, electronic tablet, digital camera) instead of a custom-designed dedicated terminal. And you will be able to access the information. Voice messages can be programmed into tags and tags can have hyperlinks to the Internet. This particular embodiment allows people with poor eyesight to access information about their immediate surroundings via voice messages.
0093(D. Example 4-Image database) Social networks can be seen as a market where users' thoughts and feelings about what they are interested in in life circulate for free. When young people attend rock concerts and festivals, they often post their presence and excitement on social networks. When they meet a celebrity or best friend, it is common to take a picture of the event and upload it to a social network as evidence of the event. In places where many people gather, it is common for people to take pictures, which can lead to unintentional reflection of other people in other people's images (although negative here). Not considered as a thing). People have uploaded their images to the image database and other people who may have attended the same event also have them in other people's images (also in a purely positive form). You can see it. The present invention allows people to look for their pictures, regardless of who took the pictures. Through this example, people may also capture images that may be of value to others (ie, strangers).
0094The present invention is not limited to the above-mentioned exemplary examples, and it is clear that there are other examples as well. The present invention is defined in the appended claims.
21 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21
Every citation, both ways
| Document | Relation | Office |
|---|---|---|
| JP2005197797A | Cites | Japan |
| JP2010176176A | Cites | Japan |
| US20110169613A1 | Cites | United States of America |
| JP2006270456A | Cites | Japan |
| JP2010119125A | Cites | Japan |
| JP2006031419A | Cites | Japan |
| JP2011018366A | Cites | Japan |
| JP2006040035A | Cites | Japan |
| US20120068813A1 | Cites | United States of America |
| JP2006279568A | Cites | Japan |
| JP2005196236A | Cites | Japan |
23 members in 12 offices
Priority claims3
| Document | Office | Kind | Date |
|---|---|---|---|
| 20120982 | Norway | – | |
| 20120982 | Norway | A | |
| 2013050143 | Norway | W |
Members23
| Document | Office | Kind | |
|---|---|---|---|
| NO20120982A1 | Norway | A1 | |
| WO2014035257A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2014035257A9 | World Intellectual Property Organization (WIPO) | A9 | |
| EP2890953A1 | European Patent Office (EPO) | A1 | |
| KR20150079563A | Republic of Korea | A | |
| CN104838230A | China | A | |
| US2015235064A1 | United States of America | A1 | |
| NO336454B1 | Norway | B1 | |
| JP2015529921A | Japan | A | |
| EP2890953A4 | European Patent Office (EPO) | A4 | |
| HK1213317A | Hong Kong, China | A | |
| HK1213317A1 | Hong Kong, China | A1 | |
| RU2015111574A | Russian Federation | A | |
| US9626539B2 | United States of America | B2 | |
| CN104838230B | China | B | |
| RU2656576C2 | Russian Federation | C2 | |
| JP6411348B2This record | Japan | B2 | |
| KR102144141B1 | Republic of Korea | B1 | |
| EP2890953B1 | European Patent Office (EPO) | B1 | |
| LT2890953T | Lithuania | T | |
| EP3879473A1 | European Patent Office (EPO) | A1 | |
| ES2873518T3 | Spain | T3 | |
| PL2890953T3 | Poland | T3 |
17 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Written request for extension of timeJAPANESE INTERMEDIATE CODE: A601A601 | A601 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Written request for extension of timeJAPANESE INTERMEDIATE CODE: A601A601 | A601 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Report on retrievalJAPANESE INTERMEDIATE CODE: A971007A977 | A977 | |
| Written request for application examinationJAPANESE INTERMEDIATE CODE: A621A621 | A621 | |
| Written submission of copy of amendment under article 34 pctJAPANESE INTERMEDIATE CODE: A529A529 | A529 |
Numbers
- Publication
- 6411348
- Application
- 2015529718
Titles2
- Japanese
- 画像内における物体の識別のための装置、システム及び方法、並びにトランスポンダ
- English
- Devices, systems and methods for identifying objects in images, and transponders
Classification
- CPC, 9
- G01S3/48
- G01C11/04
- G06K7/10366
- G06Q10/0877
- G01S1/00
- G06T3/00
- G06T3/053
- G06Q10/087
- G06K7/10316
- IPC, 1
- G06K7 10
