Transmitter, program, control method, and information communication device
3 claims: 3 independent, 0 dependent
- 1複数の露光ラインを有するイメージセンサを用い て送 信機から情報を取得する情報通信装置を制御するプログラムであって、 前記送信機は、 照明板と、 前記照明板の背面側から光を照射する光源と、 前記光源の輝度を変化させるマイクロコントローラと、を備え、 前記マイクロコントローラは、前記光源を輝度変化させることにより、前記光源から前記照明板を介して第1の識別情報を送信し、 前記照明板の前面側にバーコードが配置されており、前記バーコードに第2の識別情報が符号化されており、 前記プログラムは、前記情報通信装置のコンピュータに対し、 通常撮影モードが選択された場合 、前 記送信機を撮影させることにより通常画像データを取得させ、前記通常画像データの前記バーコードを復調させることによって前記第 2 の識別情報を取得させ、 取得した前記第2の識別情報をサーバに送信し、前記サーバから前記第2の識別情報に対応する情報を取得し、 可視光通信モードが選択された場合 、前 記送信機を撮影させることにより画像データを取得させ、前記画像データに現れる、前記 輝度変化 に対応す るパ ターンによって特定されるデータを復調させることにより前記第 1 の識別情報を取得させ 、取得した前記第1の識別情報をサーバに送信し、前記サーバから前記第1の識別情報に対応する情報を取得し、 前記第2の識別情報に対応する情報と、前記第1の識別情報に対応する情報とは同じ情報であり、前記第2の識別情報が送信されるサーバと、前記第1の識別情報が送信されるサーバとが同じである、 プログラム。
- 2複数の露光ラインを有するイメージセンサを用い て送 信機から情報を取得する情報通信装置を制御する制御方法であって、 前記送信機は、 照明板と、 前記照明板の背面側から光を照射する光源と、 前記光源の輝度を変化させるマイクロコントローラと、を備え、 前記マイクロコントローラは、前記光源を輝度変化させることにより、前記光源から前記照明板を介して第1の識別情報を送信し、 前記照明板の前面側にバーコードが配置されており、前記バーコードに第2の識別情報が符号化されており、 前記制御方法は、前記情報通信装置のコンピュータに対し、 通常撮影モードが選択された場合 、前 記送信機を撮影させることにより通常画像データを取得させ、前記通常画像データの前記バーコードを復調させることによって前記第 2 の識別情報を取得させ、 取得した前記第2の識別情報をサーバに送信し、前記サーバから前記第2の識別情報に対応する情報を取得し、 可視光通信モードが選択された場合 、前 記送信機を撮影させることにより画像データを取得させ、前記画像データに現れる、前記 輝度変化 に対応す るパ ターンによって特定されるデータを復調させることにより前記第 1 の識別情報を取得させ 、取得した前記第1の識別情報をサーバに送信し、前記サーバから前記第1の識別情報に対応する情報を取得し、 前記第2の識別情報に対応する情報と、前記第1の識別情報に対応する情報とは同じ情報であり、前記第2の識別情報が送信されるサーバと、前記第1の識別情報が送信されるサーバとが同じである、 制御方法。
- 3複数の露光ラインを有するイメージセンサを用い て送 信機から情報を取得する情報通信装置であって、 前記送信機は、 照明板と、 前記照明板の背面側から光を照射する光源と、 前記光源の輝度を変化させるマイクロコントローラと、を備え、 前記マイクロコントローラは、前記光源を輝度変化させることにより、前記光源から前記照明板を介して第1の識別情報を送信し、 前記照明板の前面側にバーコードが配置されており、前記バーコードに第2の識別情報が符号化されており、 前記情報通信装置は、 通常撮影モードが選択された場合 、前 記送信機を撮影することにより通常画像データを取得し、前記通常画像データの前記バーコードを復調することによって前記第 2 の識別情報を取得 し、取得した前記第2の識別情報をサーバへ送信し、前記サーバから前記第2の識別情報に対応する情報を取得 する第1取得部と、 可視光通信モードが選択された場合 、前 記送信機を撮影することにより画像データを取得し、前記画像データに現れる、前記 輝度変化 に対応す るパ ターンによって特定されるデータを復調することにより前記第 1 の識別情報を取得 し、取得した前記第1の識別情報をサーバへ送信し、前記サーバから前記第1の識別情報に対応する情報を取得 する第2取得部と、を備え 、 前記第2の識別情報に対応する情報と、前記第1の識別情報に対応する情報とは同じ情報であり、前記第2の識別情報が送信されるサーバと、前記第1の識別情報が送信されるサーバとが同じである、 情報通信装置。
Independent claims3
1,801 paragraphs, as filed
The present invention relates to a communication method between a mobile terminal such as a smartphone, a tablet or a mobile phone and a home electric appliance such as an air conditioner, a lighting device or a rice cooker.
In recent home networks, in addition to linking AV home appliances by IP (Internet Protocol) connection via Ethernet (registered trademark) or wireless LAN (Local Area Network), management of power consumption in response to environmental problems and from outside the home With the Home Energy Management System (HEMS), which has functions such as power ON / OFF, the introduction of a home appliance cooperation function that connects various home appliances to a network is progressing. However, there are some home appliances that do not have sufficient computing power to have a communication function, and some home appliances that are difficult to install a communication function in terms of cost.
In order to solve such a problem, in Patent Document 1, in an optical space transmission device that transmits information to free space using light, limited transmission is performed by performing communication using a plurality of monochromatic light sources of illumination light. Among the devices, the technology for efficiently realizing communication between devices is described.
<p><patcit num="1"><text>Japanese Patent Application Laid-Open No. 2002-290335</text></patcit></p>
<p> However, the conventional method is limited to the case where the applied device has a three-color light source such as lighting. The present invention provides an information communication method that solves such a problem and enables communication between various devices including devices having low computing power.</p>
<p><u style="single"> The program according to one embodiment of the present invention is a program for controlling an information communication device that acquires information from a transmitter using an image sensor having a plurality of exposure lines , wherein the transmitter includes a lighting plate and the lighting. A light source that irradiates light from the back surface side of the plate and a microcomputer that changes the brightness of the light source are provided. The identification information of 1 is transmitted, a bar code is arranged on the front side of the lighting plate, the second identification information is encoded in the bar code, and the program is transmitted to the computer of the information communication device. On the other hand, when the normal shooting mode is selected, the normal image data is acquired by shooting the transmitter, and the second identification information is acquired and acquired by demolishing the bar code of the normal image data. The second identification information is transmitted to the server, the information corresponding to the second identification information is acquired from the server, and when the visible light communication mode is selected, the transmitter is photographed to take an image data. Is acquired, and the first identification information is acquired by demodulating the data specified by the pattern corresponding to the change in brightness that appears in the image data, and the acquired first identification information is transmitted to the server. , The information corresponding to the first identification information is acquired from the server, the information corresponding to the second identification information and the information corresponding to the first identification information are the same information, and the second The server to which the identification information of the above is transmitted is the same as the server to which the first identification information is transmitted.</u> Further, the transmitter according to one embodiment of the present invention includes a lighting plate, a light source that irradiates light from the back surface side of the lighting plate, and a microcontroller that changes the brightness of the light source. By changing the brightness of the light source, the first identification information is transmitted from the light source via the lighting plate, a bar code is arranged on the front side of the lighting board, and the second identification is attached to the bar code. The information is encoded, and the first identification information and the second identification information are the same information.</p><p> It should be noted that these comprehensive or specific embodiments may be realized in recording media such as systems, methods, integrated circuits, computer programs or computer-readable CD-ROMs, systems, methods, integrated circuits, computer programs. And may be realized by any combination of recording media.</p>
<p> According to the present invention, it is possible to realize an information communication method that enables communication between various devices including devices having low computing power.</p>
<figref num="1">It is a figure which shows an example of the observation method of the luminance of the light emitting part in Embodiment 1. FIG.</figref><figref num="2">It is a figure which shows an example of the observation method of the luminance of the light emitting part in Embodiment 1. FIG.</figref><figref num="3">It is a figure which shows an example of the observation method of the luminance of the light emitting part in Embodiment 1. FIG.</figref><figref num="4A">It is a figure which shows an example of the observation method of the luminance of the light emitting part in Embodiment 1. FIG.</figref><figref num="4B">It is a figure which shows an example of the observation method of the luminance of the light emitting part in Embodiment 1. FIG.</figref><figref num="4C">It is a figure which shows an example of the observation method of the luminance of the light emitting part in Embodiment 1. FIG.</figref><figref num="4D">It is a figure which shows an example of the observation method of the luminance of the light emitting part in Embodiment 1. FIG.</figref><figref num="4E">It is a figure which shows an example of the observation method of the luminance of the light emitting part in Embodiment 1. FIG.</figref><figref num="4F">It is a figure which shows an example of the observation method of the luminance of the light emitting part in Embodiment 1. FIG.</figref><figref num="4G">It is a figure which shows an example of the observation method of the luminance of the light emitting part in Embodiment 1. FIG.</figref><figref num="4H">It is a figure which shows an example of the observation method of the luminance of the light emitting part in Embodiment 1. FIG.</figref><figref num="4I">It is a figure which shows an example of the observation method of the luminance of the light emitting part in Embodiment 1. FIG.</figref><figref num="5">It is a figure which shows an example of the observation method of the luminance of the light emitting part in Embodiment 1. FIG.</figref><figref num="6">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="7">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="8">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="9">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="10">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="11">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="12">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="13">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="14">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="15">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="16">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="17">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="18">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="19">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="20">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="21">It is a figure which shows an example of the signal modulation method in Embodiment 1. FIG.</figref><figref num="22">It is a figure which shows an example of the detection method of the light emitting part in Embodiment 1. FIG.</figref><figref num="23">It is a figure which shows an example of the detection method of the light emitting part in Embodiment 1. FIG.</figref><figref num="24">It is a figure which shows an example of the detection method of the light emitting part in Embodiment 1. FIG.</figref><figref num="25">It is a figure which shows an example of the detection method of the light emitting part in Embodiment 1. FIG.</figref><figref num="26">It is a figure which shows an example of the detection method of the light emitting part in Embodiment 1. FIG.</figref><figref num="27">It is a figure which shows the timeline of the transmission signal in Embodiment 1 and the image which image | imaged the light emitting part.</figref><figref num="28">It is a figure which shows an example of the transmission of the signal by the position pattern in Embodiment 1. FIG.</figref><figref num="29">It is a figure which shows an example of the receiving apparatus in Embodiment 1. FIG.</figref><figref num="30">It is a figure which shows an example of the transmission device in Embodiment 1. FIG.</figref><figref num="31">It is a figure which shows an example of the transmission device in Embodiment 1. FIG.</figref><figref num="32">It is a figure which shows an example of the transmission device in Embodiment 1. FIG.</figref><figref num="33">It is a figure which shows an example of the transmission device in Embodiment 1. FIG.</figref><figref num="34">It is a figure which shows an example of the transmission device in Embodiment 1. FIG.</figref><figref num="35">It is a figure which shows an example of the transmission device in Embodiment 1. FIG.</figref><figref num="36">It is a figure which shows an example of the transmission device in Embodiment 1. FIG.</figref><figref num="37">It is a figure which shows an example of the transmission device in Embodiment 1. FIG.</figref><figref num="38">It is a figure which shows an example of the structure of the light emitting part in Embodiment 1. FIG.</figref><figref num="39">It is a figure which shows an example of the signal carrier wave in Embodiment 1. FIG.</figref><figref num="40">It is a figure which shows an example of the image pickup part in Embodiment 1. FIG.</figref><figref num="41">It is a figure which shows an example of the estimation of the position of the receiving device in Embodiment 1. FIG.</figref><figref num="42">It is a figure which shows an example of the estimation of the position of the receiving device in Embodiment 1. FIG.</figref><figref num="43">It is a figure which shows an example of the estimation of the position of the receiving device in Embodiment 1. FIG.</figref><figref num="44">It is a figure which shows an example of the estimation of the position of the receiving device in Embodiment 1. FIG.</figref><figref num="45">It is a figure which shows an example of the estimation of the position of the receiving device in Embodiment 1. FIG.</figref><figref num="46">It is a figure which shows an example of setting of transmission information in Embodiment 1. FIG.</figref><figref num="47">It is a figure which shows an example of setting of transmission information in Embodiment 1. FIG.</figref><figref num="48">It is a figure which shows an example of setting of transmission information in Embodiment 1. FIG.</figref><figref num="49">It is a block diagram which shows an example of the component of the receiving apparatus in Embodiment 1. FIG.</figref><figref num="50">It is a block diagram which shows an example of the component of the transmission device in Embodiment 1. FIG.</figref><figref num="51">It is a figure which shows an example of the receiving procedure in Embodiment 1. FIG.</figref><figref num="52">It is a figure which shows an example of the procedure of self-position estimation in Embodiment 1. FIG.</figref><figref num="53">It is a figure which shows an example of the transmission control procedure in Embodiment 1. FIG.</figref><figref num="54">It is a figure which shows an example of the transmission control procedure in Embodiment 1. FIG.</figref><figref num="55">It is a figure which shows an example of the transmission control procedure in Embodiment 1. FIG.</figref><figref num="56">It is a figure which shows an example of information provision in a station premises in Embodiment 1. FIG.</figref><figref num="57">It is a figure which shows an example of the boarding service in Embodiment 1. FIG.</figref><figref num="58">It is a figure which shows an example of the in-store service in Embodiment 1. FIG.</figref><figref num="59">It is a figure which shows an example of establishment of a wireless connection in Embodiment 1. FIG.</figref><figref num="60">It is a figure which shows an example of the adjustment of the communication range in Embodiment 1. FIG.</figref><figref num="61">It is a figure which shows an example of the use indoors in Embodiment 1. FIG.</figref><figref num="62">It is a figure which shows an example of the outdoor use in Embodiment 1. FIG.</figref><figref num="63">It is a figure which shows an example of the instruction of the direction in Embodiment 1. FIG.</figref><figref num="64">It is a figure which shows an example of the use of a plurality of image pickup apparatus in Embodiment 1. FIG.</figref><figref num="65">It is a figure which shows an example of the transmitter autonomous control in Embodiment 1. FIG.</figref><figref num="66">It is a figure which shows an example of setting of transmission information in Embodiment 1. FIG.</figref><figref num="67">It is a figure which shows an example of setting of transmission information in Embodiment 1. FIG.</figref><figref num="68">It is a figure which shows an example of setting of transmission information in Embodiment 1. FIG.</figref><figref num="69">It is a figure which shows an example of the combination with the 2D bar code in Embodiment 1. FIG.</figref><figref num="70">It is a figure which shows an example of the creation and use of the map in Embodiment 1.</figref><figref num="71">It is a figure which shows an example of the state acquisition and operation of the electronic device in Embodiment 1. FIG.</figref><figref num="72">It is a figure which shows an example of recognition of the electronic device in Embodiment 1. FIG.</figref><figref num="73">It is a figure which shows an example of the display of the augmented reality object in Embodiment 1. FIG.</figref><figref num="74">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="75">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="76">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="77">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="78">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="79">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="80">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="81">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="82">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="83">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="84">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="85">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="86">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="87">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="88">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="89">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="90">It is a figure which shows an example of the user interface in Embodiment 1. FIG.</figref><figref num="91">It is a figure which shows the application example to ITS in Embodiment 2.</figref><figref num="92">It is a figure which shows the application example to ITS in Embodiment 2.</figref><figref num="93">It is a figure which shows the application example to the position information notification system and the facility system in Embodiment 2. FIG.</figref><figref num="94">It is a figure which shows the application example to the supermarket system in Embodiment 2.</figref><figref num="95">It is a figure which shows the application example to the communication of a mobile phone terminal and a camera in Embodiment 2. FIG.</figref><figref num="96">It is a figure which shows the application example to the underwater communication in Embodiment 2.</figref><figref num="97">It is a figure for demonstrating the example of service provision to a user in Embodiment 3. FIG.</figref><figref num="98">It is a figure for demonstrating the example of service provision to a user in Embodiment 3. FIG.</figref><figref num="99">It is a flowchart which shows the case which the receiver in Embodiment 3 processes a plurality of signals received from a transmitter at the same time.</figref><figref num="100">It is a figure which shows an example of the case which realizes communication between devices by mutual communication in Embodiment 3. FIG.</figref><figref num="101">It is a figure for demonstrating the service using the directivity characteristic in Embodiment 3. FIG.</figref><figref num="102">It is a figure for demonstrating another example of the service provision example to a user in Embodiment 3. FIG.</figref><figref num="103">It is a figure which shows the format example of the signal included in the light source emitted by the transmitter in Embodiment 3. FIG.</figref><figref num="104">It is a figure which shows the principle of Embodiment 4.</figref><figref num="105">It is a figure which shows an example of the operation of Embodiment 4.</figref><figref num="106">It is a figure which shows an example of the operation of Embodiment 4.</figref><figref num="107">It is a figure which shows an example of the operation of Embodiment 4.</figref><figref num="108">It is a figure which shows an example of the operation of Embodiment 4.</figref><figref num="109A">It is a figure which shows an example of the operation of Embodiment 4.</figref><figref num="109B">It is a figure which shows an example of the operation of Embodiment 4.</figref><figref num="109C">It is a figure which shows an example of the operation of Embodiment 4.</figref><figref num="110">It is a figure which shows an example of the operation of Embodiment 4.</figref><figref num="111">It is a figure which shows an example of the operation of Embodiment 4.</figref><figref num="112">It is a figure which shows an example of the operation of Embodiment 4.</figref><figref num="113">It is a figure which shows an example of the operation of Embodiment 4.</figref><figref num="114">It is a figure which shows an example of the operation of Embodiment 4.</figref><figref num="115">It is a figure which shows an example of the operation of Embodiment 4.</figref><figref num="116">It is a figure which shows an example of the operation of Embodiment 4.</figref><figref num="117">It is a figure which shows an example of the operation of Embodiment 4.</figref><figref num="118">It is a timing diagram of the transmission signal in the information communication apparatus of Embodiment 5.</figref><figref num="119">It is a figure which shows the relationship between the transmission signal and the reception signal in Embodiment 5.</figref><figref num="120">It is a figure which shows the relationship between the transmission signal and the reception signal in Embodiment 5.</figref><figref num="121">It is a figure which shows the relationship between the transmission signal and the reception signal in Embodiment 5.</figref><figref num="122">It is a figure which shows the relationship between the transmission signal and the reception signal in Embodiment 5.</figref><figref num="123">It is a figure which shows the relationship between the transmission signal and the reception signal in Embodiment 5.</figref><figref num="124">It is a figure which shows the example of the environment in the house in Embodiment 6.</figref><figref num="125">It is a figure which shows the example of the communication between the home electric appliance and the smartphone in Embodiment 6.</figref><figref num="126">It is a figure which shows one example of the structure of the transmission side apparatus in Embodiment 6.</figref><figref num="127">It is a figure which shows one example of the structure of the receiving side apparatus in Embodiment 6.</figref><figref num="128">It is a figure which shows the flow of the process of transmitting information to a receiving side apparatus by blinking LED of the transmitting side apparatus in Embodiment 6.</figref><figref num="129">It is a figure which shows the flow of the process of transmitting information to a receiving side apparatus by blinking LED of the transmitting side apparatus in Embodiment 6.</figref><figref num="130">It is a figure which shows the flow of the process of transmitting information to a receiving side apparatus by blinking LED of the transmitting side apparatus in Embodiment 6.</figref><figref num="131">It is a figure which shows the flow of the process of transmitting information to a receiving side apparatus by blinking LED of the transmitting side apparatus in Embodiment 6.</figref><figref num="132">It is a figure which shows the flow of the process of transmitting information to a receiving side apparatus by blinking LED of the transmitting side apparatus in Embodiment 6.</figref><figref num="133">It is a figure for demonstrating the procedure which communicates with a user and a device using visible light in Embodiment 7.</figref><figref num="134">It is a figure for demonstrating the procedure which communicates with a user and a device using visible light in Embodiment 7.</figref><figref num="135">It is a figure for demonstrating the procedure from the purchase of a device to the initial setting of a device in Embodiment 7.</figref><figref num="136">It is a figure for demonstrating the service only for a serviceman when the device in Embodiment 7 breaks down.</figref><figref num="137">It is a figure for demonstrating the service for confirming a cleaning situation using a vacuum cleaner and visible light communication in Embodiment 7.</figref><figref num="138">It is a schematic diagram of the home delivery service support using optical communication in Embodiment 8.</figref><figref num="139">It is a flowchart for demonstrating the home delivery service support using optical communication in Embodiment 8.</figref><figref num="140">It is a flowchart for demonstrating the home delivery service support using optical communication in Embodiment 8.</figref><figref num="141">It is a flowchart for demonstrating the home delivery service support using optical communication in Embodiment 8.</figref><figref num="142">It is a flowchart for demonstrating the home delivery service support using optical communication in Embodiment 8.</figref><figref num="143">It is a flowchart for demonstrating the home delivery service support using optical communication in Embodiment 8.</figref><figref num="144">It is a flowchart for demonstrating the home delivery service support using optical communication in Embodiment 8.</figref><figref num="145">It is a figure for demonstrating the process of registering a user and a mobile phone in use to the server in Embodiment 9.</figref><figref num="146">It is a figure for demonstrating the process which performs the analysis of the user voice characteristic in Embodiment 9.</figref><figref num="147">It is a figure for demonstrating the process which prepares the speech recognition process in Embodiment 9.</figref><figref num="148">It is a figure for demonstrating the process of collecting sound from the peripheral sound collecting equipment in Embodiment 9.</figref><figref num="149">It is a figure for demonstrating the analysis process of the environmental sound characteristic in Embodiment 9.</figref><figref num="150">It is a figure for demonstrating the process of canceling the sound from the sound output device existing in the periphery in Embodiment 9. FIG.</figref><figref num="151">It is a figure for demonstrating the process of selecting the cooking menu and setting the operation content to the microwave oven in Embodiment 9.</figref><figref num="152">It is a figure for demonstrating the process of acquiring the notification sound for a microwave oven in Embodiment 9 from the DB of a server or the like, and setting it in a microwave oven.</figref><figref num="153">It is a figure for demonstrating the process which adjusts the notification sound of the microwave oven in Embodiment 9.</figref><figref num="154">It is a figure which shows the example of the waveform of the notification sound set in the microwave oven in Embodiment 9.</figref><figref num="155">It is a figure for demonstrating the process which displays the cooking content in Embodiment 9.</figref><figref num="156">It is a figure for demonstrating the process of recognizing the notification sound of the microwave oven in Embodiment 9.</figref><figref num="157">It is a figure for demonstrating the process which performs the process of collecting the sound from the peripheral sound collecting device and recognizing the microwave oven notification sound in Embodiment 9.</figref><figref num="158">It is a figure for demonstrating the process of notifying a user of the end of operation of the microwave oven in Embodiment 9.</figref><figref num="159">It is a figure for demonstrating the process of confirming the mobile phone operation state in Embodiment 9.</figref><figref num="160">It is a figure for demonstrating the process which performs the tracking of the user position in Embodiment 9.</figref><figref num="161">While canceling the voice from the voice output device, it recognizes the notification sound of the home appliance, makes the electronic device that can communicate recognize the current position of the user (operator), and from the recognition result of the user position, the position is close to the user position. It is a figure which showed that a certain device is made to notify a user.</figref><figref num="162">It is a figure which shows the contents of the database held in the server, the mobile phone, or the microwave oven in Embodiment 9. FIG.</figref><figref num="163">In the figure which shows that the user in Embodiment 9 cooks based on the cooking procedure displayed on the mobile phone, and the user operates the display content of the mobile phone by voice such as "Next" and "Return". be.</figref><figref num="164">It is a figure which shows that the user moves to another place at the time of starting the operation of the microwave oven in Embodiment 9, waiting for the end of the operation, while boiling a simmered dish, and the like.</figref><figref num="165">For devices that are connected to a mobile phone via a network and that can recognize the user's position and the existence of the user, such as cameras, microphones, and motion sensors, the mobile phone It is a figure which shows that the instruction to detect a user is transmitted from.</figref><figref num="166">As an example of user detection in the ninth embodiment, it is a figure showing that a camera attached to a television recognizes a user's face, and a motion sensor of an air conditioner recognizes the movement or existence of the user.</figref><figref num="167">It is a figure which shows that the user was detected from the device which detected the user, and the relative position from the detected device to the user is transmitted to the mobile phone.</figref><figref num="168">It is a figure which shows that the mobile phone recognizes the operation end sound of the microwave oven in Embodiment 9.</figref><figref num="169">A mobile phone that recognizes the end of microwave oven operation sends a command to notify the user of the end of microwave oven operation to a device that has a screen display function or a voice output function among the devices that are detecting the user. It is a figure which shows.</figref><figref num="170">It is a figure which shows that the apparatus which received the instruction in Embodiment 9 notifies a user of the notification content.</figref><figref num="171">It is a figure which shows that the operation end sound of a microwave oven is recognized by a device which is connected to a mobile phone via a network, has a microphone, and exists in the vicinity of the microwave oven.</figref><figref num="172">It is a figure which shows that the end of operation of a microwave oven is notified to a mobile phone from the device which recognized this.</figref><figref num="173">When the mobile phone receives the notification of the end of operation of the microwave oven, if the mobile phone is near the user, it indicates that the user is notified of the end of operation of the microwave oven by using the screen display or voice output of the mobile phone. It is a figure.</figref><figref num="174">It is a figure which shows that the user is notified of the end of operation of a microwave oven.</figref><figref num="175">It is a figure which shows that the user who received the notification of the end of operation of a microwave oven moves to the kitchen.</figref><figref num="176">It indicates that information such as the end of operation is transmitted from the microwave oven to the mobile phone by wireless communication, a notification command is given from the mobile phone to the TV that the user is watching, and the user is notified by the screen display or voice of the TV. It is a figure.</figref><figref num="177">It is a figure which shows that the information such as the end of operation is transmitted from the microwave oven to the television which a user is watching by wireless communication, and the user is notified by using the screen display and voice of the television.</figref><figref num="178">It is a figure which shows that the user is notified by the screen display of the television, and the voice.</figref><figref num="179">It is a figure which shows that the information is notified to the user in a remote place.</figref><figref num="180">It is a figure which showed that the information is transmitted to a mobile phone via a personal computer or the like when a direct communication cannot be performed from a microwave oven to a mobile phone which is a hub.</figref><figref num="181">It is a figure which shows that the mobile phone which received the communication of FIG. 180 traces the information communication path in the reverse direction, and sends information such as an operation command to a microwave oven.</figref><figref num="182">It is a figure which showed that the information is notified to the user when the air conditioner which is an information source device cannot communicate directly with the mobile phone which becomes a hub.</figref><figref num="183">It is a figure for demonstrating the system which used the communication device which used the radio wave of 700-900MHz.</figref><figref num="184">It is a figure which shows that the mobile phone of a remote place notifies a user of information.</figref><figref num="185">It is a figure which shows that the mobile phone of a remote place notifies a user of information.</figref><figref num="186">In the same case as in FIG. 185, it is a figure when the television on the second floor plays the role of the relay device instead of the device that relays the notification recognition device and the information notification device.</figref><figref num="187">It is a figure which shows the example of the environment in the house in Embodiment 10.</figref><figref num="188">It is a figure which shows the example of the communication between the home electric appliance and the smartphone in Embodiment 10.</figref><figref num="189">It is a figure which shows the structure of the transmission side apparatus in Embodiment 10.</figref><figref num="190">It is a figure which shows the structure of the receiving side apparatus in Embodiment 10.</figref><figref num="191">FIG. 187 is a sequence diagram in the case where the transmitting side terminal (TV) performs wireless LAN authentication with the receiving side terminal (tablet) by using optical communication.</figref><figref num="192">It is a sequence diagram in the case of performing authentication in the application in Embodiment 10.</figref><figref num="193">It is a flowchart which shows the operation of the transmitting side terminal in Embodiment 10.</figref><figref num="194">It is a flowchart which shows the operation of the receiving side terminal in Embodiment 10.</figref><figref num="195">It is a sequence diagram that the mobile AV terminal 1 in Embodiment 11 transmits data to the mobile AV terminal 2.</figref><figref num="196">It is a screen transition diagram when the mobile AV terminal 1 in Embodiment 11 transmits data to the mobile AV terminal 2.</figref><figref num="197">It is a screen transition diagram when the mobile AV terminal 1 in Embodiment 11 transmits data to the mobile AV terminal 2.</figref><figref num="198">It is a system schematic diagram when the mobile AV terminal 1 in Embodiment 11 is a digital camera.</figref><figref num="199">It is a system schematic diagram when the mobile AV terminal 1 in Embodiment 11 is a digital camera.</figref><figref num="200">It is a system schematic diagram when the mobile AV terminal 1 in Embodiment 11 is a digital camera.</figref><figref num="201">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="202">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="203">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="204">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="205">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="206">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="207">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="208">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="209">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="210">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="211">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="212">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="213">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="214">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="215">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="216">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="217">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="218">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="219">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="220">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="221">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="222">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="223">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="224">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="225">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="226">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="227">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="228">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="229">It is a figure which shows the state of the receiver in Embodiment 12.</figref><figref num="230">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="231">It is a figure which shows the state of the receiver in Embodiment 12.</figref><figref num="232">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="233">It is a figure which shows the state of the receiver in Embodiment 12.</figref><figref num="234">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="235">It is a figure which shows the state of the receiver in Embodiment 12.</figref><figref num="236">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="237">It is a figure which shows the state of the receiver in Embodiment 12.</figref><figref num="238">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="239">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="240">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="241">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="242">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="243">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="244">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="245">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="246">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="247">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="248">It is a figure which shows the luminance change of the transmitter in Embodiment 12.</figref><figref num="249">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="250">It is a figure which shows the luminance change of the transmitter in Embodiment 12.</figref><figref num="251">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="252">It is a figure which shows the luminance change of the transmitter in Embodiment 12.</figref><figref num="253">It is a flowchart which shows an example of the processing operation of the transmitter in Embodiment 12.</figref><figref num="254">It is a figure which shows the luminance change of the transmitter in Embodiment 12.</figref><figref num="255">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="256">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="257">It is a flowchart which shows an example of the processing operation of the transmitter in Embodiment 12.</figref><figref num="258">It is a figure which shows the configuration example of the transmitter in Embodiment 12.</figref><figref num="259">It is a figure which shows the configuration example of the transmitter in Embodiment 12.</figref><figref num="260">It is a figure which shows the configuration example of the transmitter in Embodiment 12.</figref><figref num="261">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="262">It is a figure which shows the example of the display and shooting by a receiver and a transmitter in Embodiment 12.</figref><figref num="263">It is a flowchart which shows an example of the processing operation of the transmitter in Embodiment 12.</figref><figref num="264">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="265">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="266">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="267">It is a figure which shows the state of the receiver in Embodiment 12.</figref><figref num="268">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="269">It is a figure which shows the state of the receiver in Embodiment 12.</figref><figref num="270">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="271">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="272">It is a figure which shows an example of the wavelength of the transmitter in Embodiment 12.</figref><figref num="273">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="274">It is a figure which shows the configuration example of the system including the receiver and the transmitter in Embodiment 12.</figref><figref num="275">It is a flowchart which shows an example of the processing operation of the system in Embodiment 12.</figref><figref num="276">It is a figure which shows the configuration example of the system including the receiver and the transmitter in Embodiment 12.</figref><figref num="277">It is a flowchart which shows an example of the processing operation of the system in Embodiment 12.</figref><figref num="278">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="279">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="280">It is a figure which shows the configuration example of the system including the receiver and the transmitter in Embodiment 12.</figref><figref num="281">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="282">It is a figure which shows the application example of the receiver and the transmitter in Embodiment 12.</figref><figref num="283">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="284">It is a figure which shows the configuration example of the system including the receiver and the transmitter in Embodiment 12.</figref><figref num="285">It is a flowchart which shows an example of the processing operation of the system in Embodiment 12.</figref><figref num="286">It is a flowchart which shows an example of the processing operation of the receiver in Embodiment 12.</figref><figref num="287A">It is a figure which shows an example of the structure of the transmitter in Embodiment 12.</figref><figref num="287B">It is a figure which shows another example of the structure of the transmitter in Embodiment 12.</figref><figref num="288">It is a flowchart which shows an example of the processing operation of a receiver and a transmitter in Embodiment 12.</figref><figref num="289">It is a flowchart which shows an example of the processing operation about a receiver and a transmitter in Embodiment 13.</figref><figref num="290">It is a flowchart which shows an example of the processing operation about a receiver and a transmitter in Embodiment 13.</figref><figref num="291">It is a flowchart which shows an example of the processing operation about a receiver and a transmitter in Embodiment 13.</figref><figref num="292">It is a flowchart which shows an example of the processing operation about a receiver and a transmitter in Embodiment 13.</figref><figref num="293">It is a flowchart which shows an example of the processing operation about a receiver and a transmitter in Embodiment 13.</figref><figref num="294">It is a figure which shows the application example of the transmitter in Embodiment 13.</figref><figref num="295">It is a figure which shows the application example of the transmitter in Embodiment 13.</figref><figref num="296">It is a figure which shows the application example of the transmitter in Embodiment 13.</figref><figref num="297">It is a figure which shows the application example of the transmitter and the receiver in Embodiment 13.</figref><figref num="298">It is a figure which shows the application example of the transmitter and the receiver in Embodiment 13.</figref><figref num="299">It is a figure which shows the application example of the transmitter and the receiver in Embodiment 13.</figref><figref num="300">It is a figure which shows the application example of the transmitter and the receiver in Embodiment 13.</figref><figref num="301A">It is a figure which shows the example of the transmission signal in Embodiment 13.</figref><figref num="301B">It is a figure which shows the other example of the transmission signal in Embodiment 13.</figref><figref num="302">It is a figure which shows the example of the transmission signal in Embodiment 13.</figref><figref num="303A">It is a figure which shows the example of the transmission signal in Embodiment 13.</figref><figref num="303B">It is a figure which shows the other example of the transmission signal in Embodiment 13.</figref><figref num="304">It is a figure which shows the example of the transmission signal in Embodiment 13.</figref><figref num="305A">It is a figure which shows the example of the transmission signal in Embodiment 13.</figref><figref num="305B">It is a figure which shows the example of the transmission signal in Embodiment 13.</figref><figref num="306">It is a figure which shows the application example of the transmitter in Embodiment 13.</figref><figref num="307">It is a figure which shows the application example of the transmitter in Embodiment 13.</figref><figref num="308">It is a figure for demonstrating the image pickup element in Embodiment 13.</figref><figref num="309">It is a figure for demonstrating the image pickup element in Embodiment 13.</figref><figref num="310">It is a figure for demonstrating the image pickup element in Embodiment 13.</figref><figref num="311A">It is a flowchart which shows the processing operation of the receiving apparatus (imaging apparatus) which concerns on the modification of each embodiment.</figref><figref num="311B">It is a figure which contrasts and shows the normal image pickup mode and the macro image pickup mode which concerns on the modification of each embodiment.</figref><figref num="312">It is a figure which shows the display device which displays the image, etc. which concerns on the modification of each embodiment.</figref><figref num="313">It is a figure which shows an example of the processing operation of the display device which concerns on the modification of each embodiment.</figref><figref num="314">It is a figure which shows the example of the part which transmits a signal in the display device which concerns on the modification of each embodiment.</figref><figref num="315">It is a figure which shows the other example of the processing operation of the display device which concerns on the modification of each embodiment.</figref><figref num="316">It is a figure which shows the other example of the part which transmits a signal in the display device which concerns on the modification of each embodiment.</figref><figref num="317">It is a figure which shows the further example of the processing operation by the display device which concerns on the modification of each embodiment.</figref><figref num="318">It is a figure which shows the structure of the communication system which includes the transmitter and the receiver which concerns on the modification of each embodiment.</figref><figref num="319">It is a flowchart which shows the processing operation of the communication system which concerns on the modification of each embodiment.</figref><figref num="320">It is a figure which shows the example of the signal transmission which concerns on the modification of each embodiment.</figref><figref num="321">It is a figure which shows the example of the signal transmission which concerns on the modification of each embodiment.</figref><figref num="322">It is a figure which shows the example of the signal transmission which concerns on the modification of each embodiment.</figref><figref num="323A">It is a figure which shows the example of the signal transmission which concerns on the modification of each embodiment.</figref><figref num="323B">It is a figure which shows the example of the signal transmission which concerns on the modification of each embodiment.</figref><figref num="323C">It is a figure which shows the example of the signal transmission which concerns on the modification of each embodiment.</figref><figref num="323D">It is a flowchart which shows the processing operation of the communication system including the display or the projector and the receiver which concerns on the modification of each embodiment.</figref><figref num="324">It is a figure which shows the example of the transmission signal which concerns on the modification of each embodiment.</figref><figref num="325">It is a figure which shows the example of the transmission signal which concerns on the modification of each embodiment.</figref><figref num="326">It is a figure which shows the example of the transmission signal which concerns on the modification of each embodiment.</figref><figref num="327A">It is a figure which shows an example of the image pickup element of the receiver which concerns on the modification of each embodiment.</figref><figref num="327B">It is a figure which shows the structural example of the internal circuit of the image pickup apparatus of the receiver which concerns on the modification of each embodiment.</figref><figref num="327C">It is a figure which shows the example of the transmission signal which concerns on the modification of each embodiment.</figref><figref num="327D">It is a figure which shows the example of the transmission signal which concerns on the modification of each embodiment.</figref><figref num="328A">It is a figure for demonstrating the image pickup mode of the receiver which concerns on the modification of each embodiment.</figref><figref num="328B">It is a flowchart which shows the processing operation using the special image pickup mode A of the receiver which concerns on the modification of each embodiment.</figref><figref num="329A">It is a figure for demonstrating another image pickup mode of the receiver which concerns on the modification of each embodiment.</figref><figref num="329B">It is a flowchart which shows the processing operation using the special image pickup mode B of the receiver which concerns on the modification of each embodiment.</figref><figref num="330A">It is a figure for demonstrating still another image pickup mode of the receiver which concerns on the modification of each embodiment.</figref><figref num="330B">It is a flowchart which shows the processing operation using the special image pickup mode C of the receiver which concerns on the modification of each embodiment.</figref><figref num="331A">It is a flowchart of the information communication method which concerns on one aspect of this invention.</figref><figref num="331B">It is a block diagram of the information communication apparatus which concerns on one aspect of this invention.</figref><figref num="331C">It is a flowchart of the information communication method which concerns on one aspect of this invention.</figref><figref num="331D">It is a block diagram of the information communication apparatus which concerns on one aspect of this invention.</figref><figref num="332">It is a figure which shows an example of the image obtained by the information communication method which concerns on one aspect of this invention.</figref><figref num="333A">It is a flowchart of the information communication method which concerns on other aspects of this invention.</figref><figref num="333B">It is a block diagram of the information communication apparatus which concerns on other aspects of this invention.</figref><figref num="334A">It is a flowchart of the information communication method which concerns on still another aspect of this invention.</figref><figref num="334B">It is a block diagram of the information communication apparatus which concerns on still another aspect of this invention.</figref><figref num="335">FIG. 335 is a diagram showing an example of each mode of the receiver according to the fourteenth embodiment.</figref><figref num="336">FIG. 336 is a diagram showing an example of the photographing operation of the receiver in the fourteenth embodiment.</figref><figref num="337">FIG. 337 is a diagram showing another example of the photographing operation of the receiver according to the fourteenth embodiment.</figref><figref num="338A">FIG. 338A is a diagram showing another example of the photographing operation of the receiver in the fourteenth embodiment.</figref><figref num="338B">FIG. 338B is a diagram showing another example of the photographing operation of the receiver in the fourteenth embodiment.</figref><figref num="338C">FIG. 338C is a diagram showing another example of the photographing operation of the receiver in the fourteenth embodiment.</figref><figref num="339A">FIG. 339A is a diagram showing an example of the camera arrangement of the receiver in the fourteenth embodiment.</figref><figref num="339B">FIG. 339B is a diagram showing another example of the camera arrangement of the receiver in the fourteenth embodiment.</figref><figref num="340">FIG. 340 is a diagram showing an example of the display operation of the receiver in the fourteenth embodiment.</figref><figref num="341">FIG. 341 is a diagram showing an example of the display operation of the receiver in the fourteenth embodiment.</figref><figref num="342">FIG. 342 is a diagram showing an example of the operation of the receiver according to the fourteenth embodiment.</figref><figref num="343">FIG. 343 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="344">FIG. 344 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="345">FIG. 345 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="346">FIG. 346 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="347">FIG. 347 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="348">FIG. 348 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="349">FIG. 349 is a diagram showing an example of the operation of the receiver, the transmitter, and the server in the fourteenth embodiment.</figref><figref num="350">FIG. 350 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="351">FIG. 351 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="352">FIG. 352 is a diagram showing an example of initial setting of the receiver in the fourteenth embodiment.</figref><figref num="353">FIG. 353 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="354">FIG. 354 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="355">FIG. 355 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="356">FIG. 356 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="357">FIG. 357 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="358">FIG. 358 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="359A">FIG. 359A is a diagram showing a pen used for operating the receiver in the fourteenth embodiment.</figref><figref num="359B">FIG. 359B is a diagram showing the operation of the receiver using the pen in the 14th embodiment.</figref><figref num="360">FIG. 360 is a diagram showing an example of the appearance of the receiver according to the fourteenth embodiment.</figref><figref num="361">FIG. 361 is a diagram showing another example of the appearance of the receiver according to the fourteenth embodiment.</figref><figref num="362">FIG. 362 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="363A">FIG. 363A is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="363B">FIG. 363B is a diagram showing an application example using the receiver in the fourteenth embodiment.</figref><figref num="364A">FIG. 364A is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="364B">FIG. 364B is a diagram showing an application example using the receiver in the fourteenth embodiment.</figref><figref num="365A">FIG. 365A is a diagram showing an example of the operation of the transmitter according to the fourteenth embodiment.</figref><figref num="365B">FIG. 365B is a diagram showing another example of the operation of the transmitter according to the fourteenth embodiment.</figref><figref num="366">FIG. 366 is a diagram showing another example of the operation of the transmitter according to the fourteenth embodiment.</figref><figref num="367">FIG. 367 is a diagram showing another example of the operation of the transmitter according to the fourteenth embodiment.</figref><figref num="368">FIG. 368 is a diagram showing an example of a communication mode between a plurality of transmitters and a receiver in the 14th embodiment.</figref><figref num="369">FIG. 369 is a diagram showing an example of the operation of a plurality of transmitters according to the fourteenth embodiment.</figref><figref num="370">FIG. 370 is a diagram showing another example of the communication mode between the plurality of transmitters and the receiver in the fourteenth embodiment.</figref><figref num="371">FIG. 371 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="372">FIG. 372 is a diagram showing an application example of the receiver according to the fourteenth embodiment.</figref><figref num="373">FIG. 373 is a diagram showing an application example of the receiver according to the fourteenth embodiment.</figref><figref num="374">FIG. 374 is a diagram showing an application example of the receiver according to the fourteenth embodiment.</figref><figref num="375">FIG. 375 is a diagram showing an application example of the transmitter according to the fourteenth embodiment.</figref><figref num="376">FIG. 376 is a diagram showing an application example of the transmitter according to the fourteenth embodiment.</figref><figref num="377">FIG. 377 is a diagram showing an application example of the receiving method according to the fourteenth embodiment.</figref><figref num="378">FIG. 378 is a diagram showing an application example of the transmitter according to the fourteenth embodiment.</figref><figref num="379">FIG. 379 is a diagram showing an application example of the transmitter according to the fourteenth embodiment.</figref><figref num="380">FIG. 380 is a diagram showing an application example of the transmitter according to the fourteenth embodiment.</figref><figref num="381">FIG. 381 is a diagram showing another example of the operation of the receiver in the fourteenth embodiment.</figref><figref num="382">FIG. 382 is a flowchart showing an example of the operation of the receiver in the fifteenth embodiment.</figref><figref num="383">FIG. 383 is a flowchart showing another example of the operation of the receiver in the fifteenth embodiment.</figref><figref num="384A">FIG. 384A is a block diagram showing an example of the transmitter according to the fifteenth embodiment.</figref><figref num="384B">FIG. 384B is a block diagram showing another example of the transmitter according to the fifteenth embodiment.</figref><figref num="385">FIG. 385 is a diagram showing a configuration example of a system including a plurality of transmitters according to the fifteenth embodiment.</figref><figref num="386">FIG. 386 is a block diagram showing another example of the transmitter according to the fifteenth embodiment.</figref><figref num="387A">FIG. 387A is a diagram showing an example of the transmitter according to the fifteenth embodiment.</figref><figref num="387B">FIG. 387B is a diagram showing an example of the transmitter according to the fifteenth embodiment.</figref><figref num="387C">FIG. 387C is a diagram showing an example of the transmitter according to the fifteenth embodiment.</figref><figref num="388A">FIG. 388A is a diagram showing an example of the transmitter according to the fifteenth embodiment.</figref><figref num="388B">FIG. 388B is a diagram showing an example of the transmitter according to the fifteenth embodiment.</figref><figref num="389">FIG. 389 is a diagram showing an example of processing operations of the receiver, the transmitter, and the server in the fifteenth embodiment.</figref><figref num="390">FIG. 390 is a diagram showing an example of processing operations of the receiver, the transmitter, and the server in the fifteenth embodiment.</figref><figref num="391">FIG. 391 is a diagram showing an example of processing operations of the receiver, the transmitter, and the server in the fifteenth embodiment.</figref><figref num="392A">FIG. 392A is an explanatory diagram for explaining synchronization of a plurality of transmitters in the fifteenth embodiment.</figref><figref num="392B">FIG. 392B is an explanatory diagram for explaining synchronization of a plurality of transmitters in the fifteenth embodiment.</figref><figref num="393">FIG. 393 is a diagram showing an example of the operation of the transmitter and the receiver in the fifteenth embodiment.</figref><figref num="394">FIG. 394 is a diagram showing an example of the operation of the transmitter and the receiver in the fifteenth embodiment.</figref><figref num="395">FIG. 395 is a diagram showing an example of the operation of the transmitter, receiver, and server in the fifteenth embodiment.</figref><figref num="396">FIG. 396 is a diagram showing an example of the operation of the transmitter and the receiver in the fifteenth embodiment.</figref><figref num="397">FIG. 397 is a diagram showing an example of the appearance of the receiver according to the fifteenth embodiment.</figref><figref num="398">FIG. 398 is a diagram showing an example of the operation of the transmitter, receiver, and server in the fifteenth embodiment.</figref><figref num="399">FIG. 399 is a diagram showing an example of the operation of the transmitter and the receiver in the fifteenth embodiment.</figref><figref num="400">FIG. 400 is a diagram showing an example of the operation of the transmitter and the receiver in the fifteenth embodiment.</figref><figref num="401">FIG. 401 is a diagram showing an example of the operation of the transmitter and the receiver in the fifteenth embodiment.</figref><figref num="402">FIG. 402 is a diagram showing an example of the operation of the transmitter and the receiver in the fifteenth embodiment.</figref><figref num="403A">FIG. 403A is a diagram showing an example of the configuration of information transmitted by the transmitter in the fifteenth embodiment.</figref><figref num="403B">FIG. 403B is a diagram showing another example of the configuration of the information transmitted by the transmitter in the fifteenth embodiment.</figref><figref num="404">FIG. 404 is a diagram showing an example of a four-value PPM modulation method using a transmitter in the fifteenth embodiment.</figref><figref num="405">FIG. 405 is a diagram showing an example of the PPM modulation method by the transmitter in the fifteenth embodiment.</figref><figref num="406">FIG. 406 is a diagram showing an example of the PPM modulation method in the transmitter according to the fifteenth embodiment.</figref><figref num="407A">FIG. 407A is a diagram showing an example of a luminance change pattern corresponding to a header (preamble portion) in the fifteenth embodiment.</figref><figref num="407B">FIG. 407B is a diagram showing an example of the luminance change pattern in the fifteenth embodiment.</figref><figref num="408A">FIG. 408A is a diagram showing an example of a pattern of luminance change in the fifteenth embodiment.</figref><figref num="408B">FIG. 408B is a diagram showing an example of the luminance change pattern in the fifteenth embodiment.</figref><figref num="409">FIG. 409 is a diagram showing an example of the operation of the transmitter configured as the television in the sixteenth embodiment.</figref><figref num="410">FIG. 410 is a diagram showing an example of the operation of the transmitter and the receiver in the 16th embodiment.</figref><figref num="411">FIG. 411 is a diagram showing an example of the operation of the transmitter, receiver, and server according to the sixteenth embodiment.</figref><figref num="412">FIG. 412 is a diagram showing an example of the operation of the transmitter and the receiver in the 16th embodiment.</figref><figref num="413">FIG. 413 is a diagram showing an example of the operation of the transmitter in the 16th embodiment.</figref><figref num="414">FIG. 414 is a diagram showing an example of the operation of the transmitter according to the sixteenth embodiment.</figref><figref num="415">FIG. 415 is a diagram showing an example of the operation of the transmitter in the 16th embodiment.</figref><figref num="416">FIG. 416 is a diagram for explaining the imaging in the 16th embodiment.</figref><figref num="417">FIG. 417 is a diagram for explaining transmission and imaging in the 16th embodiment.</figref><figref num="418">FIG. 418 is a diagram for explaining the transmission in the sixteenth embodiment.</figref><figref num="419">FIG. 419 is a diagram showing an example of the transmission signal of the 17th embodiment.</figref><figref num="420">FIG. 420 is a diagram showing an example of the transmission signal of the 17th embodiment.</figref><figref num="421A">FIG. 421A is a diagram showing an example of a captured image (bright line image) of the receiver according to the seventeenth embodiment.</figref><figref num="421B">FIG. 421B is a diagram showing an example of a captured image (bright line image) of the receiver according to the seventeenth embodiment.</figref><figref num="421C">FIG. 421C is a diagram showing an example of a captured image (bright line image) of the receiver in the 17th embodiment.</figref><figref num="422A">FIG. 422A is a diagram showing an example of a captured image (bright line image) of the receiver in the 17th embodiment.</figref><figref num="422B">FIG. 422B is a diagram showing an example of a captured image (bright line image) of the receiver in the 17th embodiment.</figref><figref num="423A">FIG. 423A is a diagram showing an example of a captured image (bright line image) of the receiver in the 17th embodiment.</figref><figref num="423B">FIG. 423B is a diagram showing an example of a captured image (bright line image) of the receiver in the 17th embodiment.</figref><figref num="423C">FIG. 423C is a diagram showing an example of a captured image (bright line image) of the receiver in the 17th embodiment.</figref><figref num="424">FIG. 424 is a diagram showing an example of a captured image (bright line image) of the receiver according to the 17th embodiment.</figref><figref num="425">FIG. 425 is a diagram showing an example of a transmission signal according to the 17th embodiment.</figref><figref num="426">FIG. 426 is a diagram showing an example of the operation of the receiver in the 17th embodiment.</figref><figref num="427">FIG. 427 is a diagram showing an example of an instruction to the user to be displayed on the screen of the receiver in the 17th embodiment.</figref><figref num="428">FIG. 428 is a diagram showing an example of an instruction to the user to be displayed on the screen of the receiver according to the seventeenth embodiment.</figref><figref num="429">FIG. 429 is a diagram showing an example of the signal transmission method according to the seventeenth embodiment.</figref><figref num="430">FIG. 430 is a diagram showing an example of the signal transmission method according to the seventeenth embodiment.</figref><figref num="431">FIG. 431 is a diagram showing an example of the signal transmission method according to the seventeenth embodiment.</figref><figref num="432">FIG. 432 is a diagram showing an example of the signal transmission method according to the seventeenth embodiment.</figref><figref num="433">FIG. 433 is a diagram for explaining the use case in the seventeenth embodiment.</figref><figref num="434">FIG. 434 is a diagram showing an information table transmitted by the smartphone to the server in the 17th embodiment.</figref><figref num="435">FIG. 435 is a block diagram of the server according to the seventeenth embodiment.</figref><figref num="436">FIG. 436 is a flowchart showing the entire processing of the system according to the seventeenth embodiment.</figref><figref num="437">FIG. 437 is a diagram showing an information table transmitted by the server in the 17th embodiment to the smartphone.</figref><figref num="438">FIG. 438 is a diagram showing a screen flow displayed on the wearable device from the time when the user receives information from the server in front of the store to the time when the user actually purchases the product in the 17th embodiment.</figref><figref num="439">FIG. 439 is a diagram for explaining another use case in the seventeenth embodiment.</figref><figref num="440">FIG. 440 is a diagram showing a service providing system using the receiving method described in the embodiment described above.</figref><figref num="441">FIG. 441 is a flowchart showing the flow of service provision.</figref><figref num="442">FIG. 442 is a flowchart showing service provision in another example. The description of the steps overlapping with FIG. 441 will be omitted.</figref><figref num="443">FIG. 443 is a flowchart showing service provision in another example.</figref><figref num="444A">FIG. 444A is a flowchart of an information communication method according to one aspect of the present invention.</figref><figref num="444B">FIG. 444B is a block diagram of an information communication device according to an aspect of the present invention.</figref>
The information communication method according to one aspect of the present invention is an information communication method for acquiring information from a subject, and includes a position information transmission step for transmitting position information indicating the position of an image sensor used for photographing the subject, and the above-mentioned information communication method. The exposure line included in the image sensor in the list receiving step of receiving the ID list including a plurality of identification information associated with the position indicated by the position information and the image obtained by photographing the subject by the image sensor. The exposure time setting step for setting the exposure time of the image sensor and the exposure time set by the image sensor for the subject whose brightness changes so that the emission line corresponding to the above is generated according to the change in the brightness of the subject. An image acquisition step of acquiring an emission line image including the emission line by taking a picture with, and an information acquisition step of acquiring information by demolishing data specified by the emission line pattern included in the acquired emission line image. And a search step of searching the ID list for the acquired identification information including the information.
As a result, the information transmitted by the change in the brightness of the subject is acquired by the exposure of the exposure line of the image sensor, so that, for example, no special communication device for performing wireless communication is required, and the information is between various devices. Communication can be enabled. Further, as shown in FIG. 389 described later, for example, since the ID list has been received in advance, even if the acquired information "bc" is only a part of the identification information, appropriate identification is made based on the ID list. Information "abcd" can be specified.
If the acquired identification information including the information is not uniquely specified in the search step, new information is acquired by repeating the image acquisition step and the information acquisition step, and the information communication is performed. The method may further include a re-search step of searching the ID list for identification information including the acquired information and the new information.
As a result, even if the acquired information "b" is only a part of the identification information and the identification information is not uniquely specified by the information alone, as shown in FIG. 389 described later, for example, a new one. Information "c" is acquired, so that appropriate identification information "abcd" can be identified based on the new information and the ID list.
Further, in the information communication method, there is no identification information including the acquired information in the transmission step for transmitting the acquired information and the ID list associated with the position indicated by the location information. In some cases, it may include an error receiving step for receiving error notification information for notifying an error.
As a result, for example, as shown in FIG. 391 described later, when the acquired identification information is not in the ID list, the error notification information is received, so that the user of the receiver who has received the error notification information can receive the error notification information. It is possible to easily understand that the information associated with the acquired identification information cannot be obtained.
Further, in the image acquisition step, by photographing a plurality of the subjects while the image sensor is being moved, the emission line image including a plurality of portions where the emission lines appear is acquired, and in the information acquisition step. By demodulating the data specified by the pattern of the emission line of the portion for each of the portions, the positions of the plurality of subjects are acquired, and the information communication method further obtains the plurality of the acquired portions of the subject. A position estimation step for estimating the position and imaging direction of the image sensor based on the respective positions of the subject and the moving state of the image sensor may be included.
As a result, as shown in FIG. 350, which will be described later, for example, the position and imaging direction of the receiver including the image sensor can be accurately estimated by the brightness change due to the subject such as a plurality of lights.
Further, in the image acquisition step, the emission line image is acquired by photographing a plurality of the subjects reflected on the reflective surface, and in the information acquisition step, the emission line is obtained according to the intensity of the emission line included in the emission line image. Information may be acquired by separating the plurality of bright lines corresponding to each of the subjects and demodulating the data specified by the pattern of the bright lines corresponding to the subject for each subject.
As a result, as shown in FIG. 370, which will be described later, appropriate information can be obtained from each of the subjects even when the brightness of each of the subjects such as a plurality of lights changes.
Further, the information communication method further includes a related information acquisition step of transmitting the identification information to the server and acquiring the related information associated with the identification information from the server, and is broadcast in the image acquisition step. The subject displaying the received content is photographed, and in the search step, the channel used for broadcasting the content displayed by the subject and the reference time which is the time when the content is displayed are taken. The identification information including the Information may be obtained from the server.
As a result, as shown in FIG. 412 described later, for example, not only the related information associated with the information (channel and reference time) transmitted from the transmitter which is the subject whose brightness changes, but also the related information associated with the peripheral time. Information is also acquired. Therefore, even if the information desired by the user cannot be acquired from the transmitter in the first information acquisition step due to the timing of imaging the transmitter being shifted, the information is associated with the information. Related information can be obtained from the server.
Further, the information communication method further captures a lighting image showing the lighting range in which the subject is lit by photographing the subject which is lit without changing the brightness for transmitting a signal. A range specifying step for specifying a range from the emission line image, which is at the same position as the lighting range in the lighting image and has the same size and shape as the lighting range, as a signal transmission range. In the information acquisition step, the data specified by the pattern of the emission line included in the specified signal transmission range may be demolished.
As a result, for example, as shown in FIG. 410 described later, the signal transmission range is specified from the emission line image based on the lighting range (normal lighting range), so that the range in which the signal is transmitted is selected from the emission line image. It can be identified accurately. For example, even if there is a dark part at the end within the signal transmission range, a signal indicating that there is no emission line is transmitted from that part without misunderstanding that no signal is transmitted from that part. Can be properly recognized.
Further, the information communication method further has a determination step of determining a pattern of luminance change by modulating a signal to be transmitted, and the display, which is the subject, has the determined pattern while displaying an image. Therefore, it includes a transmission step of transmitting the signal to be transmitted by changing the brightness, and in the transmission step, light is emitted with a brightness equal to or higher than a predetermined brightness in order to display the image in the display. Only the portion where the brightness is changed may change in brightness according to the pattern.
As a result, as shown in FIG. 413, which will be described later, for example, in the display, the portion set to a brightness lower than the predetermined brightness for displaying the image, that is, the dark portion does not change the brightness. The darkness can be stabilized and darker gradation can be expressed.
Further, the display is provided with a backlight, and the information communication method further displays an image for the left eye and an image for the right eye, which are illuminated by the backlight with a predetermined brightness, in order. 3 The transmission step and the three-dimensional image display step may be alternately repeated, including a three-dimensional image display step.
As a result, for example, as shown in FIG. 409 described later, when the brightness is changed in the transmission step, the view of both eyes of the user is closed by the 3D glass, and the image for the left eye and the right eye in the 3D image display step. When the images for are displayed in order, if only the view of the user's eyes corresponding to those images is opened by the 3D glass, the user can see the 3D image in which the flicker due to the change in brightness is suppressed. can do.
Further, the information communication method further includes a detection step in which a sensor detects a person within the viewing angle of the display or a person in the vicinity of the viewing angle, and in the transmission step, the peripheral is included. When a person within the viewing angle is detected, the signal to be transmitted is transmitted by changing the brightness by a larger amount of change than when a person within the viewing angle is detected, and the viewing angle is within the range. If neither the person inside nor the person in the vicinity is detected, the transmission of the signal due to the change in brightness may be stopped.
As a result, for example, as shown in FIG. 414 described later, when a person in the vicinity of the viewing angle is detected, the display shows a normal amount of change (when a person within the viewing angle is detected). The brightness changes with a change amount larger than the change amount of the brightness change). Therefore, the receiver carried by the person can appropriately receive the signal by imaging the display whose brightness changes even if it is outside the viewing angle.
Further, the information communication method further receives and records a plurality of images included in the broadcast content and a signal broadcast in association with each of the plurality of images, and records the signals in each of the signals. The recording step of acquiring the associated related information from the server and storing it in the recording medium, the determination step of determining the pattern of the brightness change corresponding to each of the signals by modulating each of the signals, and the above-mentioned Each time the display, which is the subject, displays an image included in the recorded content, the signal is transmitted by changing the brightness according to the pattern determined for the signal associated with the image. When the signal associated with any one of the plurality of images included in the content is searched for as the identification information in the display transmission step and the search step, the association associated with the signal. The signal to be broadcast in association with the image for each image included in the content includes the related information selection step of selecting the information from the plurality of related information stored in the recording medium. The broadcast time at which the image is broadcast and the channel used for broadcasting the content may be indicated.
As a result, even when the recorded content is played back, the broadcast time and the channel associated with each of the plurality of images included in the content are transmitted by the brightness change, so that the association related to those images is transmitted. Information can be obtained appropriately.
Further, the information communication method further receives and records a plurality of images included in the broadcast content and a signal broadcast in association with each of the plurality of images, and records the signals in each of the signals. The recording step of acquiring the associated related information from the server and storing it in the recording medium, the determination step of determining the pattern of the brightness change corresponding to each of the signals by modulating each of the signals, and the above-mentioned Each time the display, which is the subject, displays an image included in the recorded content, the signal is transmitted by changing the brightness according to the pattern determined for the signal associated with the image. When the signal associated with any one of the plurality of images included in the content is searched for as the identification information in the display transmission step and the search step, the association associated with the signal. The signal to be broadcast in association with the image for each image included in the content includes the related information selection step of selecting the information from the plurality of related information stored in the recording medium. Image identification information for identifying the image in the content may be shown.
As a result, even when the recorded content is played back, the image identification information associated with each of the plurality of images included in the content is transmitted by the brightness change, so that the related information related to those images is transmitted. Can be obtained properly.
It should be noted that these comprehensive or specific embodiments may be realized in recording media such as systems, methods, integrated circuits, computer programs or computer-readable CD-ROMs, systems, methods, integrated circuits, computer programs. Alternatively, it may be realized by any combination of recording media.
Hereinafter, embodiments will be specifically described with reference to the drawings.
It should be noted that all of the embodiments described below are comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, the order of steps, etc. shown in the following embodiments are examples, and are not intended to limit the present invention. Further, among the components in the following embodiments, the components not described in the independent claim indicating the highest level concept are described as arbitrary components.
(Embodiment 1) Hereinafter, the first embodiment will be described.
(Observation of the brightness of the light emitting part) When capturing a single image, we propose an imaging method that starts and ends exposure at different times for each image sensor, instead of exposing all the image sensors at the same timing. .. FIG. 1 shows an example in which the image sensors arranged in a row are exposed at the same time and the exposure start times are shifted in the order in which the rows are closer to each other. Here, it is called an exposure line of an image pickup element that is exposed at the same time, and a line of pixels on an image corresponding to the image pickup element is called an emission line.
When a blinking light source is photographed on the entire surface of the image sensor using this image pickup method, bright lines (bright and dark lines of pixel values) along the exposure line are generated on the captured image as shown in FIG. .. By recognizing this emission line pattern, it is possible to estimate a change in light source luminance at a speed higher than the imaging frame rate. As a result, by transmitting the signal as a change in the brightness of the light source, it is possible to perform communication at a speed higher than the imaging frame rate. When a light source expresses a signal by taking two kinds of luminance values, the lower luminance value is called low (LO) and the higher luminance value is called high (HI). Low may be in a state where the light source is not shining, or may be shining weaker than high.
By this method, information is transmitted at a speed exceeding the imaging frame rate.
When there are 20 exposure lines where the exposure times do not overlap in one captured image and the frame rate of imaging is 30 fps, it is possible to recognize a change in brightness in a 1-millisecond cycle. When there are 1000 exposure lines where the exposure times do not overlap, it is possible to recognize a change in brightness with a period of 1 / 30,000 second (about 33 microseconds). The exposure time is set to be shorter than, for example, 10 milliseconds.
FIG. 2 shows a case where the exposure of one exposure line is completed and then the exposure of the next exposure line is started.
In this case, when the number of frames per second (frame rate) is f and the number of exposure lines constituting one image is l, if information is transmitted depending on whether each exposure line receives a certain amount of light or not, the maximum is transmitted. Information can be transmitted at a speed of fl bits per second.
In addition, when the exposure is performed with a time difference for each pixel instead of each line, communication is possible at a higher speed.
At this time, the number of pixels per exposure line is m pixels, and when information is transmitted depending on whether or not each pixel receives a certain amount of light or more, the maximum transmission speed is flm bits per second.
As shown in Fig. 3, if the exposure state of each exposure line due to the light emission of the light emitting part can be recognized at multiple levels, the light emission time of the light emitting part can be controlled in a unit time shorter than the exposure time of each exposure line. , More information can be transmitted.
If the exposure state can be recognized at the Elv stage, information can be transmitted at a maximum speed of flElv bits per second.
Further, the basic cycle of transmission can be recognized by causing the light emitting unit to emit light at a timing slightly deviated from the exposure timing of each exposure line.
FIG. 4A shows the case where the exposure of the next exposure line is started before the exposure of one exposure line is completed. That is, the exposure times of adjacent exposure lines partially overlap with each other. With such a configuration, it is possible to increase the number of samples within a predetermined time as compared with the case of (1) waiting for the end of the exposure time of one exposure line and starting the exposure of the next exposure line. By increasing the number of samples within a predetermined time, it becomes possible to more appropriately detect an optical signal generated by an optical transmitter as a subject. That is, it is possible to reduce the error rate when detecting an optical signal. Further, (2) the exposure time of each exposure line can be lengthened as compared with the case of waiting for the end of the exposure time of one exposure line and starting the exposure of the next exposure line, so that the subject is dark. However, it is possible to obtain a brighter image. That is, it is possible to improve the S / N ratio. It should be noted that, in all the exposure lines, it is not necessary that the exposure times of the adjacent exposure lines partially overlap with each other, and the exposure times of some of the exposure lines do not partially overlap with each other. It is also possible to. By configuring some exposure lines so that they do not partially overlap in time, it is possible to suppress the generation of neutral colors due to the overlap of exposure times on the imaging screen, and it becomes possible to detect bright lines more appropriately. ..
In this case, the exposure time is calculated from the brightness of each exposure line, and the light emitting state of the light emitting unit is recognized.
When the brightness of each exposure line is determined by the binary value of whether or not the brightness is equal to or higher than the threshold value, in order to recognize the state of not emitting light, the light emitting unit indicates the state of not emitting light for each line. Must continue for longer than the exposure time of.
FIG. 4B shows the effect of the difference in exposure time when the exposure start times of the exposure lines are the same. 7500a is the case where the exposure end time of the previous exposure line is equal to the exposure start time of the next exposure line, and 7500b is the case where the exposure time is longer than that. As in the case of 7500b, the exposure time of adjacent exposure lines can be extended by partially overlapping the exposure times. That is, the light incident on the image sensor increases, and a bright image can be obtained. Further, since the imaging sensitivity for capturing an image of the same brightness can be suppressed to a low level, an image with less noise can be obtained, and communication errors can be suppressed.
FIG. 4C shows the effect of the difference in the exposure start time of each exposure line when the exposure times are the same. 7501a is the case where the exposure end time of the previous exposure line and the exposure start time of the next exposure line are equal, and 7501b is the case where the exposure of the next exposure line is started earlier than the exposure end of the previous exposure line. As in the case of 7501b, it is possible to increase the number of lines that can be exposed per hour by adopting a configuration in which the exposure times of adjacent exposure lines partially overlap with each other. As a result, the resolution becomes higher and a large amount of information can be obtained. By increasing the sample interval (= difference in exposure start time), the change in light source brightness can be estimated more accurately, the error rate can be reduced, and the change in light source brightness in a shorter time can be recognized. be able to. By having the exposure times overlap, it is possible to recognize the blinking of the light source shorter than the exposure time by utilizing the difference in the exposure amount of the adjacent exposure lines.
As described in FIGS. 4B and 4C, in a configuration in which each exposure line is sequentially exposed so that the exposure times of adjacent exposure lines partially overlap with each other, the exposure time is set to be longer than that in the normal shooting mode. By using the emission line pattern generated by setting it short for signal transmission, it is possible to dramatically improve the communication speed. Here, by setting the exposure time during visible light communication to 1/480 seconds or less, it is possible to generate an appropriate emission line pattern. Here, the exposure time must be set to the exposure time <1/8 × f, where frame frequency = f. The blanking that occurs during shooting is up to half the size of one frame. That is, since the blanking time is less than half of the shooting time, the actual shooting time is 1 / 2f in the shortest time. Further, since it is necessary to receive the information of four values within the time of 1 / 2f, at least the exposure time needs to be shorter than 1 / (2f × 4). Since the normal frame rate is 60 frames / sec or less, by setting the exposure time to 1/480 sec or less, it is possible to generate an appropriate emission line pattern in the image data and perform high-speed signal transmission. Become.
FIG. 4D shows the advantage of short exposure times when the exposure times of the exposure lines do not overlap. When the exposure time is long, even if the light source has a binary brightness change like 7502a, a neutral color part is formed like 7502e in the captured image, and it tends to be difficult to recognize the brightness change of the light source. There is. However, as in the 7502d, after the end of exposure of one exposure line, until the start of exposure of the next exposure line, a predetermined free time (predetermined waiting time) t<sub>D2</sub>By providing the above, it is possible to easily recognize the change in the brightness of the light source. That is, it becomes possible to detect a more appropriate emission line pattern such as 7502f. A configuration such as the 7502d that provides a predetermined free time without exposure has an exposure time t.<sub>E</sub>The time difference between the exposure start times of each exposure line t<sub>D</sub>It can be realized by making it smaller than. When the normal shooting mode has a configuration in which the exposure times of adjacent exposure lines partially overlap with each other, the exposure time is set shorter than in the normal shooting mode until a predetermined free time without exposure occurs. By doing so, it can be realized. Further, even when the normal shooting mode is equal to the exposure end time of the previous exposure line and the exposure start time of the next exposure line, the exposure time is set short until a predetermined non-exposure time occurs. It can be realized. Also, like 7502g, the interval t of the exposure start time of each exposure line.<sub>D</sub>By increasing the amount of free time (predetermined waiting time) t, which is not exposed until the start of exposure of the next exposure line after the end of exposure of one exposure line.<sub>D2</sub>Can be configured to provide. In this configuration, since the exposure time can be lengthened, a bright image can be captured and noise is reduced, so that error tolerance is high. On the other hand, this configuration has the disadvantage that the number of samples is small, as in the 7502h, because the number of exposure lines that can be exposed within a certain period of time is small, so it is desirable to use them properly depending on the situation. For example, by using the former configuration when the imaging target is bright and using the latter configuration when it is dark, it is possible to reduce the estimation error of the light source luminance change.
It should be noted that, in all the exposure lines, it is not necessary that the exposure times of the adjacent exposure lines partially overlap with each other, and the exposure times of some of the exposure lines do not partially overlap with each other. It is also possible to. Further, it is not necessary to provide a predetermined free time (predetermined waiting time) for all exposure lines from the end of exposure of one exposure line to the start of exposure of the next exposure line, and it is not necessary to provide a partial exposure. It is also possible to have a configuration in which the lines partially overlap in time. With such a configuration, it is possible to take advantage of each configuration. In addition, the same readout method is used in the normal shooting mode in which shooting is performed at a normal frame rate (30 fps, 60 fps) and the visible light communication mode in which shooting is performed with an exposure time of 1/480 second or less for visible light communication. Alternatively, the signal may be read out by the circuit. By reading the signal by the same reading method or circuit, it is not necessary to use different circuits for the normal shooting mode and the visible light communication mode, and the circuit scale can be reduced.
Figure 4E shows the minimum change time t of the light source brightness.<sub>S</sub>And the exposure time t<sub>E</sub>And the time difference t of the exposure start time of each exposure line<sub>D</sub>And the relationship with the captured image. t<sub>E</sub>+ t<sub>D</sub><t<sub>S</sub>In this case, one or more exposure lines are always imaged with the light source unchanged from the start to the end of the exposure, so an image with clear brightness such as 7503d is obtained, and the change in brightness of the light source is recognized. Cheap. 2t<sub>E</sub>> t<sub>S</sub>In this case, a bright line with a pattern different from the change in the brightness of the light source may be obtained, and it becomes difficult to recognize the change in the brightness of the light source from the captured image.
Figure 4F shows the transition time t of the light source brightness.<sub>T</sub>And the time difference t of the exposure start time of each exposure line<sub>D</sub>Shows the relationship with. t<sub>T</sub>Compared to t<sub>D</sub>The larger the value, the fewer the exposure lines that become neutral colors, and the easier it is to estimate the brightness of the light source. t<sub>D</sub>> t<sub>T</sub>At this time, the number of exposure lines for neutral colors is 2 or less in a row, which is desirable. t<sub>T</sub>Is less than 1 microsecond when the light source is LED, and about 5 microseconds when the light source is organic EL.<sub>D</sub>By setting the value to 5 microseconds or more, it is possible to easily estimate the brightness of the light source.
Figure 4G shows the high frequency noise of the light source brightness t.<sub>HT</sub>And the exposure time t<sub>E</sub>Shows the relationship with. t<sub>HT</sub>Compared to t<sub>E</sub>The larger the value, the less the influence of high-frequency noise on the captured image, and the easier it is to estimate the brightness of the light source. t<sub>E</sub>Is t<sub>HT</sub>When it is an integral multiple of, the influence of high frequency noise disappears, and the estimation of the light source brightness becomes the easiest. To estimate the light source brightness, t<sub>E</sub>> t<sub>HT</sub>Is desirable. The main cause of high frequency noise comes from switching power circuits, and in many switching power supplies for lamps, t<sub>HT</sub>Is less than 20 microseconds, so t<sub>E</sub>By setting the value to 20 microseconds or more, the brightness of the light source can be easily estimated.
Figure 4H shows t<sub>HT</sub>Exposure time t when is 20 microseconds<sub>E</sub>It is a graph showing the relationship between and the magnitude of high frequency noise. t<sub>HT</sub>Considering that there are variations depending on the light source, from the graph, t<sub>E</sub>Is a value equal to the value when the amount of noise is maximized, and it is efficient to set it as 15 microseconds or more, 35 microseconds or more, 54 microseconds or more, or 74 microseconds or more. Can be confirmed. From the viewpoint of high frequency noise reduction, t<sub>E</sub>Is desirable, but as mentioned above, t<sub>E</sub>There is also a property that it becomes easier to estimate the light source brightness in that the smaller the value is, the less likely it is that a neutral color portion is generated. Therefore, when the cycle of change in light source brightness is 15 to 35 microseconds, t<sub>E</sub>Is 15 microseconds or more, and when the cycle of change in light source brightness is 35 to 54 microseconds, t<sub>E</sub>Is 35 microseconds or more, and when the cycle of change in light source brightness is 54 to 74 microseconds, t<sub>E</sub>Is 54 microseconds or more, and when the cycle of change in light source brightness is 74 microseconds or more, t<sub>E</sub>Should be set to 74 microseconds or longer.
Figure 4I shows the exposure time t.<sub>E</sub>And the recognition success rate are shown. Exposure time t<sub>E</sub>Has a relative meaning to the time when the brightness of the light source is constant, so the period t at which the brightness of the light source changes<sub>S</sub>The exposure time t<sub>E</sub>The horizontal axis is the value divided by (relative exposure time). From the graph, it can be seen that if the recognition success rate is to be almost 100%, the relative exposure time should be 1.2 or less. For example, when the transmission signal is 1 kHz, the exposure time may be about 0.83 milliseconds or less. Similarly, if the recognition success rate is 95% or more, the relative exposure time should be 1.25 or less, and if the recognition success rate is 80% or more, the relative exposure time should be 1.4 or less. In addition, the recognition success rate drops sharply when the relative exposure time is around 1.5, and becomes almost 0% at 1.6, so it is clear that the relative exposure time should not exceed 1.5. It can also be seen that after the recognition rate became 0 at 7507c, it increased again at 7507d, 7507e, and 7507f. Therefore, if you want to capture a bright image with a long exposure time, you can use the exposure time with relative exposure times of 1.9 to 2.2, 2.4 to 2.6, and 2.8 to 3.0. For example, these exposure times may be used as the intermediate mode in FIG. 335.
As shown in FIG. 5, depending on the image pickup apparatus, there may be a time (blanking) during which no exposure is performed.
If blanking is present, the brightness of the light emitting part at that time cannot be observed.
By repeatedly transmitting the same signal twice or more by the light emitting unit or by adding an error correction code, transmission loss due to blanking can be prevented.
In order to prevent the same signal from always being transmitted during blanking, the light emitting unit transmits a signal at a period that is relatively prime to the period at which the image is taken or a cycle shorter than the period at which the image is taken.
(Signal modulation method) When using visible light as the carrier, keep the moving average value of the brightness of the light emitting part constant when the time resolution of human vision (about 5 ms to 20 ms) is used as the window width. By making the light emitting part emit light, as shown in FIG. 6, the light emitting part of the transmitting device appears to humans to emit light with a uniform brightness, and at the same time, the receiving device observes the change in the brightness of the light emitting part. Can be done.
There is a modulation method shown in FIG. 7 as a modulation method for causing the light emitting part to emit light so as to keep the moving average value of the brightness of the light emitting part constant when the time resolution of human vision is the window width. When the modulated signal is 0, it emits light, when it is 1, it emits light, and if there is no bias in the transmission signal, the average value of the brightness of the light emitting portion is about 50% of the brightness at the time of light emission.
It is assumed that switching between light emission and non-light emission is sufficiently faster than the time resolution of human vision.
There is a modulation method shown in FIG. 8 as a modulation method for causing the light emitting part to emit light so as to keep the moving average value of the brightness of the light emitting part constant when the time resolution of human vision is the window width. When the modulated signal is 0, it emits light, when it is 1, it emits light, and if there is no bias in the transmission signal, the average value of the brightness of the light emitting portion is about 75% of the brightness at the time of light emission.
Compared with the modulation method of FIG. 7, the coding efficiency is 0.5, which is equivalent, but the average luminance can be increased.
There is a modulation method shown in FIG. 9 as a modulation method for causing the light emitting part to emit light so as to keep the moving average value of the brightness of the light emitting part constant when the time resolution of human vision is the window width. When the modulated signal is 0, it emits light, when it is 1, it emits light, and if there is no bias in the transmission signal, the average value of the brightness of the light emitting portion is about 87.5% of the brightness at the time of light emission.
Compared with the modulation methods shown in FIGS. 7 and 8, the coding efficiency is inferior to 0.375, but the average luminance can be kept high.
In the same manner below, it is possible to perform modulation that increases the average luminance in a trade-off with the coding efficiency.
There is a modulation method shown in FIG. 10 as a modulation method for causing the light emitting part to emit light so as to keep the moving average value of the brightness of the light emitting part constant when the time resolution of human vision is the window width.
When the modulated signal is 0, it emits light, when it is 1, it emits light, and if there is no bias in the transmission signal, the average value of the brightness of the light emitting portion is about 25% of the brightness at the time of light emission.
By periodically changing the modulation method in combination with the modulation method shown in FIG. 8, it is possible to make the light emitting portion appear to be blinking for a human being or an image pickup device having a long exposure time.
Similarly, by changing the modulation method, it is possible to make the light emitting unit appear to emit light while changing the brightness arbitrarily to a human being or an image pickup device having a long exposure time.
When visible light is used as the carrier wave, the light emitting part is made to emit light so as to periodically change the moving average value of the brightness of the light emitting part when the time resolution of human vision is taken as the window width, as shown in FIG. At the same time as the light emitting part of the transmitting device appears to be blinking or changing at an arbitrary rhythm to humans, the receiving device can observe the light emitting signal.
The same effect can be obtained by making the LED part of the LCD TV that uses the LED light source as the backlight emit light. In this case, at least, by reducing the contrast of the screen portion of the optical communication unit and bringing it closer to white, optical communication with a low error rate becomes possible. The communication speed can be further increased by making the entire surface or the screen part used for communication white.
When a television display or the like is used as the light emitting part, the moving average value of the brightness of the light emitting part when the time resolution of human vision is taken as the window width is adjusted so as to be the brightness of the image desired to be shown to humans. As in 12, the receiver can observe the emission signal at the same time that the human can see the normal television image.
By adjusting the moving average value of the brightness of the light emitting part when the window width is about the time per frame of the captured image to the value of the signal when the signal is transmitted for each imaged frame, as shown in FIG. By observing the light emitting state of the transmitter for each exposure line when shooting from a short distance, and observing the light emitting state of the transmitter for each imaging frame when shooting from a long distance, signals are sent at two different speeds. Propagation is possible.
When the image is taken from a short distance, the signal that can be received when the image is taken from a long distance can also be received.
FIG. 14 is a diagram showing how light emission is observed for each exposure time.
Since the brightness of the imaged pixel is proportional to the average brightness of the imaged object during the exposure time of the image sensor, the emission pattern 2217a is observed as it is like 2217b if the exposure time is short, and 2217c if the exposure time is long. , 2217d, 2217e are observed.
The 2217a is a modulation method in which the modulation method shown in FIG. 8 is repeatedly used in a fractal manner.
By using such a light emission pattern, it is possible to simultaneously transmit a large amount of information to a receiving device having an image pickup device having a short exposure time and a small amount of information to a receiving device having an image pickup device having a long exposure time.
The receiving device recognizes that 1 is received when the brightness of the pixel at the estimated position of the light emitting unit is above a certain level and 0 is received when the brightness is below a certain level over one line of the exposure line or a certain number of lines.
If 1 continues, it is indistinguishable from a normal light emitting part (not transmitting a signal and always shining), and if 0 continues, it is indistinguishable from the case where the light emitting part does not exist.
Therefore, the transmitting device may transmit different numbers when the same numbers are consecutive by a certain number.
Further, as shown in FIG. 15, the header portion including 1 and 0 and the body portion for transmitting the signal may be separately transmitted. In this case, the same number does not appear more than 5 times in a row.
If the light emitting part is not reflected in one exposure line, or if there is a blanking, the entire state of the light emitting part cannot be captured by the image pickup device of the receiving device.
Therefore, it is necessary to indicate which part of the signal the signal is.
As shown in FIG. 16, there is a method of transmitting the data unit and the address unit indicating the position of the data together.
In order to facilitate the reception of signals by the receiving device, the length of the light emitting pattern is the size of the data part and the address part combined so that the light emitting pattern is captured in one captured image by the receiving device. It is desirable to set it short enough.
As shown in FIG. 17, there is a method in which the transmitting device transmits the reference unit and the data unit, and the receiving device recognizes the position of the data from the difference from the time when the reference unit is received.
As shown in FIG. 18, the transmitting device transmits the reference unit, the address pattern unit, and the data unit, and the receiving device obtains the data of the data unit and the pattern of its position from the address pattern unit next to the reference unit, and the pattern thereof. And, there is a method of recognizing the position of data from the difference between the time when the reference unit is received and the time when each data is received.
By using multiple types of address patterns, not only data can be sent uniformly, but also important data and data to be processed first can be sent first, and the number of repetitions can be increased more than other data. can do.
If the light emitting part does not appear on all the exposure lines, or if there is a blanking, the entire state of the light emitting part cannot be captured by the image pickup device of the receiving device.
As shown in FIG. 19, by adding a header part, it is possible to detect a signal delimiter and detect an address part and a data part.
For the light emission pattern of the header part, a pattern that does not appear in the address part or the data part is used.
For example, when the modulation method shown in Table 2200.2a is used, the light emission pattern of the header portion can be set to "0011".
Further, when the pattern of the header portion is set to "11110011", the average brightness becomes equal to that of other portions, and flicker when viewed by the human eye can be suppressed. Since this header part has high redundancy, it is possible to superimpose information on it as well. For example, when the pattern of the header portion is "11100111", it is possible to indicate that the content to be communicated between the transmitting devices is being transmitted.
In order to facilitate the reception of signals by the receiving device, a light emitting pattern having a size that combines the data unit, the address unit, and the header unit so that the light emitting pattern is captured in one captured image by the receiving device. It is desirable to set the length of the.
In FIG. 20, the transmitting device determines the transmission order of information according to the priority.
For example, the number of transmissions is proportional to the priority.
Since the receiving device cannot continuously receive the signal when the light emitting portion of the transmitting device does not fill the image area of the receiving device or when there is blanking, the higher the frequency of transmission, the faster the receiving device is likely to be received.
FIG. 21 shows a pattern in which a plurality of nearby transmitters transmit information in synchronization with each other.
By transmitting common information at the same time by multiple transmitters, it can be regarded as one large transmitter, and the image pickup unit of the receiver can take a large image, so information can be received faster and from a greater distance. It becomes.
The transmitting device transmits individual information during a time period in which the light emitting unit of the nearby transmitting device emits uniform light (not transmitting a signal) so as not to be confused with the light emitting pattern of a nearby transmitting device.
The transmitting device may learn the light emitting pattern of the transmitting device existing nearby by receiving the light emitting pattern of the transmitting signal existing nearby by the light receiving unit, and determine its own light emitting pattern. Further, the transmitting device may determine its own light emitting pattern by the command of another transmitting device by receiving the light emitting pattern of the transmission signal existing nearby by the light receiving unit. Further, the transmitting device may determine the light emission pattern according to the instruction of the centralized control device.
(Detection of light emitting part) As a method of determining which part of the captured image the light emitting part is imaged, as shown in FIG. 22, the number of lines in which the light emitting part is imaged is counted in the direction perpendicular to the exposure line. There is a method in which the row in which the light emitting portion is imaged is set as the row in which the light emitting portion is present.
In the part near the edge of the light emitting part, the degree of light reception fluctuates, and it is easy to make a mistake in determining whether or not the light emitting part is imaged. Extract the signal from the result.
As a method of determining which part of the captured image the light emitting part is imaged, as shown in FIG. 23, the midpoint of the part where the light emitting part is imaged is obtained for each exposure line, and an approximate line (straight line) connecting the obtained points is obtained. , Or there is a method of presuming that the light emitting part exists on the quadratic curve).
As shown in FIG. 24, the estimated position of the light emitting unit in the previous frame may be set as a prior probability, and the estimated position of the light emitting unit may be updated from the information of the current frame.
The current estimated position of the light emitting unit may be updated from the values of the 9-axis sensor and the gyro sensor during this period.
Explaining FIG. 25, when the light emitting unit 2212b is imaged when the imaging range is 2212a, images such as the captured images 2212c, 2212d, and 2212e are captured.
By summing the light emitting parts of the captured images 2212c, 2212d, and 2212e, a composite image 2212f is obtained, and the position of the light emitting part in the captured image can be specified.
The receiving device detects on / off of the light emission of the light emitting unit from the position of the specified light emitting unit.
When the key method shown in FIG. 8 is used, the emission probability is 0.75, so when the sum of n images is taken, the probability that the light emitting part in the composite image 2212f appears to emit light is 1-0.25.<sup>n</sup>Is. Here, for example, when n = 3, this probability is about 0.984.
It should be noted that the accuracy is higher when the posture of the image pickup unit is estimated from the sensor values of the gyro sensor and the 9-axis sensor, the image pickup direction is compensated, and then the image is synthesized. However, when the number of composite images is small, the imaging time is short, so that there is little adverse effect even if the imaging direction is not compensated.
FIG. 26 is a diagram showing a case where the receiving device captures a plurality of light emitting units.
When a plurality of light emitting units emit the same signal, one transmission signal is acquired from both light emission patterns. When a plurality of light emitting units emit different signals, different transmission signals are acquired from different light emission patterns.
If the data values of the same address of the transmission signal are different, different signals are transmitted. Depending on the pattern of the header portion of the transmission signal, it may be determined that the same or different signal as the nearby transmission device is being transmitted.
When the light emitting units are substantially adjacent to each other, it may be considered that the same signal is transmitted.
FIG. 27 shows a timeline of the transmission signal at this time and an image of the light emitting unit.
(Transmission of signal by position pattern) In FIG. 28, the light emitting units 2216a, 2216c, and 2216e emit light uniformly, and the light emitting units 2216b, 2216d, and 2216f emit signals by the light emitting pattern. It should be noted that the light emitting units 2216b, 2216d, and 2216f may simply emit light so as to look like a striped pattern if the receiving device takes an image for each exposure line.
In FIG. 28, the light emitting units 2216a to 2216f may be light emitting units of the same transmitter or separate transmitters.
The transmitting device expresses a signal to be transmitted by a pattern (position pattern) of the position of the light emitting unit that is transmitting the signal and the position of the light emitting unit that is not transmitting the signal.
In FIG. 28, since there are 6 transmitters, it is possible to transmit a signal of 2 to the 6th power = 64 values. Position patterns that look the same when viewed from different directions should not be used, but the above patterns can be identified by specifying the imaging direction with a 9-axis sensor of the receiving device or the like. It should be noted that more signals may be transmitted by changing which light emitting unit is transmitting the signal depending on the time.
The transmission device can also transmit a signal according to a position pattern in a part of the time zone and transmit a signal in a light emission pattern in another time zone. For example, in some time zones, all light emitting units can be synchronized and the ID and position information of the transmitting device can be transmitted in a light emitting pattern.
Since the arrangement pattern of the light emitting unit exists almost infinitely, it is difficult for the receiving device to store all the position patterns in advance.
Therefore, the receiving device uses the ID and position information of the transmitting device transmitted by the transmitting device according to the light emission pattern, the estimated position of the receiving device by the wireless base station, and the position information of the receiving device estimated by the GPS, the gyro sensor, or the 9-axis sensor. With the key, a list of position patterns existing in the vicinity is acquired from the server, and the position pattern is analyzed based on the list.
According to this method, the signal expressed by the position pattern does not have to be unique in the world, it is sufficient if the same position pattern does not exist near (a radius of several meters to 300 meters), and the transmitter has few light emitting parts. Can solve the problem that the number of position patterns that can be expressed is small.
The size, shape, and position information of the light emitting unit can be acquired from the server, and the position of the receiving device can be estimated from these information, the size and shape of the captured position pattern, and the lens characteristics of the imaging unit.
(Receiver) As shown in Fig. 29, the communication devices that mainly receive are mobile phones, digital still cameras, digital video cameras, head-mounted displays, robots (for cleaning, nursing care, industrial use, etc.) and surveillance cameras. Etc. are conceivable. However, the receiving device is not limited to these.
The receiving device is a communication device that mainly receives signals, and may transmit signals by the method of the present embodiment or other methods.
(Transmitting device) As shown in Fig. 30, lighting (for home, store, office, underground shopping mall, street, etc.), flashlight, home appliances, robots, and other electronic devices are mainly used as communication devices for transmission. Conceivable. However, the transmitting device is not limited to these.
The transmitting device is a communication device that mainly transmits signals, and may receive signals by the method of the present embodiment or other methods.
As shown in FIG. 31, it is desirable that the light emitting unit has a high-speed light-emitting / non-light-emitting switching, such as an LED illumination or a liquid crystal display using an LED backlight. However, the light emitting unit is not limited to these.
In addition to the above, the light emitting portion may be illuminated by a fluorescent lamp, an incandescent lamp, a mercury lamp, an organic EL display, or the like.
Since the transmission efficiency is higher when the light emitting unit is imaged larger, the transmission device may be provided with a plurality of light emitting units that emit light in synchronization as shown in FIG. 32. Further, since the transmission efficiency becomes higher as the image is larger in the direction perpendicular to the exposure line of the image sensor, the light emitting units may be installed side by side in a row. Further, when the receiving device is naturally held, the light emitting unit may be arranged perpendicular to the exposure line. Further, when it is expected that the image is taken from a plurality of directions, the light emitting portion may be arranged in a cross shape as shown in FIG. 33. Further, when it is expected that the image is taken from a plurality of directions, a circular light emitting unit may be used or the light emitting unit may be arranged in a circular shape as shown in FIG. 34. Further, since the transmission efficiency is higher when a large image is taken, the transmitting device may cover the light emitting portion with a diffuser plate as shown in FIG. 35.
As shown in FIG. 36, the light emitting units that transmit different signals are arranged at a distance so as not to be imaged at the same time. Further, as shown in FIG. 37, the light emitting unit that transmits different signals is provided with a light emitting unit that does not transmit signals in between so that the images are not simultaneously imaged.
(Structure of light emitting portion) FIG. 38 is a diagram showing a desirable structure of the light emitting portion.
By making the light emitting part and the surrounding material low in reflectance as in 2311a, it is easy for the receiving device to recognize the state of light emission even when the light is shining around the light emitting part. Can be done. Further, by providing a bulk that blocks external light as in 2311b, it is possible to facilitate the receiving device to recognize the state of light emission when the light is applied to the periphery of the light emitting portion. Further, by providing the light emitting portion in the recessed portion as in 2311c, it is possible to facilitate the receiving device to recognize the state of light emission when the light is applied to the periphery of the light emitting portion.
(Signal carrier wave) As the light (electromagnetic wave) that carries the signal, the light (electromagnetic wave) in the frequency band of the near infrared band to the visible light band to the near ultraviolet band shown in FIG. 39, which can be received by the receiving device, is used.
(Image pickup unit) In FIG. 40, the image pickup unit of the receiving device detects the light emitting unit 2310b that emits pattern light within the image pickup range 2310a.
Instead of using another exposure line, the image pickup control unit repeatedly uses the exposure line 2310c at the center position of the light emitting unit to acquire the captured image 2310d.
The captured image 2310d is an image of the same place with different exposure times. By scanning the pixel showing the light emitting part of the captured image 2310d in the direction perpendicular to the exposure line, the light emitting pattern of the light emitting part can be observed.
By this method, even if the light emitting portion is present only in a part of the captured image, the change in brightness of the light emitting portion can be observed for a longer time. Therefore, it is possible to read a signal even when a small light emitting unit or a light emitting unit is imaged from a distance.
When there is no blanking, by this method, if the light emitting portion is captured even in a part of the image pickup apparatus, all the changes in the brightness of the light emitting portion can be observed.
When the time for exposing one line is longer than the time for starting to expose the next line, the same effect can be obtained by taking an image using a plurality of exposure lines at the center of the light emitting portion.
If control is possible on a pixel-by-pixel basis, imaging is performed using only the point closest to the center of the light emitting unit or a plurality of points in the vicinity thereof. At this time, by shifting the exposure start time of each pixel, it is possible to detect the light emitting state of the light emitting unit at a finer cycle.
It should be noted that, while mainly using the 2310c exposure line, in rare cases, another exposure line is used for imaging, and the captured image is combined. Images (videos) similar to can be combined. By displaying this composite image to the user, the user can operate the receiving device or use it for image stabilization.
For image stabilization, sensor values such as a gyro sensor and a 9-axis sensor can be used, or an image captured by an image pickup device different from the image pickup device that is capturing the light emitting portion can be used. ..
It is desirable that the portion closer to the center of the light emitting portion be the exposure line or the exposure pixel because the light emitting portion is less likely to deviate from the exposure line or the exposure pixel when the camera shakes when the light emitting portion is closer to the center than the edge of the light emitting portion.
Since the edge portion of the light emitting portion has low emission brightness, it is desirable to use an exposure line or an exposure pixel as a portion having high brightness as far as possible from the periphery of the light emitting portion.
(Estimation of the position of the receiving device) In FIG. 41, the transmitting device transmits its own position information, the size of the light emitting device, the shape of the light emitting device, and the ID of the transmitting device. Here, the position information includes the latitude and longitude of the central portion of the light emitting device, the altitude, the height from the floor surface, and the like.
The receiving device estimates the imaging direction from the information obtained from the 9-axis sensor and the gyro sensor. The receiving device estimates the distance from the receiving device to the light emitting device from the size and shape of the light emitting device transmitted from the transmitting device, the size and shape of the light emitting device in the captured image, and the information of the imaging device. Here, the information of the image pickup device includes the focal length of the lens, the distortion of the lens, the size of the image pickup element, the distance between the lens and the image pickup element, the size of the reference size object in the image pickup image, and the image pickup device. A comparison table of the distance to the imaged object is included.
Further, the receiving device estimates the position information of the receiving device from the information transmitted from the transmitting device, the imaging direction, and the distance from the receiving device to the light emitting device.
In FIG. 42, the transmitting device transmits the position information in which it is installed, the size of the light emitting unit, the shape of the light emitting unit, and the ID of the transmitting device. Here, the position information includes the latitude and longitude of the central portion of the light emitting portion, the altitude, the height from the floor surface, and the like.
The receiving device estimates the imaging direction from the information obtained from the 9-axis sensor and the gyro sensor. The receiving device estimates the distance from the receiving device to the light emitting unit from the size and shape of the light emitting unit transmitted from the transmitting device, the size and shape of the light emitting unit in the captured image, and the information of the imaging device. Here, the information of the image pickup device includes the focal length of the lens, the distortion of the lens, the size of the image pickup element, the distance between the lens and the image pickup element, the size of the reference size object in the image pickup image, and the image pickup device. A comparison table of the distance to the imaged object is included.
Further, the receiving device estimates the position information of the receiving device from the information transmitted from the transmitting device, the imaging direction, and the distance from the receiving device to the light emitting unit. The receiving device estimates the moving direction and the moving distance from the information obtained from the 9-axis sensor and the gyro sensor. The receiving device estimates the position information of the receiving device by using the position information estimated at a plurality of points and the positional relationship between the points estimated from the moving direction and the moving distance.
For example, a point<maths num="1"><img file="JP6970146B2_D0001.tif" /></maths>Random field of the position information of the receiving device estimated in<maths num="2"><img file="JP6970146B2_D0002.tif" /></maths>point<maths num="3"><img file="JP6970146B2_D0003.tif" /></maths>From the point<maths num="4"><img file="JP6970146B2_D0004.tif" /></maths>Random field of movement direction and movement distance estimated when moving to<maths num="5"><img file="JP6970146B2_D0005.tif" /></maths>Then, the random field of the position information finally estimated is<maths num="6"><img file="JP6970146B2_D0006.tif" /></maths>Can be calculated.
Further, in FIG. 42, the transmitting device may transmit the position information in which it is installed and the ID of the transmitting device. Here, the position information includes the latitude and longitude of the central portion of the light emitting device, the altitude, the height from the floor surface, and the like.
In this case, the receiving device estimates the imaging direction from the information obtained from the 9-axis sensor and the gyro sensor. The receiving device estimates the position information of the receiving device by a three-point survey method.
In FIG. 43, the transmitting device transmits the ID of the transmitting device.
The receiving device receives the ID of the transmitting device, and obtains the position information of the transmitting device installed, the size of the light emitting device, the shape of the light emitting device, and the like from the Internet. Here, the position information includes the latitude and longitude of the central portion of the light emitting device, the altitude, the height from the floor surface, and the like.
The receiving device estimates the imaging direction from the information obtained from the 9-axis sensor and the gyro sensor. The receiving device estimates the distance from the receiving device to the light emitting device from the size and shape of the light emitting device transmitted from the transmitting device, the size and shape of the light emitting device in the captured image, and the information of the imaging device. Here, the information of the image pickup device includes the focal length of the lens, the distortion of the lens, the size of the image pickup element, the distance between the lens and the image pickup element, the size of the reference size object in the image pickup image, and the image pickup device. A comparison table of the distance to the imaged object is included.
Further, the receiving device estimates the position information of the receiving device from the information obtained from the Internet, the imaging direction, and the distance from the receiving device to the light emitting device.
In FIG. 44, the transmitting device transmits its own installed position information and the ID of the transmitting device. Here, the position information includes the latitude and longitude of the central portion of the light emitting device, the altitude, the height from the floor surface, and the like.
The receiving device estimates the imaging direction from the information obtained from the 9-axis sensor and the gyro sensor. The receiving device estimates the position information of the receiving device by a method of triangulation.
In FIG. 45, the transmitting device transmits its own location information and the ID of the transmitting device. The position information includes the latitude and longitude of the central portion of the light emitting device, the altitude, the height from the floor surface, and the like.
The receiving device estimates the imaging direction from the information obtained from the 9-axis gyro sensor. The receiving device estimates the position information of the receiving device by a method of triangulation. In addition, the receiving device estimates the attitude change and movement of the receiving device from the gyro sensor and the 9-axis sensor. The receiving device may simultaneously adjust the zero point of the 9-axis sensor and calibrate it.
(Setting of transmission information) In FIG. 46, the receiving device 2606c captures the light emission pattern of the transmitting device 2606b, acquires the transmitted signal, and estimates the position of the receiving device.
While the receiver 2606c is being moved, it estimates the distance and direction of movement from changes in the captured image and sensor values of the 9-axis sensor and gyro sensor.
The receiving device takes an image of the light receiving unit of the transmitting device 2606a, estimates the center position of the light emitting unit, and transmits the position to the transmitting device.
Since the position estimation of the light emitting unit requires information on the size of the light emitting device, it is better for the transmitting device to transmit the size information of the light emitting unit even if a part of the information to be transmitted is missing. desirable. If the size of the light emitting unit is unknown, the height of the ceiling is estimated from the distance between the transmitter 2606b and the receiver 2606c used for position estimation in the receiver, and the result is used to estimate the height of the ceiling. Estimate the distance between and the receiver 2606c.
Transmission includes transmission by a light emission pattern, transmission by a sound pattern, and transmission by wireless radio waves. The light emission pattern of the transmission device and its time may be stored and later transmitted to the transmission device or the centralized control device.
The transmission device and the centralized control device identify the transmission device imaged by the reception device from the light emission pattern and the time, and store the position information in the transmission device.
In FIG. 47, a point for setting a position is specified by designating one point of the transmitting device as a point in the image captured by the receiving device.
The receiving device calculates the positional relationship from the position setting point to the center of the light emitting unit of the transmitting device, and transmits the position obtained by adding the positional relationship to the set position to the transmitting device.
In FIG. 48, the receiving device receives the transmitted signal by imaging the transmitting device. Communicates with servers and electronic devices based on the received signals.
For example, the receiving device acquires information on the transmitting device, the position / size of the transmitting device, service information related to the position, and the like from the server using the ID of the transmitting device included in the signal as a key.
Further, for example, the receiving device estimates the position of the receiving device from the position of the transmitting device included in the signal, and acquires map information, service information related to the position, and the like from the server.
Further, for example, the receiving device acquires the modulation method of the nearby transmitting device from the server using the current rough position as a key.
Further, for example, the receiving device uses the ID of the transmitting device included in the signal as a key to send the position information of the receiving device and the transmitting device, information on the vicinity, and information on the processing performed by the receiving device in the vicinity to the server. To register with.
Further, for example, the receiving device operates the electronic device by using the ID of the transmitting device included in the signal as a key.
(Block diagram of receiving device) FIG. 49 is a configuration diagram showing a receiving device. The receiving device is composed of all of them or a part including an imaging unit and a signal analysis unit. The blocks having the same name in FIG. 49 may be the same block or may be different blocks.
The receiver device 2400af in the narrow sense is provided in smartphones, digital cameras, and the like. The input unit 2400h is composed of all or a part of the user operation input unit 2400i, the illuminance sensor 2400j, the microphone 2400k, the timekeeping unit 2400n, the position estimation unit 2400m, and the communication unit 2400p.
The image pickup unit 2400a is composed of all or a part of the lens 2400b, the image pickup element 2400c, the focus control unit 2400d, the image pickup control unit 2400e, the signal detection unit 2400f, and the image pickup information storage unit 2400g. The image pickup unit 2400a is used for user operations, changes in illuminance, sound and voice patterns, a specific time, a move of the receiver to a specific location, and other information via the communication unit. The imaging is started by being instructed by the device of.
The focus control unit 2400d performs control such as focusing on the light emitting unit 2400ae of the transmitting device and focusing so as to blur the light emitting unit 2400ae of the transmitting device so as to be large.
The exposure control unit 2400ak sets the exposure time and exposure gain.
The image pickup control unit 2400e limits the image pickup position to a specific pixel.
The signal detection unit 2400f detects pixels including the light emitting unit 2400ae of the transmission device and pixels including signal transmission by light emission from the captured image.
The image pickup information storage unit 2400g stores the control information of the focus control unit 2400d, the control information of the image pickup control unit 2400e, and the information detected by the signal detection unit 2400f. When there are a plurality of image pickup devices, image pickup may be performed at the same time, and one of them may be used for estimating the position and orientation of the receiving device.
The light emission control unit 2400ad transmits a signal by controlling the light emission pattern of the light emission unit 2400ae by input from the input unit 2400h. The light emitting control unit 2400ad acquires and records the time when the light emitting unit 2400ae emits light from the time measuring unit 2400ac.
The captured image storage unit 2400w stores the image captured by the image pickup unit 2400a.
The signal analysis unit 2400y utilizes the difference in the exposure time for each line of the image sensor to obtain the light emission pattern of the light emitting unit 2400ae of the image pickup device based on the modulation method stored in the modulation method storage unit 2400af. Get the transmitted signal.
The received signal storage unit 2400z stores the signal analyzed by the signal analysis unit 2400y.
The sensor unit 2400q is composed of all or part of the GPS2400r, the magnetic sensor 2400t, the acceleration sensor 2400s, and the gyro sensor 2400u. The magnetic sensor 2400t and the acceleration sensor 2400s may be 9-axis sensors, respectively.
The position estimation unit estimates the position and orientation of the receiving device from the information from the sensor unit, the captured image, and the received signal.
The calculation unit 2400aa displays the received signal, the estimated position of the receiving device, and the information obtained from the network 2400ah based on these (information related to the map and location, information related to the transmitting device, etc.) on the display unit 2400ab. Let me.
The calculation unit 2400aa controls the transmission device based on the information input to the input unit 2400h from the received signal and the estimated position of the reception device.
When the communication unit 2400ag uses a peer-to-peer connection method (bluetooth, etc.), the terminals communicate with each other without going through the network 2400ah.
The electronic device 2400aj is controlled by the receiving device.
The server 2400ai stores the information of the transmitting device, the position of the transmitting device, and the information related to the position of the transmitting device in association with the ID of the transmitting device.
The server 2400ai stores the modulation method of the transmitter in association with the position.
(Block diagram of transmitter) FIG. 50 is a block diagram constituting the transmitter.
The transmission device is composed of the entire configuration diagram or a part including a light emitting unit, a transmission signal storage unit, a modulation method storage unit, and a calculation unit.
The transmission device 2401ab in the narrow sense is provided in lamps, electronic devices, and robots.
The lighting control switch 2401n is a switch for switching lighting on and off.
The diffuser plate 2401p is a member attached near the light emitting unit 2401q in order to diffuse the light of the light emitting unit 2401q.
The light emitting unit 2401q turns on and off at a speed at which a light emission pattern is detected for each line by utilizing the difference in the exposure time for each line of the image sensor of the receiver of FIG. 49.
The light emitting unit 2401q is composed of a light source such as an LED or a fluorescent lamp that can be turned on and off at high speed.
The light emitting control unit 2401r controls lighting and extinguishing of the light emitting unit 2401q.
The light receiving unit 2401s is composed of a light receiving element and an image pickup element. The light receiving unit 2401s converts the intensity of the received light into an electric signal. An imaging unit may be used instead of the light receiving unit 2401s.
The signal analysis unit 2401t acquires a signal from the pattern of light received by the light receiving unit 2401s.
The calculation unit 2401u converts the transmission signal stored in the transmission signal storage unit 2401d into a light emission pattern according to the modulation method stored in the modulation method storage unit 2401e. The arithmetic unit 2401u controls communication by editing the information of the storage unit 2401a and controlling the light emission control unit 2401r based on the signal obtained from the signal analysis unit 2401t. The arithmetic unit 2401u controls communication by editing the information of the storage unit 2401a and controlling the light emission control unit 2401r based on the signal from the mounting unit 2401w. The arithmetic unit 2401u controls the light emission control unit 2401r, such as editing the information of the storage unit 2401a, based on the signal from the communication unit 2401v.
Further, the arithmetic unit 2401u edits the information of the storage unit 2401b of the attachment device 2401h. The calculation unit 2401u copies the information of the storage unit 2401b of the attachment device 2401h to the storage unit 2401a.
The calculation unit 2401u controls the light emission control unit 2401r at a predetermined time. The arithmetic unit 2401u controls the electronic device 2401zz via the network 2401aa.
The storage unit 2401a is composed of all or a part of the transmission signal storage unit 2401d, the shape storage unit 2401f, the modulation method storage unit 2401e, and the device state storage unit 2401g.
The transmission signal storage unit 2401d stores the signal transmitted from the light emitting unit 2401q.
The modulation method storage unit 2401e stores a modulation method for converting a transmission signal into a light emission pattern.
The shape storage unit 2401f stores the shapes of the transmitter and the light emitting unit 2401q.
The device state storage unit 2401g stores the state of the transmitter.
The mounting portion 2401w is composed of mounting brackets and a power supply port.
The storage unit 2401b of the mounting device 2401h stores information stored in the storage unit 2401a. The storage unit 2401a is not provided, and the storage unit 2401b of the mounting device 2401h or the storage unit 2401c of the centralized control device 2401m may be used.
When the communication unit 2401v uses a peer-to-peer connection method (bluetooth, etc.), the communication unit 2401v communicates with each other without going through the network 2401aa.
The server 2401y stores the information of the transmitting device, the position of the transmitting device, and the information related to the position of the transmitting device in association with the ID of the transmitting device. Further, the server 2401y stores the modulation method of the transmission device in association with the position.
(Reception procedure) Explaining FIG. 51, in step 2800a, it is confirmed whether or not there are multiple image pickup devices in the receiver device. If No, proceed to step 2800b, select the imaging device to be used, and proceed to step 2800c. On the other hand, if Yes, proceed to step 2800c.
In step 2800c, the exposure time (= shutter speed) is set (it is desirable that the exposure time is short).
Next, in step 2800d, the exposure gain is set.
Next, in step 2800e, an image is taken.
Next, in step 2800f, for each exposure line, a portion where a certain number or more of pixels whose brightness exceeds a certain threshold value continues is determined, and the center position of that portion is obtained.
Then, in step 2800g, a first-order or second-order approximation line connecting the above center positions is calculated.
Next, in step 2800h, the luminance value of the pixel on the approximate line of each exposure line is set as the signal value of each exposure line.
Next, in step 2800i, the assigned time per exposure line is calculated from the imaging information composed of the imaging frame rate, the resolution, the blanking time, and the like.
Next, in step 2800j, if the blanking time is below a certain level, the exposure line next to the last exposure line of a certain imaging frame is regarded as the first exposure line of the next frame. Otherwise, there are unobservable exposure lines between the last exposure line in one imaging frame and the first exposure line in the next frame, which is the number of blanking times divided by the time in charge per exposure line. Considered to exist.
Next, in step 2800k, the reference position pattern and the address pattern are read from the decoded information.
Next, in step 2800 m, a pattern indicating the reference position of the signal is detected from the signal value of each exposure line.
Next, in step 2800n, the data part and the address part are calculated based on the detected reference position.
Then, in step 2800p, a transmission signal is obtained.
(Procedure for self-position estimation) Explaining FIG. 52, first, in step 2801a, the position recognized as the current position of the receiving device or the probability map of the current position is used as the prior information of the self-position.
Next, in step 2801b, the imaging unit of the receiving device is directed to the light emitting unit of the transmitting device.
Next, in step 2801c, the orientation and elevation angle at which the image pickup device is directed are calculated from the sensor values of the 9-axis sensor and the gyro sensor.
Next, in step 2801d, the light emission pattern is imaged and the transmission signal is acquired.
Next, in step 2801e, the distance between the image pickup device and the light emitting section is calculated from the information on the size and shape of the light emitting section included in the transmission signal, the size of the light emitting section captured, and the magnification of the image pickup of the image pickup device. do.
Next, in step 2801f, the relative angle between the direction from the image pickup unit to the light emitting portion and the normal of the image pickup surface is calculated from the position of the light emitting portion in the captured image and the lens characteristics.
Next, in step 2801g, the relative positional relationship between the image pickup device and the light emitting unit is calculated from the numerical values calculated so far.
Next, in step 2801h, the position of the receiving device is calculated from the position of the light emitting unit included in the transmission signal and the relative positional relationship between the image pickup device and the light emitting unit. When a plurality of transmitting devices can be observed, the position of the receiving device can be calculated with high accuracy by calculating the coordinates of the imaging device from the signals included in each transmitting device. If a plurality of transmitters can be observed, a triangulation method can be used.
Next, in step 2801i, the probability map of the current position or the current position of the receiving device is updated from the prior information of the self-position and the calculation result of the position of the receiving device.
Next, in step 2801j, the image pickup device is moved.
Next, in step 2801k, the direction and distance of movement are calculated from the sensor values of the 9-axis sensor and the gyro sensor.
Next, in step 2801m, the direction and distance of movement are calculated from the captured image and the posture of the imaging device, and the process returns to step 2801a.
(Procedure 1 of transmission control) Explaining FIG. 53, first, in step 2802a, the user presses a button.
Next, in step 2802b, the light emitting unit is made to emit light. The signal may be represented by a light emission pattern.
Next, in step 2802c, the light emission start time, the end time, and the time when a specific pattern is transmitted are recorded.
Next, in step 2802d, an image is taken by the image pickup device.
Next, in step 2802e, the light emission pattern of the transmission device existing in the captured image is imaged, and the transmitted signal is acquired. The emission pattern may be analyzed synchronously using the recorded time. And it ends.
(Procedure 2 of transmission control) Explaining FIG. 54, first, in step 2803a, light is received by the light receiving device or image is taken by the image receiving device.
Then, in step 2803b, check if it was a specific pattern.
If No, return to step 2803a. On the other hand, if Yes, the process proceeds to step 2803c, and the start time, the end time, and the time when a specific pattern appears when the reception pattern is received or imaged are recorded.
Next, in step 2803d, the transmission signal is read from the storage unit and converted into a light emission pattern.
Next, in step 2803e, the light emitting unit is made to emit light according to the light emission pattern, and the process ends. It should be noted that the light may be emitted after a certain period of time has elapsed from the recorded time, and the process may be terminated.
(Procedure 3 of transmission control) Explaining Fig. 55, first, in step 2804a, light is received by the light receiving device, and the received light energy is converted into electricity and stored.
Next, in step 2804b, it is confirmed whether or not the stored energy has exceeded a certain level.
If No, return to step 2804a. On the other hand, if Yes, the process proceeds to step 2804c, the received light is analyzed, and the time when a specific pattern appears is recorded.
Next, in step 2804d, the transmission signal is read from the storage unit and converted into a light emission pattern.
Next, in step 2804e, the light emitting unit is made to emit light according to the above light emission pattern, and the process ends. It should be noted that the light may be emitted after a certain period of time has elapsed from the recorded time, and the process may be terminated.
(Provision of information in the station yard) Fig. 56 is a diagram explaining the situation of receiving information in the station yard.
The receiving device 2700a receives the information transmitted by the lighting device by taking an image of the lighting installed in the station facility and reading the light emission pattern and the position pattern.
The receiving device 2700a acquires lighting and facility information from the server based on the received information, and further estimates the current position of the receiving device 2700a from the size and shape of the imaged lighting.
For example, the receiving device 2700a displays information obtained based on the facility ID and location information (2700b). The receiving device 2700a downloads a map of the facility based on the facility ID, and navigates to the boarding place from the ticket information purchased by the user (2700c).
Although FIG. 56 shows an example at a railway station, the same applies to facilities such as airports, ports, and bus stops.
(Ride service) FIG. 57 is a diagram showing a state of use in a vehicle.
The receiving device 2704a owned by the passenger and the receiving device 2704b owned by the salesperson receive the signal transmitted by the lighting 2704e and estimate their current position.
In addition, each receiving device may acquire the information necessary for self-position estimation from the lighting 2704e, or may acquire the information transmitted from the lighting 2704e from the server as a key, and may acquire the information from the server at the boarding station or the ticket gate. It may be acquired in advance based on location information or the like.
The receiving device 2704a may recognize that the current position is in the vehicle from the boarding time information and the current time of the ticket purchased by the user (passenger), and may download the information associated with the vehicle. ..
Each receiving device notifies the server of its current position. The receiving device 2704a notifies the server of the user (passenger) ID, the ID of the receiving device, and the information of the ticket purchased by the user (passenger). Make sure you are a person with reserved seat rights.
By displaying the current position of the salesperson on the receiving device 2704a, the user (passenger) can consider the purchase timing of the in-vehicle sales.
When a passenger places an order for in-vehicle sales via the receiving device 2704a, the receiving device 2704a notifies the salesperson's receiving device 2704b or the server of its position, order details, and billing information. The salesperson's receiver 2704b displays a map_2704d showing the location of the orderer.
Passengers can also purchase reserved seat tickets and transit tickets via the receiving device 2704a.
The receiving device 2704a displays the vacant seat information 2704c. The receiving device 2704a notifies the server of the boarding section information of the ticket purchased by the user (passenger) and the purchase information and the billing information of the reserved seat ticket and the transit ticket from its current position.
Although FIG. 57 shows an example in a railway, the same applies to vehicles such as airplanes, ships, and bus stops.
(In-store service) FIG. 58 is a diagram showing a state of use in the store.
The receiving devices 2707b, 2707c, and 2707d receive the signal transmitted by the illumination 2707a, estimate their current position, and notify the server.
In addition, each receiving device may acquire the information necessary for self-position estimation and the server address from the illumination 2707a, or may acquire the information transmitted from the illumination 2707a from another server as a key. , May be obtained from the accounting system.
The accounting system associates the accounting information with the receiver 2707d, displays the current position of the receiver 2707d (2707c), and delivers the ordered goods.
The receiving device 2707b displays the product information based on the information transmitted from the lighting 2707a. When the purchaser places an order from the displayed product information, the receiving device 2707b notifies the server of the product information, the billing information, and the current location.
In this way, the seller can deliver the ordered product based on the position information of the receiving device 2707b, and the purchaser can purchase the product while sitting at the seat.
(Establishment of wireless connection) FIG. 59 is a diagram showing a situation in which a wireless connection certificate is communicated and a wireless connection is established.
The electronic device (digital camera) 2701b operates as an access point for wireless connection, and transmits an ID and a password as a light emission pattern as information necessary for the connection.
The electronic device (smartphone) 2701a acquires transmission information from the light emission pattern and establishes a wireless connection.
Although a wireless connection is used here, the connection to be established may be a wired connection network.
Communication between the two electronic devices may be performed via the third electronic device.
(Adjustment of communication range) FIG. 60 is a diagram showing a communication range according to a light emission pattern and a position pattern.
In the communication method using radio waves, it is difficult to limit the communication range because the radio waves reach the adjacent rooms separated by the wall.
On the other hand, in communication using a light emission pattern or a position pattern, since visible light and wavelengths in the peripheral region thereof are used, it is easy to divide the communication range by using a shield. In addition, when visible light is used, there is an advantage that the communication range can be confirmed by the human eye.
(Indoor use) FIG. 61 is a diagram showing a state of indoor use such as an underground mall.
The receiving device 2706a receives the signal transmitted by the illumination 2706b and estimates its current position. In addition, the receiving device 2706a displays the current position on a map to provide directions, and displays information on nearby stores.
In an emergency, by transmitting disaster information and evacuation information from the lighting 2706b, communication is congested, the communication base station breaks down, or the radio wave from the communication base station does not reach. Can also obtain this information. This is effective for hearing-impaired people who miss the emergency broadcast or cannot hear the emergency broadcast.
(Outdoor use) FIG. 62 is a diagram showing a state of outdoor use such as a street.
The receiving device 2705a receives the signal transmitted by the streetlight lighting 2705b and estimates its current position. In addition, the receiving device 2705a displays the current position on a map to provide directions, and displays information on nearby stores.
In an emergency, by transmitting disaster information and evacuation information from the lighting 2705b, communication is congested, the communication base station breaks down, or the radio wave from the communication base station does not reach. Can also obtain this information.
In addition, it is possible to help prevent accidents by displaying the movements of other vehicles and pedestrians on a map and notifying the user that there are vehicles and pedestrians approaching.
(Instruction of directions) FIG. 63 is a diagram showing a state of instruction of directions.
Using the information transmitted from the transmitters 2703a, 2703b, and 2703c, the receiver 2703e can download a map of the vicinity and estimate its position with an accuracy of 1 cm to several tens of centimeters.
By knowing the exact position of the receiver 2703e, the wheelchair 2703d can be driven automatically and visually impaired people can pass safely.
(Use of Multiple Imaging Devices) The receiving device of FIG. 64 includes an in-camera 2710a, a touch panel 2710b, a button 2710c, an out-camera 2710d, and a flash 2710e.
When the transmitting device is imaged by the out-camera, camera shake can be corrected by estimating the movement and posture of the receiving device from the image captured by the in-camera.
By receiving signals from other transmitting devices with the in-camera, it is possible to receive signals from a plurality of devices at the same time and improve the accuracy of self-position estimation of the receiving device.
(Transmitting device autonomous control) In FIG. 65, the transmitting device 1 receives the light emitted from the light emitting unit of the transmitting device 2 by the light receiving unit, and acquires the signal transmitted by the transmitting device 2 and the transmission timing.
If the signal to be transmitted is not stored in the storage unit of the transmission device 1, the signal is transmitted by emitting light in the same pattern in synchronization with the light emission of the transmission device 2.
On the other hand, when the signal to be transmitted is stored in the storage unit of the transmission device 1, the intersection with the transmission signal of the transmission device 2 emits light in the same pattern in synchronization with the light emission of the transmission device 2 to transmit the signal. .. The non-common part is transmitted at the time when the transmitting device 2 is not transmitting the signal. If there is no time when the transmitting device 2 does not transmit the signal, a period is appropriately set, and the non-common part is transmitted according to the period. In this case, the transmitting device 2 receives the light emitted from the transmitting device 1 at the light receiving unit, detects that different signals are transmitted at the same time, and is not common at the time when the transmitting device 1 does not transmit the signal. Send the signal of the part.
The CSMA / CD (Carrier Sense Multiple Access with Collision Detection) method is used to avoid collision of signal transmission due to light emission.
The transmitting device 1 emits light from the light emitting unit using its own information as a light emitting pattern.
The transmitting device 2 acquires the information of the transmitting device 1 from the light receiving unit.
The transmitting device creates an arrangement map of the transmitting device by exchanging information with each other among the transmitting devices capable of communicating with each other. In addition, the transmission device seeks an optimum light emission pattern as a whole so that signal transmissions due to light emission do not collide. Further, the transmitting device acquires the information obtained by the other transmitting devices by communication between the transmitting devices.
(Setting of transmission information) In FIG. 66, the transmission device is stored in the storage unit of the mounting device when the transmission device is mounted on the mounting device or when the information stored in the storage unit of the mounting device is changed. The stored information is stored in the storage unit of the transmission device. The information stored in the storage unit of the mounting device or the transmitting device includes a transmission signal and a transmission timing.
When the information stored in the storage unit is changed, the transmitting device stores the information in the storage unit of the mounting device. Edit the information in the storage unit of the mounting device and the storage unit of the transmission device from the centralized control device or switchboard. Power line communication is used for operations from the switchboard.
The shape storage unit of the transmitter stores the positional relationship between the mounting portion of the transmitter and the center position of the light emitting unit.
When transmitting the position information, the transmission device transmits the position information in which this positional relationship is added to the position information stored in the storage unit.
Information is stored in the storage unit of the mounting device at the time of construction of a building or the like. When memorizing the position information, the accurate position is memorized by using the design drawing and CAD data of the building. At the time of building a building, the position can be confirmed by transmitting the position information from the transmission device, which can be used for automation of construction, confirmation of the position where materials are used, and the like.
The mounting device notifies the centralized control device of the information of the transmitting device. The mounting device notifies the centralized control device that a device other than the transmitting device has been mounted.
In FIG. 67, the transmitting device receives light at the light receiving unit, acquires information from the light pattern at the signal analysis unit, and stores it in the storage unit. When receiving light, the transmitting device converts the information stored in the storage unit into a light emitting pattern and causes the light emitting unit to emit light.
Information about the shape of the transmitter is stored in the shape storage unit.
In FIG. 68, the transmitting device stores the signal received by the communication unit in the storage unit. Upon reception, the transmitting device converts the information stored in the storage unit into a light emitting pattern and causes the light emitting unit to emit light.
Information about the shape of the transmitter is stored in the shape storage unit.
When the transmission signal is not stored in the storage unit, the transmission device converts an appropriate signal into a light emission pattern and causes the light emission unit to emit light.
The receiving device acquires the signal transmitted by the transmitting device from the image pickup unit, and transmits the signal and the information to be stored in the transmitting device to the transmitting device and the centralized control device via the communication unit.
The transmission device and the centralized control device store the transmitted information in the storage unit of the transmission device that has transmitted the same signal as the signal acquired from the image pickup unit from the reception device.
The receiving device transmits the signal transmitted by the transmitting device at the same time as the image is captured, and the transmitting device and the centralized control device use the time to specify the transmitting device captured by the receiving device. good.
The communication unit of the receiving device may be a light emitting unit, the communication unit of the transmitting device may be a light receiving unit or an imaging unit, and information may be transmitted from the receiving device to the transmitting device using a light emitting pattern.
The communication unit of the receiving device may be used as a sounding unit, the communication unit of the transmitting device may be used as a sound collecting unit, and information may be transmitted from the receiving device to the transmitting device using a voice pattern.
(Combination with 2D barcode) FIG. 69 is a diagram showing a case of using in combination with a 2D barcode.
The user makes the communication device 2714a and the communication device 2714d face each other.
The communication device 2714a displays the transmission information on the display as a two-dimensional bar code 2714c.
The communication device 2714d reads the two-dimensional bar code 2714c with the two-dimensional bar code reading unit 2714f. The communication device 2714d expresses transmission information as a light emission pattern of the light emitting unit 2714e.
In the communication device 2714a, the light emitting unit is imaged by the image pickup unit 2714b, and the signal is read. This method enables bidirectional direct communication, and when the amount of data to be transmitted is small, communication is possible at a higher speed than communication via a server.
(Creation and use of a map) Fig. 70 is a diagram showing a state of map creation and its use.
The robot 2715a creates a map of the room_2715f by performing self-position estimation based on the signals transmitted by the lighting 2715d and the electronic device 2715c, and obtains the map information, the position information, and the ID of the lighting 2715d and the electronic device 2715c. Store in server 2715e.
Similarly, the receiver 2715b creates a map of the room_2715f from the signals transmitted by the lighting 2715d and the electronic device 2715c, the captured image while moving, and the sensor values of the gyro sensor and 9-axis sensor, and maps information and position. The information and the ID of the lighting 2715d and the electronic device 2715c are stored in the server 2715e.
The robot 2715a efficiently cleans and serves based on the map_2715f acquired from the server 2715e.
Based on the map_2715f acquired from the server 2715e, the receiving device 2715b instructs the robot 2715a on the cleaning place and the moving place, and operates the electronic device in the direction in which the receiving device is directed.
(Acquisition and operation of state of electronic device) FIG. 71 is a diagram showing a state of acquisition and operation of the state of an electronic device.
The communication device 2716a converts the control information into a light emitting pattern, and causes the light emitting unit to emit light toward the light receiving unit 2716d of the electronic device 2716b.
The electronic device 2716b reads the control information from the light emission pattern and operates according to the control information. When the electronic device 2716b receives light from the light receiving unit 2716d, the electronic device 2716b converts information indicating the state of the electronic device into a light emitting pattern, and causes the light emitting unit 2716c to emit light. Further, when there is information to be notified to the user, such as when the operation is completed or an error occurs, the electronic device 2716b converts the information into a light emitting pattern and causes the light emitting unit 2716c to emit light.
The communication device 2716a captures the light emitting unit 2716c and acquires the transmitted signal.
(Recognition of Electronic Device) FIG. 72 is a diagram showing a state of recognizing an electronic device being imaged.
The communication device 2717a has a communication path to the electronic device 2717b and the electronic device 2717e, and transmits an ID display command to each electronic device.
The electronic device 2717b receives the ID display command and transmits an ID signal in the light emitting pattern of the light emitting unit 2717c.
The electronic device 2717e receives the ID display command and transmits an ID signal in a position pattern using the light emitting units 2717f, 2717g, 2717h, and 2717i.
The ID signal transmitted by each electronic device here may be the ID possessed by the electronic device or may be the content instructed by the communication device 2717a.
The communication device 2717a recognizes the electronic device being imaged and the positional relationship between the electronic device and the receiving device from the light emitting pattern and the position pattern of the light emitting unit existing in the captured image.
In order to recognize the positional relationship between the electronic device and the receiving device, it is desirable that the electronic device is provided with three or more light emitting units.
(Display of Augmented Reality Object) FIG. 73 is a diagram showing how an augmented reality object is displayed.
The stage 2718e for displaying the augmented reality transmits information on the augmented reality object and a reference position for displaying the augmented reality object by the emission pattern and the position pattern of the light emitting units 2718a, 2718b, 2718c, and 2718d.
The receiving device superimposes the augmented reality object 2718f on the captured image and displays it based on the received information.
(User interface) When the light emitting part is out of the vicinity of the center of the image pickup range, the center of the image pickup range is directed toward the light emitting part as shown in FIG. 74 in order to direct the center of the image pickup range toward the light emitting part. Display to encourage the user to point.
When the light emitting part is out of the vicinity of the center of the image pickup range, the center of the image pickup range is directed toward the light emitting part, so that the center of the image pickup range is directed toward the light emitting part as shown in FIG. Display to encourage the user.
Even if the light emitting part cannot be confirmed in the imaging range, the position of the light emitting part can be estimated from the previous imaging results and information from the 9-axis sensor, gyro sensor, microphone, position sensor, etc. attached to the imaging terminal. Is displayed to urge the user to direct the center of the imaging range toward the light emitting unit as shown in FIG. 76.
In order to direct the center of the imaging range toward the light emitting portion, the size of the displayed figure is adjusted according to the distance at which the imaging range is desired to be moved, as shown in FIG. 77.
When the image of the light emitting portion is small, a display is performed to urge the user to take an image closer to the light emitting portion, as shown in FIG. 78, in order to obtain a larger image.
If the light emitting part is out of the center of the image pickup range and the image size is not sufficient, the user is urged to direct the center of the image pickup range toward the light emitting part as shown in FIG. 79, and the light emitting part is directed toward the light emitting part. , A display that urges the user to take an image closer to the light emitting part is performed.
When it becomes easier to receive the signal of the light emitting unit by changing the angle formed by the light emitting unit and the imaging range, a display prompting the user to rotate the imaging range is performed as shown in FIG. 80.
If the light emitting part is out of the center of the image pickup range and the signal of the light emitting part can be easily received by changing the angle between the light emitting part and the image pickup range, the center of the image pickup range is shown in FIG. Is displayed so as to urge the user to turn the image toward the light emitting unit and to rotate the imaging range.
If the size of the image of the light emitting part is not sufficient and the signal of the light emitting part can be easily received by changing the angle between the light emitting part and the imaging range, the light emitting part becomes more visible as shown in FIG. A display is performed to urge the user to approach and take an image, and to urge the user to rotate the image pickup range.
The light emitting part is out of the center of the imaging range, the image size of the light emitting part is not sufficient, and the angle between the light emitting part and the imaging range is changed to facilitate receiving the signal of the light emitting part. In this case, as shown in FIG. 83, the user is urged to direct the center of the imaging range toward the light emitting portion, the user is urged to take an image closer to the light emitting portion, and the imaging range is rotated. Display to encourage the user to do so.
While the signal is being received, as shown in FIG. 84, a display indicating that the signal is being received and an information amount of the received signal are displayed.
If the size of the signal to be received is known, the progress bar displays the ratio of the received signal and the amount of information while the signal is being received, as shown in FIG. 85.
While the signal is being received, as shown in FIG. 86, the progress bar displays the percentage of signals that have been received, the receiving location, and the amount of information of the received signal.
While the signal is being received, as shown in FIG. 87, the ratio and the amount of information of the signal whose reception is completed are displayed so as to be superimposed on the light emitting unit.
When the light emitting unit is detected, as shown in FIG. 88, it is indicated that the object is the light emitting unit by displaying, for example, blinking the light emitting unit.
While receiving the signal from the light emitting unit, as shown in FIG. 89, for example, by displaying such as blinking the light emitting unit, it is indicated that the signal is being received from the light emitting unit.
In FIG. 90, when a plurality of light emitting units are detected, the user is made to tap one of the light emitting units to specify a transmission device for receiving a signal or performing an operation.
(Embodiment 2) (Application to ITS) Hereinafter, ITS (Intelligent Transport Systems) will be described as an application example of the present invention. In this embodiment, high-speed communication of visible light communication is realized, and it can be applied to the ITS field.
FIG. 91 is a diagram illustrating communication between a transportation system equipped with a visible light communication function and a vehicle or a pedestrian. The traffic light 6003 is equipped with the visible light communication function of the present embodiment and is capable of communicating with the vehicle 6001 and the pedestrian 6002.
Information is transmitted from the vehicle 6001 and the pedestrian 6002 to the traffic light 6003 by using a headlight or a flash emitting unit of a mobile terminal owned by the pedestrian. Information is transmitted from the traffic light 6003 to the vehicle 6001 and the pedestrian 6002 by lighting the signal with the camera sensor of the traffic light 6003 or the camera sensor of the vehicle 6001.
In the following, we will continue to explain the functions of communicating with traffic support objects installed on the road, such as road lighting and road information boards, with vehicles 6001 and pedestrians 6002, but the same communication method can be applied. The explanation is omitted.
As shown in FIG. 91, the traffic light 6003 provides road traffic information to the vehicle 6001. Here, the road traffic information is information that assists driving, such as traffic congestion information, accident information, and information on surrounding service areas.
Since the traffic light 6003 is equipped with LED lighting, it is possible to provide information to the vehicle 6001 without installing a new device by communicating with this LED lighting. Further, since the vehicle 6001 generally moves at high speed, the amount of data that can be transmitted by the conventional visible light communication technology is small, but since the communication speed is improved by this embodiment, the data that can be transmitted to the vehicle It has the effect of increasing the size.
Further, the traffic light 6003 and the lighting 6004 can provide different information for each signal and lighting. This makes it possible to transmit information according to the position of the vehicle, for example, information is transmitted only to the vehicle traveling in the right turn lane.
Similarly, for pedestrian 6002, it is possible to provide information only to pedestrian 6002 at a specific spot, so for example, walking while waiting for a signal at a pedestrian crossing at a specific intersection. Only people can send information such as information about intersections with many accidents and information on spots in the city.
Furthermore, it is possible to communicate with the traffic light 6003 and another traffic light 6005. Thereby, for example, when changing the information provided by the traffic light 6003, it is possible to change the information distributed by the traffic light by the relay of the communication between the traffic lights without newly passing the signal line or the communication device through the traffic light. be. Since this method greatly improves the communication speed of visible light communication, it is possible to change the distribution information in a shorter time. This makes it possible to change the distribution information several times a day, for example. In addition, it will be possible to immediately deliver information such as snow cover and rainfall.
Further, it is also possible to provide the position information to the vehicle 6001 and the pedestrian 6002 by the lighting distributing the current position information. For facilities with rooftops such as shopping streets and tunnels, it may be difficult to acquire location information using GPS, but if visible light communication is used, it is possible to acquire location information even in such situations. There is a merit that there is. Further, since the communication speed can be made higher than the conventional one by the present embodiment, it is possible to receive information while passing through a specific spot such as a store or an intersection.
Needless to say, since this embodiment speeds up visible light communication, it can be applied to other ITS systems using visible light communication in general.
FIG. 92 is a schematic view of a case where the present invention is applied to vehicle-to-vehicle communication in which vehicles communicate with each other using visible light communication.
Vehicle 6001 transmits information to vehicle 6001a behind through brake lights and other LED lights. It is also possible to transmit data to the oncoming vehicle 6001b through headlights and other lights that illuminate the front.
In this way, it is possible to share information between vehicles by communicating between vehicles with visible light. For example, by transmitting accident information at a front intersection to a rear vehicle in a relay format, it is possible to convey traffic congestion information and alert information to the rear vehicle.
Similarly, by transmitting the sudden braking information acquired from the traffic jam information and the brake sensor information to the oncoming vehicle, it is possible to transmit the information to assist the driving to the oncoming vehicle.
Since the communication speed of visible light communication is improved by the present invention, there is an advantage that information can be transmitted while passing by an oncoming vehicle. In addition, since the interval for transmitting information to the rear vehicle is shortened, it is possible to transmit information to many vehicles in a shorter period of time. It is also possible to send audio and image information by increasing the communication speed. This makes it possible to share richer information between vehicles.
(Position Information Notification System and Facility System) FIG. 93 shows a schematic diagram of a location information notification system and a facility system using the visible light communication technology of the present embodiment. For example, a system for delivering a patient's medical record, a transported item, a medicine, etc. by a robot in a hospital will be described as a representative.
Robot 6101 is equipped with a visible light communication function. Lighting delivers location information. Robot 6101 can deliver medicines and other items to a specific hospital room by acquiring the position information of the lighting. This reduces the burden on the doctor. In addition, since the lighting does not leak to the next room, there is an effect that the robot 6101 does not make a mistake in the room.
The system using visible light communication according to the present embodiment is not limited to the hospital, and can be applied to a system for distributing location information using lighting equipment. For example, in an indoor shopping mall, the position and guidance information may be transmitted from the lighting of the guidance display board, or may be used for moving the cart in the airport.
In addition, by installing visible light communication technology in the lighting of stores, it is possible to distribute coupon information and sale information. By superimposing the information on the visible light, the user can intuitively understand that the information is obtained from the light of the store, which has the effect of improving the convenience of the user.
Also, when transmitting information inside / outside a room, if location information is distributed using wireless LAN, radio waves will leak to adjacent rooms and corridors, so prevent radio waves from flying outside the room. It was necessary to have a function to block radio waves on the outer wall. If the radio waves are still blocked by the outer wall, there is a problem that devices that communicate with the outside such as mobile phones cannot be used.
When the location information is transmitted using the visible light communication of the present embodiment, the communication is possible only within the reach of the lighting, so that there is an effect that it is easy to transmit the location information of a specific room to the user, for example. be. In addition, since the outer wall usually blocks light, there is an effect that no special device is required.
Further, in a building, a large-scale facility, or a general house, the position of the normal lighting does not change, so that the position information transmitted by each lighting does not change frequently. Since the database of location information of each lighting is updated infrequently, there is an effect that the maintenance cost of location information management is low.
(Supermarket system) Fig. 94 illustrates a supermarket system in which a device equipped with the communication method of this embodiment is mounted on a shopping cart and position information is acquired from the lighting of a product rack or indoor lighting. ..
The cart 6201 is equipped with a visible light communication device using the communication method of the present embodiment. Lighting 6100 distributes location information and information on its shelves by visible light communication. The cart can receive the product information delivered from the lighting. Also, by receiving the location information, you can know which shelf you are on, so for example, if you store the location information of the shelf in the cart, you can simply specify the shelf you want to go to or the product you want in the cart. Then, it is possible to display in the cart which direction to go.
Visible light communication makes it possible to obtain highly accurate position information such that the position of the shelf can be known, so that it is possible to acquire and utilize the movement information of the cart. For example, it is possible to create a database of the location information acquired by the cart from the lighting.
It is transmitted to the lighting together with the cart information using visible light communication, or transmitted to the server using wireless LAN or the like. Alternatively, a memory is installed in the cart, and data is collected after the store is closed, so that the route taken by the cart is collected in the server.
By collecting cart movement information, it is possible to grasp which shelves are popular and which aisle is the most routed, which is effective for marketing.
(Communication between mobile phone terminal and camera) Fig. 95 shows an application example using visible light communication of this embodiment.
The mobile phone terminal 6301 uses a flashlight to transmit data to the camera 6302. The camera 6302 receives the data transmitted by the mobile phone terminal 6301 from the optical information received by the image pickup unit.
In the mobile phone terminal 6301, the shooting setting of the camera is set in advance, and the image is transmitted to the camera 6302. This makes it possible to set the camera using the rich user interface of the mobile phone terminal.
In addition, by using the shooting sensor of the camera, when communicating between the camera and the mobile phone terminal, setting information can be transmitted from the mobile phone terminal to the camera without installing a new communication device such as a wireless LAN. It will be possible.
(Underwater communication) FIG. 96 shows a schematic diagram when the communication method of this embodiment is applied to underwater communication. Since water does not transmit radio waves, underwater divers and ships at sea and ships underwater cannot communicate wirelessly. In the visible light communication of this embodiment, it can be used underwater.
The visible light communication method of the present embodiment makes it possible to transmit data from an object or a building that emits light. When the light receiving unit is pointed at a building, guidance information and detailed information of the building can be obtained, so that useful information can be provided to tourists.
The visible light communication method of the present embodiment can be applied to communication from the lighthouse to the ship. Since it is possible to perform communication with a larger capacity than before, it is possible to exchange more detailed information.
Since the visible light communication of the present embodiment uses light, it is possible to perform communication control on a room-by-room basis, such as communication only in a specific room. For example, when accessing information that can be viewed only in a specific room in a library, the communication method of this embodiment is used, or the communication method of this embodiment is used for exchanging key information, and the actual communication is wireless. It can be applied to applications such as using communication such as LAN.
Needless to say, the communication method of the present embodiment can be diverted to all LED communication and an image pickup device provided with a MOS sensor, and can be applied to, for example, a digital camera or a smartphone.
(Embodiment 3) (Example of Service Provision) Hereinafter, in the present embodiment, an example of providing a service to a user will be described with reference to FIG. 97 as an application example of the present invention. FIG. 97 is a diagram for explaining an example of providing a service to a user in the third embodiment. FIG. 97 shows the net server 4000a, the transmitters 4000b, 4000d, 4000e, the receivers 4000c, 4000f, and the building 4000g.
The receivers 4000c and 4000f can provide services to users by receiving and processing signals from multiple transmitters 4000b, 4000d and 4000e inside and outside the house. The transmitter and receiver may be able to process signals independently and provide services to users, or they may behave or transmit according to instructions from the network in cooperation with the net server 4000a that configures the network. It may be possible to provide a service to a user while changing the signal to be output.
The transmitter and receiver may be attached to a moving object such as a person or a vehicle, may be attached to a stationary object, or may be attached to an existing object later. good.
FIG. 98 is a diagram for explaining an example of providing a service to a user in the third embodiment. FIG. 98 shows the transmitter 4001a and the receiver 4001b.
As shown in FIG. 98, the receiver 4001b receives the signals transmitted by the plurality of transmitters 4001a and processes the information contained in the signals to provide the service to the user. The contents included in the signal include device ID, location information, map, signboard, tourist information, traffic information, regional services, coupons, advertisements, product descriptions, characters, music, videos, photos, and voices that uniquely indicate the device. , Menus, broadcasts, emergency information, timetables, guides, applications, news, bulletin boards, commands to devices, personally identifiable information, coupons, credit cards, security, URLs and more.
The user performs registration processing for using the information contained in these signals on the net server in advance, and the user sends these signals to the receiver 4001b at the place where the transmitter 4001a actually transmits the signals. The service may be provided by receiving the signal at, or the service may be provided without the intervention of a net server.
FIG. 99 is a flowchart showing a case where the receiver in the present embodiment simultaneously processes a plurality of signals received from the transmitter.
First, start with step 4002a. Next, in step 4002b, the receiver receives signals from the plurality of light sources. Next, from the result received in step 4002c, the area where each light source is displayed is determined, and the signal is extracted from each light source.
The processing for the number of signals acquired in step 4002d is repeated in step 4002e based on the information contained in the signals until the number becomes 0. When the number to be processed reaches 0, the process ends in step 4002f.
FIG. 100 is a diagram showing an example of a case where communication between devices is realized by mutual communication in the third embodiment. FIG. 98 shows an example of a case where a transmitter and a plurality of transmitters / receivers 4003a, a transmitter / receiver 4003b, and a transmitter / receiver 4003c having a receiver realize communication between devices by communicating with each other. As shown in FIG. 98, the transmitter / receiver may be able to communicate only between the same devices, or may be able to communicate between different devices.
Further, in the present embodiment, the application is distributed to a mobile phone, a smartphone, a personal computer, a game machine, etc. by using the communication means provided by the present embodiment, other networks, and a detachable storage, and the application thereof. It is possible to provide services to users by using the devices (LEDs, photodiodes, image sensors) of the devices already installed from. The application may be installed in the device from the beginning.
(Example of Service Using Directivity) Further, as an application example of the present invention, a service using the characteristics of directivity of the present embodiment will be described. Specifically, it is an example of using the present invention in public facilities such as movie theaters, concert halls, museums, hospitals, public halls, schools, companies, shopping streets, department stores, government offices, food shops, and the like. In the present invention, the directivity of the signal generated by the transmitter to the receiver can be lowered as compared with the conventional visible light communication, so that information can be simultaneously transmitted to a large number of receivers existing in public facilities. Can be done.
FIG. 101 is a diagram for explaining a service utilizing the directivity characteristic in the third embodiment. FIG. 101 shows the screen 4004a, the receiver 4004b, and the illumination 4004c.
As shown in FIG. 101, by using this embodiment in a movie theater, a device (mobile phone, smartphone, personal computer, game machine, etc.) that allows a user to interfere with other users' comfortable viewing of a movie during the screening of a movie. ) Can be suppressed. The transmitter uses light emitted from the image projected on the screen 4004a displaying the movie or the lighting 4004c installed in the facility as a signal, and includes a command to control the receiver 4004b in the signal. By receiving this command, the receiver 4004b can control the operation of the receiver 4004b and prevent an act that hinders other users from watching a movie. The commands that control the receiver 4004b include power supply, sound emitted by reception, communication function, LED display, ON / OFF of vibration, and level adjustment.
Further, the receiver can control the strength of directivity by filtering the signal transmitted by the transmitter by using the strength of the light source or the like. In the present embodiment, by setting the directivity low, commands and information can be transmitted to the receivers existing in the facility all at once.
If you want to increase the directivity, the transmitter side may limit the amount of the light source, the receiver side may reduce the sensitivity to receive the light source, or the amount of the received light source may be signal-processed. , May be constrained.
When this embodiment is used in a store that receives an order from a user and processes the order on the spot, such as a food shop or a government office, the order sent by the transmitter in the user's hand is used. By receiving the signal including the above by a receiver arranged in a place overlooking the store, it is possible to detect which seat user ordered what menu. The service provider can provide a highly fair service to the user by processing the order with a time axis.
A transmitter that can transmit a signal and a receiver that can receive a signal by encrypting / decrypting the information contained in the signal using a private key or public key preset between the transmitter and the receiver. May be constrained. Further, the interception of signals from other devices may be suppressed by using a protocol such as SSL which is used as a standard in the Internet for the transmission path between the transmitter and the receiver.
(Example of service by integrating the real world and the world on the Internet) Further, as an application example of the present invention, a service of superimposing information on the real world captured by a camera and the world on the Internet to provide a service to a user will be described. do.
FIG. 102 is a diagram for explaining another example of the service provision example to the user in the third embodiment. Specifically, FIG. 102 shows an example of a service in which the present embodiment is utilized by using the camera 4005a mounted on the receiver of a mobile phone, a smartphone, a game machine, or the like. Here, FIG. 102 shows the camera 4005a, the light source 4005b, and the superimposed content 4005c.
Signals 4005d transmitted by a plurality of light sources 4005b are extracted from the imaging results of the camera 4005a, and the information contained in those signals 4005d is superimposed and displayed on the camera 4005a. The content 4005c superimposed on the camera 4005a includes character strings, images, videos, characters, applications, URLs, and the like. In addition to superimposing on the camera, information included in the signal may be processed by using voice, vibration, or the like.
FIG. 103 is a diagram showing a format example of a signal included in a light source emitted by a transmitter. FIG. 103 shows the characteristics of the light source 4006a, the service type 4006b, and the service information 4006c.
Information about the service that superimposes the signal received by the receiver on the camera 4006c filters the information that can be acquired from the signal according to the information such as service type 4006b included in the signal emitted by the transmitter and the distance from the camera to the light source. The result. The content to be filtered by the receiver may be in accordance with the preset settings of the receiver, or may be in accordance with the user's preference set in the receiver.
In addition, the receiver can estimate the distance to the transmitter that emits the signal and display the distance to the light source. The receiver estimates the distance to the transmitter by digitally processing the amount of light emitted by the transmitter captured by the camera.
However, since the amount of light of each transmitter imaged by the receiver differs depending on the position and intensity of the light source, the distance may be significantly deviated if the distance is estimated only by the amount of light of the imaged transmitter.
In order to solve this, the signal emitted by the transmitter includes the characteristic 4006a of the light source, which indicates the strength, color, type, etc. of the light source. The receiver can estimate a highly accurate distance by performing digital signal processing that takes into account the characteristics of the light source contained in the signal. If there are multiple light sources of the transmitter imaged by the receiver and the intensities of all the light sources are the same, the distance is estimated from the amount of light of the light sources. If there is a transmitter in which the intensity of the light source imaged by the receiver is not the same, the distance is not estimated only by the amount of the light source, but is estimated by combining with other distance measuring means to estimate the distance from the transmitter to the receiver.
As another distance measuring means, the distance may be estimated by using the parallax of the image captured by the binocular lens, or the distance may be estimated by using an infrared ray or a millimeter wave radar. The movement amount of the receiver may be acquired by an image taken by a 9-axis sensor or the receiver, and the distance may be estimated by combining the distance moved and the triangulation.
It should be noted that the receiver not only filters and displays the signal by using the strength and distance of the signal generated on the transmitter side, but also adjusts the directivity of the signal received from the transmitter by the receiver. It may be used as a means.
(Embodiment 4) FIG. 104 is a diagram showing the principle of the fourth embodiment. FIGS. 105 to 117 are diagrams showing an example of the operation of the fourth embodiment.
In the image sensor shown in FIG. 104 (a), the exposure time of each line 1 is delayed for each line. At a normal shutter speed, each line has a portion that overlaps in time, so that optical signals at the same time are mixed in each line and cannot be discriminated. Here, when the shutter opening time is shortened, if the exposure time is shortened below a certain shutter speed, they do not overlap as shown in FIG. 104 (a), so the optical signals are read separately in time for each line. be able to.
In this state, when the optical signal of "1011011" as shown in the upper part of (a) of FIG. 104 is given, the first optical signal of "1" is photoelectrically converted on the line 1 because it enters the shutter opening time of the line 1. , It is output as "1" of the electric signal 2a in (b) of FIG. 104. Similarly, the next optical signal "0" becomes the electric signal "0" in (b), and the 7-bit "1011011" optical signal is accurately converted into an electric signal.
Actually, as shown in FIG. 104 (b), there is a dead time due to the vertical blanking time, so that the optical signal in a part of the time zone cannot be taken out. In order to solve this, in the present embodiment, when the "normal shooting mode" is switched to the "optical signal read mode", the access address of the image pickup device such as CMOS is changed and the last read at the bottom is changed. The blank period problem is solved by reading the first read line 1a next to line 1h. Although it has a slightly adverse effect on image quality, it has the effect of being able to read continuously (seamlessly), and the transmission efficiency is greatly improved.
In this embodiment, a maximum of one symbol can be assigned to one line. Therefore, when the synchronization method described later is adopted, when a 30 fps image sensor is used for 1000 lines, a maximum transmission of 30 kbps is performed. It is theoretically possible.
When synchronizing, look at the signal of the light receiving element of the camera as shown in Fig. 105, and move the clock for line access up and down so that the maximum contrast can be obtained or the error rate of the data is low. It can be synchronized by changing to. When the clock of the line of the image sensor is faster than that of the optical signal, synchronization can be achieved by receiving one symbol of the optical signal on n lines of 2 or 3 lines as shown in FIG. 105.
In addition, a display such as the TV shown in FIG. 106 or the TV shown on the left side of FIG. When the image is taken by switching to the detection mode, 10 striped patterns peculiar to the present embodiment can be independently detected as shown in the right figure of FIG. 107, so that the transfer rate is 10 times (n times).
For example, if the image sensor of 30fps 1000 line is divided into 10 parts, it will be 300kbps. Since HD video has 1980 pixels horizontally, it can be divided into 50 lines. Then, it becomes 1.5 Mbps, and the video data of the moving image can be received. HD video can be transmitted with 200 lines.
In order to obtain the effect in this embodiment, the longest exposure time that can be detected is T.<sub>0</sub>Then, T<sub>0</sub>It is necessary to shorten the shutter time below. Assuming that the same wave number of frames is fp as shown in the upper right figure of FIG. 104, the shutter time must be less than half of 1 / fp. This is because the blanking that occurs during shooting is up to half the size of one frame. That is, since the blanking time is less than half of the shooting time, the actual shooting time is 1/2 fp in the shortest time.
However, since 4-value PPM etc. is required to suppress flicker, it is 1 (fp x 2 x 4), that is, 1/8 fp. Since the camera of a normal mobile phone has fp = 30, 60, if the shutter speed is set to 1/240, 1/480, that is, 1/480 or less, the present embodiment can be viewed while maintaining compatibility. Optical communication can be received using a camera such as a mobile phone.
Actually, since there are many mobile phones in the world that do not adopt the synchronization method in this embodiment, the communication is initially asynchronous. In this case, by receiving one symbol using a scanning line that is more than twice the clock of the optical signal, or specifically 2 to 10 times, the information rate drops, but compatible communication is realized.
In the case of lighting equipment that needs to prevent flicker, it is turned off or dimmed to emit light in one of the four bits, that is, quadrature PPM. In this case, the bit rate drops to half, but since there is no flicker, it can be used for lighting equipment and can send light and data.
In Fig. 108, a common signal is sent from all the illuminators indoors in a common time zone, and individual illuminators L are sent in individual time zones.<sub>4</sub>It shows the situation of receiving an optical signal in the situation where individual sub-information is sent from. L<sub>4</sub>Due to its small area, it takes time to send a large amount of data. Therefore, only a few bits of ID are sent in the individual time zone, and L is sent in the common time zone.<sub>1</sub>, L<sub>2</sub>, L<sub>3</sub>, L<sub>4</sub>, L<sub>5</sub>All send common information with the same content.
Explaining in detail using FIG. 109A, the common information in the time zone A below FIG. 109A is "position information of the reference position of the room, arrangement information of individual devices of each ID (difference position information from the reference position), server. URL, data broadcasting, LAN transmission data "is the two lights of the main area M, which is all the lights in the room, and the S in a part of it.<sub>1</sub>, S<sub>2</sub>, S<sub>3</sub>, S<sub>4</sub>Causes the same optical signal to be transmitted all at once. Since the entire room is illuminated with the same optical signal, there is an effect that the camera unit of the mobile phone can reliably receive data during a common time zone.
On the other hand, in the time zone B, as shown in the upper right view, the main region M does not blink and continuously emits light with a normal 1 / n light intensity. In the case of 4-value PPM, flicker can be prevented because the average amount of light does not change when emitting light at 3/4 of the normal value, that is, 75%. If it blinks within the range where the average light intensity does not change, there is no flicker, but the partial region S in the time zone B<sub>1</sub>, S<sub>2</sub>, S<sub>3</sub>, S<sub>4</sub>It is not preferable because it causes noise to receive. In time zone B, S<sub>1</sub>, S<sub>2</sub>, S<sub>3</sub>, S<sub>4</sub>Sends optical signals with different data contents. Since the main region M does not send a modulated signal, it is positionally separated like the screen of the mobile phone on the upper right. Therefore, for example, S<sub>1</sub>When the image of the region is taken out, since there is little noise, it becomes easy to detect the stripes generated in the region, and stable data can be obtained.
FIG. 109B is a diagram for explaining the operation of the transmitter and the receiver in the present embodiment.
The transmitter 8161 is configured as, for example, a signage, and changes the brightness of the area A described as "A shop" and the area B described as "B shop". As a result, signal A and signal B are transmitted from the respective regions. For example, each of the signal A and the signal B includes a common part showing common contents and a unique part showing different contents from each other. The intersections of signal A and signal B are transmitted simultaneously. As a result, when the receiver 8162 receives at least one of the intersection of signal A and the intersection of signal B, it displays an image of the entire signage. Further, the transmitter may simultaneously transmit the eigenpart of the signal A and the eigenpart of the signal B, or may transmit at different timings from each other. Upon receiving, for example, the unique portion of the signal B, the receiver 8162 displays, for example, the detailed information of the store corresponding to the above-mentioned area B.
FIG. 109C is a diagram for explaining the operation of the transmitter and the receiver in the present embodiment.
For example, transmitter 8161 simultaneously transmits the intersections of signal A and signal B, and then simultaneously transmits the unique parts of signal A and signal B that are different from each other. The receiver 8162 receives a signal from the transmitter 8161 by imaging the transmitter 8161.
Here, when the transmitter 8161 is transmitting the intersection of the signal A and the signal B, the transmitter 8161 can be regarded as one large area without being divided into two areas. As a result, the receiver 8162 can receive its intersection even if it is located far from the transmitter 8161. At this time, the receiver 8162 acquires and displays the information associated with the common portion from the server. For example, the server sends information about all the stores shown in the signage, which is the transmitter 8161, to the receiver 8162. Alternatively, the server selects the information of any store from those stores and sends it to the receiver 8162. For example, the server preferentially sends the information of the store paying the highest registration amount among those stores to the receiver 8162. Alternatively, the server transmits information about the store corresponding to the area (area A or area B) in the center of the range captured by the camera of the receiver 8162. Alternatively, the server randomly selects a store and sends the store information to the receiver 8162.
Further, the receiver 8162 can receive a unique portion of the signal A or the signal B when it is located close to the transmitter 8161. At this time, the receiver 8162 acquires the information associated with the unique part from the server.
For example, as shown in FIG. 110, when the camera is scanning in the horizontal direction (horizontal direction), in the case of 4-value PPM, the illumination L<sub>2</sub>Is photographed by a face camera, and when three stripes are visible as shown on the right side, "0101", that is, 4 bits of data per frame can be demodulated. Since there is ID data in this, it is possible to detect the position of the mobile terminal at high speed, that is, in a short time by calculating the distance difference information or placement information between the reference position information of the common data and each ID of the individual data. effective. In this way, for example, the data and positions of four light sources can be instantly recognized with one frame information simply by sending a 2-bit ID.
An example of using the low-bit ID information of this individual light source will be described with reference to FIG. 111.
In this embodiment, the common data 101 in FIG. 111 is the reference position, the URL of the server, and the placement information of each ID as shown in the figure. A large amount of data for area-specific data broadcasting is sent in a common time zone using all lighting.
As mentioned above, L in (a) of Fig. 111<sub>1</sub>, L<sub>2</sub>, L<sub>3</sub>, L<sub>4</sub>~ L<sub>8</sub>Individual ID can be demodulated by 3 bits.
Even if signals of frequency f1 and frequency f2 are sent as shown in (b) of FIG. 111, stripes peculiar to the present invention are detected in each illumination unit and converted into ID data corresponding to the frequency and ID data corresponding to the modulation data. .. When this pattern is calculated using the arrangement information, it is possible to know from which position the image was taken. That is, since the arrangement information and the reference position information of each ID can be obtained from L0, the position of the terminal can be specified.
As shown in (b) of Fig. 111, by assigning frequencies f1 and f2 to each ID and setting, for example, f1 = 1000Hz, f2 = 1100Hz, ..., f16 = 2500Hz, 16 values, that is, 4-bit values can be obtained. It can be expressed by frequency. By changing the transmission frequency at regular intervals, more signals can be transmitted. When changing the frequency or starting / ending the modulation, keeping the average brightness constant before and after the change has the effect of not causing flicker to the human eye.
Since the receiver obtains the frequency from the cycle of the signal, it is possible to reduce the reception error by allocating the signal so that the reciprocal or logarithm of the frequency is evenly spaced rather than allocating the frequency to the signal at equal intervals.
For example, if the signal is changed every 1/15 second, it can be transmitted at 60 bits per second. Since a general image pickup device captures an image of 30 frames per second, if a signal is transmitted at the same frequency for 1/15 second, the image will be reliably captured even if the transmitter is only part of the captured image. It can be performed.
Also, by transmitting a signal at the same frequency for 1/15 second, the signal is received even if there are frames that cannot be processed due to the heavy load on the receiver, or even if the image pickup device can only capture 15 frames per second. be able to.
When the luminance value is frequency-analyzed in the direction perpendicular to the exposure line by Fourier transform or the like, the frequency of the transmission signal appears as a peak. When a plurality of frequencies are imaged in one frame such as a frequency switching portion, a plurality of peaks are obtained as weaker peaks than when a single frequency signal is Fourier transformed. A protective portion may be provided in the frequency switching portion in order to prevent the front and rear frequencies from being mixed with each other.
According to this method, even when light that is sequentially transmitted at multiple frequencies is imaged on one screen, the transmission frequency can be analyzed, and it takes less than 1/15 seconds or 1/30 seconds. It can be received even if the frequency of the transmission signal is changed.
In order to recognize the order of the transmitted signals, the Fourier transform may be performed in a range shorter than one screen. Further, the captured screens may be connected and the Fourier transform may be performed in a range longer than one screen. In this case, the brightness value of the blanking time of imaging is treated as unknown. The protected part shall be a signal of a specific frequency or shall not change the luminance (frequency 0Hz).
As shown in (b) of FIG. 111, by transmitting the FM modulation signal of frequency f2, subsequently transmitting the PPM modulation signal, and alternately transmitting the FM modulation signal and the PPM modulation signal, only one of the methods can be used. Receiving is possible even with an incompatible receiver, and by assigning highly important information to an FM-modulated signal that is relatively easy to receive, it is possible to preferentially transmit highly important information.
In this embodiment, since the ID of each device and the position on the screen can be obtained at the same time, the image information and the position information linked to each ID of the lighting in the database of the cloud-type server at the URL linked to the lighting. And app programs can be downloaded and images of related products can be displayed on top of the image of the device with ID lighting using the AR method. In this case, by switching the demodulation mode of the present embodiment to the shooting mode, there is an effect that a weighted AR image can be obtained on a beautiful image.
By the way, as shown in FIG. 108, by sending the distance difference d between the light source of each ID and the reference position from north, south, east, and west to the time zone A, the illumination L<sub>4</sub>You can see the exact position in cm. Next, the height H of the ceiling and the height h are obtained from the height of the owner of the mobile phone, and the attitude information of the mobile phone is corrected by the 9-axis sensor. θ<sub>1</sub>And d = (Hh) × arctan θ<sub>1</sub>Calculate d by operations such as.
In this way, the position of the mobile phone can be obtained with high accuracy. By sending the common optical signal to the time zone A and the individual optical signal to the time zone B in this way, it is possible to reliably send a large amount of common information and a small amount of individual information such as an ID almost at the same time. The effect is obtained.
Individual light source S like the mobile terminal in the upper right of Fig. 109A<sub>1</sub>~ S<sub>4</sub>Is photographed. At time C, S, as shown in the time chart below Figure 109A.<sub>1</sub>Only sends an optical signal. Since only one stripe is generated as shown in t = C in Fig. 112, there is an effect that it can be detected without being affected by noise.
Also, two individual data may be sent such as t = D and E. By sending individual data that are most spatially separated, such as t = H and I, it is easy to separate them on the screen, which has the effect of lowering the error rate.
Under the situation of t = C in Fig. 112, S<sub>1</sub>Since it is only necessary to demodulate, it is not necessary to scan the image sensor in other areas. Therefore, like t = C, S<sub>1</sub>By reducing the number of scans so that the area of S is included, S<sub>1</sub>Only the area is scanned and the data can be demodulated. Therefore, not only speeding up, but also S<sub>1</sub>There is an effect that a large amount of data can be demodulated only in a narrow area of.
However, in this case, the area S is due to camera shake.<sub>1</sub>May deviate from the scanning range of the image sensor.
Therefore, image stabilization as shown in FIG. 113 is important. The gyro sensor built into the mobile phone cannot usually detect minute rotations with a narrow range such as camera shake.
Therefore, as shown in the left figure of FIG. 113, L<sub>2</sub>When the optical signal is received by the face camera, it is difficult to detect camera shake from the image using the image from the face camera, such as when scanning is limited. Therefore, the influence of camera shake can be reduced by turning on the in-camera, detecting camera shake from the image of the in-camera, and correcting the scanning range or the detection range. This is because the camera shake of the face camera and the camera shake of the in-camera are the same.
A normal image can be obtained from this region by slowing the shutter speed in a scanning region other than the optical signal pattern of the face camera, and camera shake correction can be performed from this image. In this case, one camera can detect camera shake and signal. This has the same effect when the in-camera shown in Fig. 113 and the right figure is used.
In FIG. 114, an optical signal is detected by a face camera, and first, the position information of the terminal is obtained.
Distance traveled from this point l<sub>2</sub>The 9-axis sensor for mobile phones is not practical because of its poor accuracy. In this case, using the in-camera opposite to the face camera, the moving distance l is as shown in Fig. 114 from the change in the posture of the terminal and the pattern on the floor surface.<sub>2</sub>Can be asked. The pattern on the ceiling may be detected with a face camera.
An actual application example will be described below.
FIG. 115 is a diagram showing a state in which a data broadcast, which is common data, is received from the ceiling lighting in the station yard, and one's position is obtained from individual data.
As shown in FIG. 116, the authentication information is displayed from the mobile terminal on which the barcode is displayed at the coffee shop, and after the terminal of the store reads the authentication information, the light is emitted from the light emitting portion of the terminal of the store, and the mobile terminal describes the present invention. It can be used to receive, perform mutual authentication, and improve security. Authentication may be performed in reverse.
When the customer with the mobile terminal sits at the table, the location information obtained is transmitted to the terminal of the store via wireless LAN, etc., and the location of the customer is displayed on the terminal of the clerk. The clerk can carry it to the location information table.
In FIG. 117, a passenger knows his / her position in a train or an aircraft by using the method of the present embodiment, and orders a product such as food at a terminal. The crew has the terminal of the present invention in the cart, and the ID number of the ordered product is displayed at the position of the ordered customer on the screen, so that the product with the ordered ID is accurately delivered to the customer.
FIG. 107 is a diagram showing a case where the method or apparatus of the present embodiment is used for the backlight of a display such as a TV. Since fluorescent lamps, LEDs, and organic EL can be subjected to low luminance modulation, the transmission of the present embodiment is possible. However, the scanning direction is important due to its characteristics. When used vertically like a smartphone, it is scanned horizontally, so a horizontally long light emitting area is provided at the bottom of the screen, and the contrast of the image on TV etc. is thin and the signal can be easily received by making it closer to white. There is.
When scanning in the vertical direction as in a digital camera, a vertical length display as shown on the right side of the screen in FIG. 106 is provided.
By providing these two in one screen and emitting the same optical signal from both, the signal can be received by the image sensor in either scanning direction.
If you try to receive a light emitting part in the vertical direction with an image sensor that scans in the horizontal direction, it is more accurate by displaying a display such as "Please rotate horizontally to take a picture" on the screen of the terminal. It is possible to encourage the user to receive at high speed.
As shown in FIG. 105, the communication speed can be significantly increased by controlling and synchronizing the scanning line reading clock of the image sensor of the camera with the light emitting pattern of the light emitting unit.
When one symbol of the emission pattern is detected by two lines as shown in (a) of FIG. 105, the pattern shown on the left is synchronized. In the case of the pattern as shown in the middle figure, since the image sensor reads quickly, the read clock of the video element is slowed down and synchronized. In the case of the pattern shown on the right, speed up and synchronize.
When one symbol in (b) of FIG. 105 is detected by three lines, the speed is slowed down in the pattern shown in the middle figure and speeded up in the figure on the right.
In this way, high-speed optical communication becomes possible.
In the case of bidirectional communication, since the lighting device of the light emitting unit is provided with an infrared light receiving unit as a motion sensor, bidirectional reception can be performed in the illuminator without adding parts by also using this for reception. .. The terminal side may transmit using a strobe light for a camera, or may be provided with an inexpensive infrared light emitting unit separately. In this way, bidirectional communication is realized without adding too many parts.
(Embodiment 5) (Signal transmission by phase modulation) FIG. 118 is a timing diagram of a transmission signal in the information communication device of the fifth embodiment.
In FIG. 118, the reference waveform (a) is a clock signal having a period T and serves as a reference for the timing of the transmission signal. The transmission symbol (b) indicates a symbol string created based on the data string to be transmitted. Here, as an example, the case of 1 bit per symbol is shown, so that it is the same as the binary of the transmission data. The transmission waveform (c) is a transmission waveform phase-modulated according to the transmission symbol with respect to the reference waveform, and the transmission light source is driven according to this waveform. Phase modulation is performed by shifting the phase of the reference waveform corresponding to the symbol. In this example, symbol 0 is assigned a phase of 0 ° and symbol 1 is assigned a phase of 180 °.
FIG. 119 is a diagram showing the relationship between the transmission signal and the reception signal in the fifth embodiment.
The transmission signal is the same as in FIG. 118, and the light source emits light only for the period when the transmission signal is 1, and this light emission period is shown in the region of the diagonal line downward to the right. The band painted with the upward-sloping diagonal line represents the period during which the pixel of the image sensor is exposed (exposure time tE), and the signal charge of the pixel of the image sensor is generated in the region overlapping the downward-sloping diagonal line indicating the light emission period. NS. The pixel value p is proportional to the area of this overlapping region. Here, the relationship of Equation 1 holds between the exposure time tE and the period T.
tE = T / 2 × (2n + 1) n: Natural number (Equation 1)
Note that FIGS. 119 to 123 show the case of n = 2. That is, tE = 2.5T.
The received waveform shows the pixel value p of each line. Here, the value of the pixel value axis is normalized with the amount of light received for one cycle as 1. As described above, since the exposure time tE has a section of T (n + 1/2), the pixel value p always exists in the range of npn + 1, and for the example of FIG. 119, 2 p 3.
FIGS. 120 to 122 are diagrams showing the relationship between the transmission signal and the reception signal for the symbol strings different from those in FIG. 119.
The transmitted signal has a preamble containing the same sequence of consecutive symbols (not shown) (eg, a sequence of consecutive symbol 0s), and the receiver receives the reference for reception by means of the sequence of consecutive symbols in this preamble. A signal is created and used as a timing signal for reading a symbol string from a received waveform. Specifically, for continuous symbol 0, as shown in FIG. 119, the received waveform returns a constant waveform that repeats 2 3 2, and a clock signal is output based on the timing at which this pixel value 3 is output. Generated as a reference signal.
Next, the symbol can be read from the received waveform by reading the received signal in one section of the reference signal, and reading it as symbol 0 in the case of pixel value 3 and symbol 1 in the case of pixel value 2. FIGS. 120 to 122 show how the symbols located in the 4th period are read out.
FIG. 123 is a diagram that summarizes FIGS. 119 to 122, and since the lines are densely arranged, the pixel division in the line direction is omitted and the drawings are continuous. Here, it is shown how the symbol values of the 4th to 8th periods are read out.
With this configuration, in the present embodiment, when the intensity of the optical signal is averaged for a time sufficiently longer than the period of the reference wave, it is always constant. Then, if the frequency of the reference wave is set appropriately high, the time can be set shorter than the time when the person perceives the change in the amount of light. Therefore, when the transmission light source is observed from the human eye, the light source is constant. It is perceived as shining. Therefore, it is not perceived as flickering of the light source, and has an advantage that the person in the previous embodiment does not feel annoyed.
Further, in a situation where the exposure time of each line is long and the time overlapping with the exposure period of the adjacent line is long, the frequency (symbol rate) of the signal can be increased by the amplitude modulation (on-off modulation) in the above-described embodiment. There is a problem that a sufficient signal transmission speed cannot be obtained without being able to do so, but in the present embodiment, since the rising and falling edges of the signal can be detected even in such a situation, the frequency of the signal is increased. It is possible to realize a high signal transmission speed.
The "phase modulation" described here means phase modulation with respect to the reference signal waveform. In the original sense, the carrier wave is optical, which is amplitude-modulated (on-off-modulated) and transmitted. Therefore, the modulation method in this signal transmission is a kind of amplitude modulation.
Further, the above-mentioned transmission signal is an example, the number of bits per symbol can be set to 2 or more, and the correspondence between the symbol and the phase shift is not limited to 0 ° and 180 °, and the offset is set. You may have.
Although the description is omitted above, the configurations, operations, and effects of the optical signal generating means and the optical signal receiving means described in the embodiments 6 to 11 described in FIGS. 124 to 200 below are shown in FIGS. 1 and later. Needless to say, even if the high-speed light emitting means and the optical signal receiving means described in the first and subsequent embodiments are replaced with each other and operated by using the drawings, the same effect can be obtained. On the contrary, the same applies even if the high-speed light emitting means and the receiving means of the first and subsequent embodiments are replaced with the low-speed light emitting means and the receiving means.
For example, in FIG. 114, an example of receiving data such as position information from an optical signal from an illuminator using a face camera which is a camera on the display side of a mobile phone and an in-camera on the opposite side in FIG. 113. showed that. In this case, by using a 9-axis sensor, the vertical direction can be detected by gravity.
As shown in Fig. 116, when receiving an optical signal from a mobile phone on a table in a restaurant, if the front side of the mobile phone is facing up in response to the signal from the 9-axis sensor, activate the face camera. If you are facing down, you can switch to the in-camera and receive the optical signal to reduce power consumption or receive the optical signal quickly, and stop unnecessary camera operation. Can be done. In this case, the direction of the camera on the table can be detected from the brightness of the camera and the same operation can be performed. In addition, when the camera switches from the shooting mode to the optical signal reception mode, a command to increase the shutter speed is sent, and a command to increase the sensitivity of the image sensor is sent to the image sensor to increase the sensitivity and brighten the image. There is. Increasing the sensitivity increases noise, but it is white noise. On the other hand, since the optical signal is in a specific frequency range, the detection sensitivity can be increased by separating or removing it with a frequency filter. Therefore, it is possible to detect an optical signal from a dark lighting device.
In the present invention, an optical signal is mainly generated from a lighting device in an indoor space, and the optical signal is generated by a camera unit of a portable terminal having a communication unit, a voice microphone, a voice speaker, a display unit, an in-camera, and a face camera camera unit. It receives and obtains position information, etc., but when going from indoors to outdoors, position information can be detected by GPS using satellites. Therefore, there is an effect that the position can be seamlessly detected by obtaining the position information of the boundary from the optical signal area, activating the signal reception from the GPS, and automatically performing this switching.
When moving from outdoors to indoors, the boundary is detected by the position information such as GPS and the optical signal is automatically switched to the position information. In addition, when a barcode is displayed on the display of a mobile phone and authentication is performed at a store or a POS terminal at the boarding gate of an aircraft, it takes a long time to respond using a server, so it is not realistic and only one-way authentication is possible.
However, since the mutual authentication can be performed by transmitting an optical signal from the light emitting unit of the reader of the terminal such as POS to the face camera unit of the mobile phone by using the present invention, the security is improved.
(Embodiment 6) Hereinafter, the flow of processing for communicating using the camera of a smartphone by transmitting information as a blinking pattern of an LED provided in the device will be described.
FIG. 124 is a diagram showing an example of the environment in the house according to the present embodiment. In the environment shown in FIG. 124, the TV 1101, the microwave oven 1106, the air purifier 1107, and the smartphone 1105 are all around the user.
FIG. 125 is a diagram showing an example of communication between a home electric appliance and a smartphone according to the present embodiment. FIG. 125 is an example of information communication, and is a diagram showing a configuration in which information is obtained by acquiring information output by each device such as the television 1101 and the microwave oven 1106 in FIG. 124 with a smartphone 1201 owned by the user. be. As shown in FIG. 125, each device transmits information using the blinking pattern of the LED, and the smartphone 1201 receives the information using the image pickup function by a camera or the like.
FIG. 126 is a diagram showing an example of the configuration of the transmitting side device 1301 in the present embodiment.
This transmitting device 1301 uses information as a blinking pattern of light by a user pressing a button, sending a transmission instruction by NFC (Near Field Communication), or detecting a state change such as a failure inside the device. Send. At this time, transmission is repeated for a certain period of time. As the information to be transmitted, a shortened ID may be used as long as it is between devices registered in advance. Further, if a wireless communication unit such as a wireless LAN or a specific power-saving wireless unit is provided, the authentication information required for the connection can be transmitted as a blinking pattern.
In addition, the transmission speed determination unit 1309 grasps the performance of the clock generator inside the device, so that if the clock generator is inexpensive and inaccurate, the transmission speed is slowed down, and if it is high, the transmission speed is increased. It is possible to perform processing to speed up. If the performance of the clock generator is poor, or by dividing the transmitted information itself into short sections, it is possible to reduce errors due to accumulation of deviations in blinking intervals due to long-term communication.
FIG. 127 is a diagram showing an example of the configuration of the receiving side device 1401 in the present embodiment.
The receiving side device 1401 determines an area where blinking light is observed from the frame image acquired by the image acquisition unit 1404. At this time, the blinking can be taken by a method of tracking an area where a certain level of brightness is observed.
The blinking information acquisition unit 1406 acquires transmitted information from the blinking pattern, and if it contains device-related information such as a device ID, it is attached to the related server on the cloud using that information. Inquire for information or interpolate using the information stored in the device in the wireless communication area stored in advance or in the receiving device. This has the effect of correcting errors due to noise when capturing the light emission pattern, and reducing the time for the user to hold the smartphone over the light emitting part of the transmitting device in order to acquire the already obtained information. ..
Hereinafter, FIG. 128 will be described.
FIG. 128 is a diagram showing a flow of processing for transmitting information to a receiving side device such as a smartphone by blinking an LED of the transmitting side device in the present embodiment. Here, it is assumed that information is transmitted by a transmitting side device having a function of communicating with a smartphone by NFC and an LED light emission pattern embedded in a part of the communication mark for NFC possessed by the transmitting side device.
First, in step 1001a, the user purchases a home electric appliance, plugs the power into the outlet for the first time, and the power is turned on.
Next, in step 1001b, check whether the initial setting information has been written. If Yes, proceed to Circle 3 in the figure. On the other hand, if No, the process proceeds to step 1001c, and the mark blinks at a blinking speed that is easy for the user to understand (example: 1 to 2/5).
Next, in step 1001d, it is confirmed whether or not the user can acquire the device information of the home appliance by touching the mark on the smartphone from NFC communication. Here, if Yes, the process proceeds to step 1001e, and the smartphone receives the device information on the cloud server and registers it in the cloud. Then, in step 1001f, the shortened ID associated with the smartphone user's account is received from the cloud, sent to the home appliance, and the process proceeds to step 1001g. In step 1001d, if No, the process proceeds to step 1001g.
Next, in step 1001g, check if there is registration with NFC. If Yes, proceed to step 1001j, blink blue twice, and then step 1001k to end the blinking.
On the other hand, in step 1001g, if No, the process proceeds to step 1001h. Then, in step 1001h, check if 30 seconds have passed. If Yes, proceed to step 1001i, and the LED part outputs the device information (device model number, NFC registration processing presence / absence, device unique ID) by blinking light, and goes to circle 2 in Fig. 129. move on.
In step 1001h, if No, the process returns to step 1001d.
Next, with reference to FIGS. 129 to 132, the flow of the process of transmitting information to the receiving side device by blinking the LED of the transmitting side device in the present embodiment will be described. Here, FIGS. 129 to 132 are diagrams showing a flow of processing for transmitting information to the receiving side device by blinking the LED of the transmitting side device.
Hereinafter, FIG. 129 will be described.
First, in step 1002a, the user launches an application that acquires the light blinking information of the smartphone.
Next, in step 1002b, the image acquisition portion acquires the blinking light. Then, the blinking area determination unit determines the blinking area from the change in the time series of the image.
Next, in step 1002c, the blinking information acquisition unit determines the blinking pattern in the blinking area and waits for preamble detection.
Next, if the preamble can be detected in step 1002d, the information of the blinking area is acquired.
Next, when the device ID information can be acquired in step 1002e, the information is sent to the server on the cloud side and the information obtained from the cloud side and the information of the blinking information acquisition unit are sent to the server on the cloud side even in the reception continuation state. Interpolate while comparing.
Next, in step 1002f, when all the information including the interpolated information is available, the smartphone or the user is notified. At this time, by displaying the related sites and GUI obtained from the cloud, richer and easier-to-understand notifications become possible, and proceed to circle 4 in Fig. 130.
Hereinafter, FIG. 130 will be described.
First, in step 1003a, when the home appliance generates a message such as a failure, the number of uses to notify the user, and the room temperature, the information transmission mode is started.
Next, in step 1003b, the mark blinks in 1 to 2 / s. At the same time, the LED also starts transmitting information.
Next, in step 1003c, check whether NFC communication has started. If No, proceed to circle 7 in FIG. 132. If Yes, proceed to step 1003d and stop the LED blinking.
Then, in step 1003e, the smartphone accesses the cloud server and displays the relevant information.
Next, in step 1003f, in the case of a failure that requires local response, search for a service person to support on the server side. Use home appliances, installation location, and location information.
Next, in step 1003g, the serviceman presses the buttons of the home appliances in the predetermined order to switch to the support mode.
Next, in step 1003h, when the blinking of the marker is visible from the smartphone on the LED of the home appliance other than the marker, some or all of the LEDs that can be seen at the same time blink so as to interpolate the information, and the process proceeds to circle 5 in FIG.
Hereinafter, FIG. 131 will be described.
First, in step 1004a, if the performance of the receiving terminal owned by the serviceman can detect high-speed blinking (example: 1000 / s times), the serviceman presses the setting button.
Next, in step 1004b, the LED of the home appliance blinks in high-speed mode, and the process proceeds to circle 6.
Hereinafter, FIG. 132 will be described.
First, in step 1005a, the blinking continues.
Then, in step 1005b, the user comes to get the LED blinking information on the smartphone.
Next, in step 1005c, the user launches an application that acquires the light blinking information of the smartphone.
Next, in step 1005d, the image acquisition portion acquires the blinking light. Then, the blinking area determination unit determines the blinking area from the change in the time series of the image.
Next, in step 1005e, the blinking information acquisition unit determines the blinking pattern in the blinking area and waits for preamble detection.
Next, if the preamble can be detected in step 1005f, the information of the blinking area is acquired.
Next, if the device ID information can be acquired in step 1005g, the information is sent to the server on the cloud side even in the reception continuation state, and the information obtained from the cloud side by the information interpolation unit and the information of the blinking information acquisition unit are input. Interpolate while comparing.
Next, in step 1005h, when all the information including the interpolated information is available, the smartphone or the user is notified. At this time, by displaying the related sites and GUI obtained from the cloud, richer and easier-to-understand notifications will be possible.
Then, the process proceeds to step 1003f in FIG.
In this way, a transmitting device such as a home electric appliance can transmit information to the smartphone by blinking the LED. Information can be transmitted even with devices that do not have communication means such as wireless functions and NFC, and rich information on servers on the cloud can be provided to users via smartphones.
Further, as shown in the present embodiment, both bidirectional communication (for example, NFC communication) and one-way communication (for example, communication due to a change in LED brightness) are performed between two devices including at least one portable device. When there is a function to send and receive data by communication method, when data is transmitted from one device to the other device by one-way communication, and when data transmission and reception is realized by two-way communication, It is possible to stop one-way communication. This is efficient because it is possible to eliminate the waste of power consumption required for one-way communication.
As described above, according to the sixth embodiment, it is possible to realize an information communication device capable of communication between various devices including a device having a small computing power.
Specifically, the information communication device of the present embodiment has an information management unit that manages device information including its own unique ID and device status information, a light emitting element, and information as a blinking pattern of the light emitting element. It has an optical transmission unit for transmission, and the optical transmission unit converts the device information into a blinking pattern of light and transmits it when there is a change in the internal state of the device.
Here, for example, a start history management unit that stores the sensed information inside the device such as its own start state and user's usage history is provided, and the optical transmission unit is registered in advance of the clock generator to be used. Performance information may be acquired and the transmission speed may be changed.
Further, for example, in the optical transmission unit, a second light emitting element is arranged around a first light emitting element for transmitting information by blinking light, and the second light emitting element is the first light emitting element. When information transmission by blinking of one light emitting element is repeated a certain number of times, light may be emitted between the end and the start of information transmission.
It should be noted that these general or specific embodiments may be realized by a system, a method, an integrated circuit, a computer program or a recording medium, and may be realized by any combination of a system, a method, an integrated circuit, a computer program or a recording medium. May be done.
(Embodiment 7) In the present embodiment, taking a vacuum cleaner as an example, the communication procedure between the device and the user using visible light communication, the initial setting using visible light communication, the repair service at the time of failure, and cleaning. Service cooperation using a machine will be explained.
FIGS. 133 and 134 are diagrams for explaining a procedure for communicating between a user and a device using visible light in the present embodiment.
Hereinafter, FIG. 133 will be described.
First, in circle 1, start.
Next, in step 2001a, the user turns on the power.
Next, in step 2001b, it is confirmed whether the initial settings such as the installation setting and the NW setting have been made as the startup process.
Here, if the initial settings have been made, the process proceeds to step 2001f, the normal operation is started, and the operation is terminated as shown in circle 3.
On the other hand, if the initial setting has not been made, the process proceeds to step 2001c, and the user is notified by "LED normal light emission" and "buzzer sound" that the initial setting is necessary.
Next, in step 2001d, device information (product number and serial number) is collected to prepare for visible light communication.
Next, in step 2001e, the user is informed that device information (product number and serial number) visible light communication is possible by "LED communication light emission", "icon display on display", "buzzer sound", and "light emission of multiple LEDs". Notice.
Then, it ends as shown in circle 2.
Next, FIG. 134 will be described.
First, start as shown in circle 2.
Next, in step 2002a, the approach of the visible light receiving terminal is detected by the proximity sensor, the illuminance sensor and the human sensor.
Next, in step 2002b, visible light communication is started by using the detected thing as a trigger.
Next, in step 2002c, the user acquires device information on the visible light receiving terminal.
Then, it ends as shown in circle 4. Alternatively, the process proceeds to any of step 2002f to step 2002i.
When proceeding to step 2002f, the detection by the "sensitivity sensor" and the detection that the room is turned off by "cooperation with the dimming device" are stopped, and the light emission of the device information is stopped, as shown in circle 5. finish. Further, when the process proceeds to step 2002g, the visible light receiving terminal notifies that the device information has been acquired and sensed by "NFC communication" and "NW communication", and ends. If the process proceeds to step 2002h, the device information is stopped by detecting the detachment of the visible light receiving terminal, and the process ends. If the process proceeds to step 2002i, after a certain period of time, the light emission of the device information is stopped and the process ends.
If it is not detected in step 2002a, proceed to step 2002d and perform "brighten", "make the sound louder", "move the icon", etc. to make the notification that visible light communication is possible after a certain period of time. Strengthen and notify. Now go back to step 2002d. Alternatively, the process proceeds to step 2002e, and after a certain period of time, the process proceeds to step 2002i.
FIG. 135 is a diagram for explaining a procedure for the user in the present embodiment to purchase the device and perform the initial setting of the device.
In FIG. 135, first, it starts as shown by circle 4.
Next, in step 2003a, the position information of the smartphone that received the device information is acquired by GPS (Global Positioning System).
Next, in step 2003b, if the smartphone has user information, user information such as a user name, a telephone number, and an e-mail address is collected in the terminal. Alternatively, in step 2003c, if the smartphone does not have user information, user information is collected from peripheral devices through the NW.
Next, in step 2003d, the device information, the user information, and the location information are transmitted to the cloud server.
Next, in step 2003e, the information required for the initial setting and the activation information are collected using the device information and the location information.
Next, in step 2003f, the linkage information such as the IP, the authentication method, and the available services required for setting the linkage with the user-registered device is collected. Alternatively, in step 2003g, the device information and the setting information are transmitted to the user-registered device through the NW to set the cooperation with the peripheral device.
Next, in step 2003h, user setting is performed using the device information and the user information.
Next, in step 2003i, the initial setting information, activity information, and cooperation setting information are sent to the smartphone.
Next, in step 2003j, the initial setting information, activation information, and cooperation setting information are transmitted to the home appliance by NFC.
Next, in step 2003k, set the device with the initial setting information, activation information, and linkage setting information.
Then, it ends as shown in circle 6.
FIG. 136 is a diagram for explaining a service dedicated to a serviceman when the device in the present embodiment fails.
In FIG. 136, first, it starts as shown by circle 3.
Next, in step 2004a, history information such as an operation log and a user operation log generated during the normal operation of the device is recorded in the local storage medium.
Next, in step 2004b, at the same time as the failure occurs, error information such as an error code and error details is recorded, and an LED abnormal light emission notifies that visible light communication is possible.
Next, in step 2004c, the serviceman's special command is executed to switch to the LED high-speed light emission mode and start high-speed communication of visible light.
Next, in step 2004d, it is determined whether the nearby terminal is a normal smartphone or a serviceman's dedicated receiving terminal. Here, if the process proceeds to step 2004e, in the case of a smartphone, error information is acquired and the process ends.
On the other hand, when the process proceeds to step 2004f, in the case of a serviceman, the dedicated receiving terminal acquires error information and history information.
Next, in step 2004g, the device information, error information, and history information are sent to the cloud to acquire the repair method. Here, when the process proceeds to step 2004h, the LED high-speed light emission mode is canceled by executing a special command of the serviceman, and the process ends.
On the other hand, when proceeding to step 2004i, the related products of the device information, the product information of similar products, the selling price of the nearest store, and the new product information are acquired from the cloud server.
Next, in step 2004j, the user information is acquired through visible light communication between the user's smartphone and the serviceman's dedicated terminal, and the product is ordered from the nearest store through the cloud server.
Then, it ends as shown in circle 9.
FIG. 137 is a diagram for explaining a service for confirming a cleaning status using a vacuum cleaner and visible light communication in the present embodiment.
First, start as shown in circle 3.
Then, in step 2005a, the cleaning information during normal operation of the device is recorded.
Next, in step 2005b, the dirt information is created in combination with the floor plan information and encrypted and compressed.
Here, when the process proceeds to step 2005c, the dirt information is recorded in the local storage medium triggered by the compression of the dirt information. Alternatively, when the process proceeds to step 2005d, the dirt information is transmitted to the lighting equipment by visible light communication with the temporary stop of cleaning (stop of suction processing) as a trigger. Alternatively, when the process proceeds to step 2005e, the dirt information is transmitted to the home local server and the cloud server by NW, triggered by the recording of the dirt information.
Next, in step 2005f, the device information, the storage location, and the decoding key are transmitted to the smartphone by visible light communication, triggered by the transmission and recording of the dirt information.
Next, in step 2005g, dirt information is acquired and decoded through NW and NFC.
Then, it ends as shown in circle 10.
As described above, according to the sixth embodiment, it is possible to realize a visible light communication system including an information communication device that enables communication between various devices including a device having a small computing power.
Specifically, the visible light communication system (FIG. 133) including the information communication device of the present embodiment has a visible light transmission availability determination unit for determining whether or not the preparation for visible light transmission is completed, and a visible light transmission. It is equipped with a visible light transmission notification unit that notifies the user that it is inside, and visible light communication that visually and audibly notifies the user when visible light communication becomes possible. It is a system. As a result, the user's convenience can be improved by notifying the user of the situation in which visible light can be received by the LED emission mode of "emission color", "sound", "icon display", and "multiple LED emission".
Preferably, as described with reference to FIG. 134, a terminal proximity sensing unit that detects the approach of the visible light receiving terminal, and a visible light transmission determining unit that determines the start and stop of visible light transmission depending on the position of the visible light receiving terminal. It may be a visible light communication system that starts the visible light transmission by triggering the detection of the proximity of the visible light receiving terminal by the terminal proximity sensing unit.
Here, for example, as described with reference to FIG. 134, a visible light communication system may be used in which the visible light transmission is stopped by triggering the detection of the detachment of the visible light receiving terminal by the terminal proximity sensing unit. Further, for example, as described with reference to FIG. 134, a peripheral illuminance sensing unit for detecting the extinguishing of the room is installed, and the visible light transmission is stopped by triggering the detection of the extinguishing of the room by the peripheral illuminance sensing unit. It may be a visible light communication system. By sensing the approach and departure of the visible light receiving terminal and the extinguishing of the room, visible light communication is started only in situations where visible light communication is possible, eliminating unnecessary visible light communication and saving energy. Can be changed.
Further, as described with reference to FIG. 134, for example, a visible light communication time monitoring unit that measures the time during which visible light transmission is performed and a visible light transmission notification unit that notifies the user that visible light transmission is in progress. As a visible light communication system that is installed and enhances visual and auditory notification to the user, triggered by the fact that visible light communication is performed for a certain amount or more but the visible light receiving terminal does not approach. May be good. Further, as described with reference to FIG. 134, for example, even after the visible light transmission notification unit enhances the notification, the visible light transmission time has been set for a certain period of time or longer, but the visible light transmission terminal does not approach the visible light receiving terminal. It may be a visible light communication system that stops optical transmission.
As a result, when the user does not receive visible light even after a certain period of visible light transmission time, the user is requested to receive visible light and stopped, thereby preventing the user from forgetting to receive and erasing the visible light, which is convenient for the user. Can be improved.
Further, the visible light communication system (FIG. 135) including the information communication device of the present embodiment has a visible light reception determination unit that determines that visible light communication has been received, and a reception terminal position acquisition for acquiring the terminal position. It is equipped with a unit and a device setting information collection unit that acquires device information and position information and collects device setting information, and acquires the position of the receiving terminal with the reception of visible light as a trigger to set the device. It may be a visible light communication system that collects necessary information. As a result, by using the acquisition of device information by visible light communication as a trigger, the device setting and the location information and user information required for user registration are automatically collected and set, so that the user can input and register. It can be omitted to improve convenience.
Here, as further described with reference to FIG. 137, a device information management unit that manages device information, a device-related management unit that manages the similarity between devices, and a store information management that manages store information that sells devices. It is equipped with a department and a nearest store search unit that searches for the nearest store from location information, and the nearest store and price where similar devices are sold triggered by receiving device information and location information. It may be a visible light communication system to be acquired. As a result, the convenience of the user can be improved by collecting the sales status and the sales store of the related device according to the device information and eliminating the trouble of searching for the device.
In addition, the visible optical communication system (FIG. 135) including the information communication device of the present embodiment has a user information monitoring unit that monitors that user information is stored in the terminal, and user information from peripheral devices through a NW. It is equipped with a user information collection unit that collects user information and a user registration processing unit that acquires user information and device information and registers the user, and collects user information from peripheral devices that can be accessed by triggering the absence of user information. It may be a visible optical communication system that collects information and registers users together with device information. Thereby, a visible light communication by the trigger to acquire the device information by Shin, instrument settings, and, by setting the positional information and the user information required for user registration to automatically collect, registration and input by the user Can be omitted to improve convenience.
Further, the visible light communication system (FIG. 136) including the information communication device of the present embodiment includes a command determination unit that accepts special commands, a visible light communication frequency, and a visible light communication speed adjustment unit that operates a plurality of LED linkages. , And it may be a visible light communication system that speeds up visible light communication by adjusting the frequency of visible light communication by accepting special commands and adjusting the number of transmission LEDs. Here, for example, as described with reference to FIG. 137, a terminal type determination unit that determines the type of a nearby terminal by NFC communication and a transmission information type determination unit that determines information to be transmitted according to the terminal type are installed. , It may be a visible light communication system that changes the amount of information transmitted by a nearby terminal and the visible light communication speed. As a result, high-speed communication can be enabled and user convenience can be improved by adjusting the frequency of visible light communication and the number of transmission LEDs according to the receiving terminal to change the speed and transmission information of visible light communication.
Further, the visible light communication system (FIG. 137) including the information communication device of the present embodiment has a cleaning information recording unit for recording cleaning information, a layout information recording unit for recording layout information, and layout information and cleaning information. It is equipped with an information synthesis unit that generates dirt location information by superimposing it, and an operation monitoring unit that monitors the stoppage of normal operation. It may be a visible light communication system for transmission.
It should be noted that these general or specific embodiments may be realized by a system, a method, an integrated circuit, a computer program or a recording medium, and may be realized by any combination of a system, a method, an integrated circuit, a computer program or a recording medium. May be done.
(Embodiment 8) In the present embodiment, Web information using optical communication and cooperation of devices will be described by taking a home delivery service as an example.
FIG. 138 shows an outline of the present embodiment. That is, FIG. 138 is a schematic diagram of home delivery service support using optical communication in the present embodiment.
Specifically, the orderer places an order for a product from the product purchase site using the mobile terminal 3001a. When the order is completed, the order number will be issued from the product purchase site. The mobile terminal 3001a that received the order number is transmitted to the doorphone home appliance 3001b using NFC communication.
The doorphone home appliance 3001b displays the order number received from the mobile terminal 3001a on the monitor of its own device to indicate to the user that the transmission has been completed.
The doorphone internal unit 3001b transmits a blinking instruction and a blinking pattern of the LED built in the doorphone external unit 3001c to the doorphone external unit 3001c. The blinking pattern is generated by the doorphone home appliance 3001b according to the order number received from the mobile terminal 3001a.
The doorphone outside unit 3001c blinks the LED according to the blinking pattern specified by the doorphone inside unit 3001b.
Instead of a mobile terminal, the environment may be such that the product purchase site on WWW3001d such as a PC can be accessed.
As a means of transmitting from the mobile terminal 3001a to the doorphone home appliance 3001b, a home network may be used in addition to NFC communication.
The mobile terminal 3001a may directly send the order number to the doorphone outside unit 3001c without mediating the doorphone inside unit 3001b.
When there is an order from the orderer, the order number is transmitted from the delivery order receiving server 3001e to the delivery person mobile terminal 3001f. When the delivery person arrives at the delivery destination, optical communication is performed using the LED blinking pattern generated based on the order number in both directions of the delivery person mobile terminal 3001f and the doorphone outside unit 3001c.
Next, it will be described with reference to FIGS. 139 to 144. FIGS. 139 to 144 are flowcharts for explaining the home delivery service support using optical communication in the third embodiment of the invention.
FIG. 139 shows the flow from the orderer placing an order to the issuance of the order number. Hereinafter, FIG. 139 will be described.
In step 3002a, the orderer mobile terminal 3001a makes a reservation for home delivery using the Web browser or application of the smartphone. Then, proceed to circle 1 in FIG.
Further, the orderer mobile terminal 3001a waits in the order number transmission waiting state in step 3002b following the circle 2 in FIG. Then, in step 3002c, it is confirmed whether the order number destination device is touched. If Yes, proceed to step 3002d and send the order number by NFC touching the doorphone room unit (if the doorphone is on the same network as the smartphone, you can also send it via the network). On the other hand, if No, the process returns to step 3002b.
The doorphone home appliance 3001b first waits for the LED blinking request from another terminal in step 3002e. Then, in step 3002f, the order number is received from the smartphone. Then, in step 3002g, the LED blinking instruction of the doorphone outdoor unit is issued according to the received order number. Then, proceed to circle 3 in FIG. 142.
The doorphone outside unit 3001c first waits for the LED blinking instruction from the doorphone inside unit in step 3002h. Then, proceed to circle 7 in FIG. 142.
The delivery person mobile terminal 3001f waits in the order notification waiting state in step 3002i. Then, in step 3002j, it is confirmed whether or not the order notification has been notified from the home delivery order server. If No, the process returns to step 3002i. If Yes, proceed to step 3002k and receive information such as order number and delivery address. Then, in step 3002n, the user waits until the camera is activated for the LED light emission instruction of the order number received or the LED light emission recognition of another device. Then, proceed to circle 5 in FIG. 141.
FIG. 140 shows the flow until the orderer places a home delivery order on the orderer mobile terminal 3001a. Hereinafter, FIG. 140 will be described.
The delivery order server 3001e first waits for the order number in step 3003a. Then, in step 3003b, it is confirmed whether or not the delivery order has been received. If No, the process returns to step 3003a. If Yes, proceed to step 3003c and issue an order number for the received courier order. Then, in step 3003d, the delivery person is notified that the delivery order has been received, and the process ends.
The orderer mobile terminal 3001a selects the order contents from the menu presented by the home delivery order server in step 3003e following the circle 1 in FIG. 139. Then, in step 3003f, the order is confirmed and sent to the delivery server. Then, in step 3003g, check whether the order number has been received. If No, the process returns to step 3003f. If Yes, proceed to step 3003h, display the received order number, and display a prompt to touch the doorphone home appliance. Then, proceed to circle 2 in FIG. 139.
FIG. 141 shows a flow in which a delivery person uses the delivery person mobile terminal 3001f to perform optical communication with the doorphone external unit 3001c of the delivery destination. Hereinafter, FIG. 141 will be described.
Following the circle 5 in FIG. 139, the delivery person mobile terminal 3001f confirms in step 3004a whether or not to activate the camera to recognize the LED of the doorphone outside unit 3001c of the delivery destination. Here, in the case of No, the process returns to circle 5 in FIG. 139.
On the other hand, if Yes, proceed to step 3004b and check the LED blinking of the delivery destination doorphone outside unit with the camera of the delivery person's mobile terminal.
Next, in step 3004c, the LED light emission of the doorphone outdoor unit is recognized and collated with the order number.
Next, in step 3004d, check whether the LED blinking of the doorphone outside unit matches the order number. If Yes, proceed to Circle 6 in Fig. 143.
If No, check if other LEDs can be seen blinking in the camera. Then, if Yes, the process returns to step 3004c, and if No, the process ends.
FIG. 142 shows a flow for collating the order number between the doorphone internal unit 3001b and the doorphone external unit 3001c. Hereinafter, FIG. 142 will be described.
The doorphone external unit 3001c confirms whether or not the LED blinking instruction was given from the doorphone internal unit in step 3005a following the circle 7 in FIG. 139. If No, return to the circle in Figure 139. If Yes, proceed to step 3005b and blink the LED according to the blinking LED instructed by the doorphone home appliance. Then, proceed to circle 8 in Fig. 143.
Further, the doorphone outdoor unit 3001c notifies the doorphone internal unit of the LED blinking recognized by the camera of the doorphone external unit in step 3005c following the circle 9 in FIG. 143. Then, proceed to circle 10 in FIG.
Following the circle 3 in FIG. 139, the doorphone internal unit 3001c issues an LED blinking instruction according to the order number to the doorphone external unit in step 3005d. Next, in step 3005e, the camera of the doorphone outside unit waits until the LED blinking of the delivery person mobile terminal is recognized. Next, in step 3005f, it is confirmed whether or not the doorphone outside unit has notified that the LED blinking has been recognized. If No, the process returns to step 3005e. If yes, in step 3005g, match the LED blinking of the doorphone outside unit with the order number. Next, in step 3005h, check whether the LED blinking of the doorphone outside unit matches the order number. If Yes, proceed to Circle 11 in FIG. On the other hand, in the case of No, in step 3005i, the doorphone outside unit is instructed to stop blinking the LED, and the process ends.
FIG. 143 shows the flow between the doorphone external unit 3001c and the delivery person mobile terminal 3001f after collating the order number. Hereinafter, FIG. 143 will be described.
Following the circle 6 in FIG. 141, the delivery person mobile terminal 3001f starts blinking the LED according to the order number held by the delivery person mobile terminal in step 3006a.
Next, in step 3006b, place the blinking LED part within the range where the camera can shoot from the doorphone outside unit.
Next, in step 3006c, whether or not the LED blinking of the doorphone outside unit LED is indicated by whether the LED blinking of the delivery person mobile terminal taken by the camera of the doorphone outside unit matches the order number held by the doorphone inside unit. Check.
If No, the process returns to step 3006b. On the other hand, if Yes, the process proceeds to step 3006e, the matching pros and cons are displayed on the delivery person's mobile terminal, and the process ends.
Further, as shown in FIG. 143, whether or not the doorphone external unit 3001c recognized the LED blinking of the delivery person mobile terminal by the camera of the doorphone external unit in step 3006f following the circle 8 in FIG. 142. To confirm. If Yes, proceed to circle 9 in FIG. 142. If No, return to the circle in Fig. 142.
FIG. 144 shows the flow between the doorphone outside unit 3001c and the delivery person mobile terminal 3001f after collating the order number. Hereinafter, FIG. 144 will be described.
Following the circle 11 in FIG. 142, the doorphone external unit 3001c confirms whether or not the LED blinking notified by the doorphone internal unit has been notified in step 3007a as to whether or not it matches the order number. Here, in the case of No, the process returns to circle 11 in FIG. 142. On the other hand, if Yes, proceed to step 3007b, blink the LED indicating the pros and cons of matching with the doorphone outside unit, and finish.
Further, as shown in FIG. 144, the doorphone home appliance 3001b notifies the orderer by a ringing sound output indicating that the delivery person has arrived at the doorphone home appliance in step 3007c following the circle 10 in FIG. 142. do. Next, in step 3007d, the doorphone external unit is instructed to stop blinking the LED and blink the LED indicating that the order number is matched. And it ends.
If the delivery destination is absent in the condominium, a delivery box for storing the delivery may be installed at the entrance or the like. If the orderer is absent at the time of delivery, the delivery person stores the delivery in the delivery box. By using the LED of the delivery person mobile terminal 3001f to perform optical communication with the camera of the doorphone outside unit 3001c and transmitting the size of the delivery item, the doorphone outside unit 3001c automatically matches the size of the delivery item. It is also possible to make only the delivery box available.
As described above, according to the eighth embodiment, it is possible to realize Web information using optical communication and cooperation of devices.
(Embodiment 9) Hereinafter, the ninth embodiment will be described.
(Registering the User and the Mobile Phone in Use in the Server) FIG. 145 is a diagram for explaining a process of registering the user and the mobile phone in use in the server in the present embodiment. Hereinafter, FIG. 145 will be described.
First, in step 4001b, the user launches the application.
Then, in step 4001c, the information of this user and the mobile phone is queried to the server.
Next, in step 4001d, it is confirmed whether the user information and the information of the mobile phone in use are registered in the DB of the server.
If Yes, the process proceeds to step 4001f, (processing a) the analysis of the user voice characteristics is started as parallel processing, and the process proceeds to B in FIG. 147.
On the other hand, if No, proceed to 4001e, register the mobile phone ID and user ID in the mobile phone table of the DB, and proceed to B in FIG. 147.
(Processing a: Analysis of User Voice Characteristics) FIG. 146 is a diagram for explaining a process for analyzing user voice characteristics in the present embodiment. Hereinafter, FIG. 146 will be described.
First, in step 4002a, sound is collected from the microphone.
Next, in step 4002b, it is confirmed whether or not the voice collected as a result of voice recognition is presumed to be the user's voice. If No, the process returns to step 4002a.
If Yes, proceed to step 4002c and check if the generated content is a keyword used in this application ("Next", "Back", etc.). If Yes, the process proceeds to step 4002f, the voice data is registered in the user keyword voice table of the server, and the process proceeds to step 4002d. On the other hand, if No, the process proceeds to step 4002d.
Next, in step 4002d, the voice characteristics (frequency, sound pressure, speech speed) are analyzed.
Next, in step 4002e, the analysis result is registered in the user voice characteristic table of the mobile phone and the server.
(Preparation for voice recognition processing) FIG. 147 is a diagram for explaining a process for preparing for voice recognition processing in the present embodiment. Hereinafter, FIG. 147 will be described.
First, following B in the figure, in step 4003a, an operation (user operation) to display the cooking menu list is performed.
Next, in step 4003b, the cooking menu list is obtained from the server.
Next, in step 4003c, the cooking menu list is displayed on the mobile screen.
Then, in step 4004d, sound collection is started from the microphone connected to the mobile phone.
Next, in step 4003e, sound collection from peripheral sound collecting devices is started as parallel processing (processing b).
Next, in step 4003f, (processing c) the analysis of the environmental sound characteristics is started as parallel processing.
Next, in step 4003g, cancellation of audio from audio output devices existing in the vicinity (processing d) is started as parallel processing.
Next, in step 4003h, the user voice characteristics are acquired from the DB of the server.
Finally, in step 4003i, the recognition of the user voice is started, and the process proceeds to C in FIG. 151.
(Process b: Sound collection from peripheral sound collecting devices) FIG. 148 is a diagram for explaining a process for collecting sound from peripheral sound collecting devices in the present embodiment. Hereinafter, FIG. 148 will be described.
First, in step 4004a, a device (sound collecting device) that can communicate from a mobile phone and can collect sound is searched for.
Then, in step 4004b, check if you have found a sound collector.
Here, if No, the process ends. If Yes, proceed to step 4004c to acquire the position information of the sound collector and the microphone characteristic information from the server.
Then, in step 4004d, check if the information exists on the server.
If Yes, proceed to step 4004e and check if the installation position of the sound collecting device is sufficiently close to the position of the mobile phone and it is possible to collect the user's voice. If No in step 4004e, the process returns to step 4004a. On the other hand, if Yes in step 4004e, the process proceeds to step 4004f and the sound collecting device is made to start collecting sound. Then, in step 4004g, the sound collected by the sound collecting device is transmitted to the mobile phone until there is a command to end the sound collecting process. It should be noted that the collected voice may not be transmitted to the mobile phone as it is, but the result of voice recognition may be transmitted to the mobile phone. Further, the voice transmitted to the mobile phone is processed in the same manner as the voice collected from the microphone connected to the mobile phone, and the process returns to step 4004a.
If the result is No in step 4004d, the process proceeds to step 4004h to cause the sound collecting device to start collecting sound. Then, in step 4004i, a signal sound is output from the mobile phone. Then, in step 4004j, the sound collected by the sound collecting device is transmitted to the mobile phone. Then, in step 4004k, it is confirmed whether or not the signal sound can be recognized from the voice sent from the sound collecting device. Here, if Yes, the process proceeds to step 4004g, and if No, the process returns to step 4004a.
(Processing c: Analysis of environmental sound characteristics) FIG. 149 is a diagram for explaining the analysis processing of the environmental sound characteristics in the present embodiment. Figure 149 will be described below.
First, in step 4005f, a list of devices owned by this user, excluding those whose position is sufficiently far from the position of the microwave oven, is obtained. The data of the sound emitted by these devices is acquired from the DB.
Next, in step 4005g, the characteristics (frequency, sound pressure, etc.) of the acquired sound data are analyzed and retained as environmental sound characteristics. It should be noted that the sound emitted by a rice cooker or the like near the microwave oven is particularly prone to misrecognition, so a high degree of importance is set and maintained.
Next, in step 4005a, sound is collected from the microphone.
Next, in step 4005b, it is confirmed whether the collected voice is the user's voice, and if Yes, the process returns to step 4005a. If No, proceed to step 4005c and analyze the characteristics (frequency, sound pressure) of the collected voice.
Next, in step 4005d, the environmental sound characteristics are updated from the analysis results.
Next, in step 4005e, check whether the end flag is set, and if Yes, end. If No, return to step 4005a.
(Process d: Canceling the sound from the audio output device existing in the vicinity) FIG. 150 is a diagram for explaining the process of canceling the sound from the audio output device existing in the periphery in the present embodiment. Hereinafter, FIG. 150 will be described.
First, in step 4006a, a device capable of outputting voice (voice output device) is searched for with a device capable of communication.
Next, in step 4006b, it is confirmed whether or not the audio output device has been found, and if No, the process ends. If Yes, the process proceeds to step 4006c, and the signal sound including various frequencies is output to the audio output device.
Next, in step 4006d, the sound is collected by the mobile phone and the sound collecting device (each sound collecting device) of FIG. 148, and the signal sound output from the voice output device is collected.
Next, in step 4006e, it is confirmed whether or not the signal sound can be collected and recognized, and if No, the process ends. If Yes, proceed to step 4006f, and the transmission characteristics from the audio output device to each sound collection device (relationship between the output volume for each frequency and the sound collection volume, and the delay time from the signal sound output to the sound collection). To analyze.
Next, in step 4006g, it is confirmed whether the audio data output by the audio output device can be accessed from the mobile phone.
If Yes, proceed to step 4006h, acquire the audio source, output location, and volume output from the audio output device until there is a command to end the cancel process, and use each sound collecting device while considering the transmission characteristics. Cancels the sound output by the sound output device from the collected sound. Return to step 4006a. On the other hand, in the case of No, the process proceeds to step 4006i, the audio output from the audio output device is acquired until there is a command to end the cancellation process, and the audio is output from the audio collected by each sound collector while considering the transmission characteristics. Cancel the audio output by the device and return to step 4006a.
(Selection of cooking menu and setting of operation contents in microwave oven) FIG. 151 is a diagram for explaining a process of selecting a cooking menu and setting operation contents in a microwave oven in the present embodiment. Hereinafter, FIG. 151 will be described.
First, following C in the figure, in step 4007a, select the menu you want to cook (user operation).
Next, in step 4007b, (user operation) recipe parameters (amount to be cooked, strength of taste, degree of baking, etc.) are set.
Next, in step 4007c, the recipe data matching the recipe parameters and the microwave oven operation content setting command are acquired from the server.
Next, in step 4007d, the user is urged to touch the mobile phone with the non-contact IC tag embedded in the microwave oven.
Then, in step 4007e, check whether to detect the touch to the microwave oven.
If No, the process returns to step 4007e. If Yes, proceed to step 4007f and send the microwave oven setting command acquired from the server to the microwave oven. As a result, all the microwave oven settings required for this recipe are set, and the user can cook by simply pressing the operation start button of the microwave oven.
Next, in step 4007g, (processing e) the notification sound for the microwave oven is acquired from the DB of the server or the like and set in the microwave oven.
Next, in step 4007h, (process f) adjust the notification sound of the microwave oven, and proceed to D in FIG. 155.
(Processing e: The notification sound for the microwave oven is acquired from the DB of the server or the like and set in the microwave oven) FIG. 152 shows that the notification sound for the microwave oven in the present embodiment is acquired from the DB of the server or the like and set in the microwave oven. It is a figure for demonstrating the process to perform. Hereinafter, FIG. 152 will be described.
First, in step 4008a, the user brings (= touches) the mobile phone to the non-contact IC tag embedded in the microwave oven.
Next, in step 4008b, the microwave oven is inquired whether the notification sound data for the mobile phone (sound data output at the time of operating the microwave oven or at the end of operation) is registered in the microwave oven.
Next, in step 4008c, it is confirmed whether the notification sound data for this mobile phone is registered in the microwave oven.
Here, if Yes, the process ends. If No, proceed to step 4008d and check if the notification sound data for this mobile phone is registered in the mobile phone. If Yes, the process proceeds to step 4008h, the notification sound data registered in the mobile phone is registered in the microwave oven, and the process ends. On the other hand, if No, proceed to step 4008e and refer to the DB of the server or mobile phone or microwave oven.
Next, in step 4008f, if the notification sound data for this mobile phone (notification sound data that this mobile phone can easily recognize) is in the DB, that data, if not, the notification sound for general mobile phones. Obtain data (notification sound data that is generally easy for mobile phones to recognize) from the DB.
Next, in step 4008g, the acquired notification sound data is registered in the mobile phone.
Next, in step 4008h, the notification sound data registered in the mobile phone is registered in the microwave oven, and the process ends.
(Process f: Adjustment of the notification sound of the microwave oven) FIG. 153 is a diagram for explaining the process of adjusting the notification sound of the microwave oven in the present embodiment. Hereinafter, FIG. 153 will be described.
First, in step 4009a, the notification sound data of this microwave oven registered in the mobile phone is acquired.
Next, in step 4009b, it is confirmed whether or not the overlap between the frequency of the notification sound for this terminal and the frequency of the environmental sound is above a certain level.
Here, if No, the process ends.
On the other hand, if Yes, proceed to step 4009c and set the volume of the notification sound to be sufficiently louder than the ambient sound. Or change the frequency of the notification sound.
Here, to give an example of how to create a notification sound with a changed frequency, if the microwave oven can output the sound of (c) in FIG. 154, the notification sound is created with the pattern of (c) and the process ends. .. If (c) is not possible and (b) is possible, a notification sound is created with the pattern of (b) and the process ends. If only (a) is possible, create a notification sound with the pattern of (a) and end.
FIG. 154 is a diagram showing an example of the waveform of the notification sound set in the microwave oven in the present embodiment.
The waveform shown in (a) of FIG. 154 is a simple rectangular wave and can be output by most audio output devices. Since it is easy to be confused with voices other than the notification sound, it should be output several times, and if some of them can be recognized, it should be recognized that the notification sound has sounded.
The waveform shown in FIG. 154 (b) is a waveform obtained by finely dividing the waveform of (a) by a short-time rectangular wave, and can be output if the operating clock of the audio output device is sufficiently fast. This sound is heard by the human ear in the same way as the sound of (a), but in machine recognition, it has a larger amount of information than (a) and has the property of being difficult to be confused with sounds other than the notification sound.
The waveform shown in (c) of FIG. 154 is a waveform in which the time length of the audio output portion is changed, and is called a PWM waveform. It is more difficult to output than (b), but it has more information than (b), and it is possible to improve the recognition rate and simultaneously convey the information that you want to convey from the microwave oven to the mobile phone.
The waveforms in (b) and (c) of FIG. 154 are less likely to be erroneously recognized than those in (a) of FIG. 154, but the same waveform is repeated several times as in (a) of FIG. 154. , The recognition rate can be further improved.
(Display of cooking contents) FIG. 155 is a diagram showing an example of a waveform of a notification sound set in a microwave oven in the present embodiment. Hereinafter, FIG. 155 will be described.
First, following D in the figure, the cooking content is displayed in step 4011a.
Next, in step 4011b, check whether the cooking content is operating the microwave oven.
Here, if Yes, the process proceeds to step 4011c, the user is notified to put the food in the microwave oven and press the operation start button, and the process proceeds to E in FIG. 156.
On the other hand, in the case of No, the process proceeds to step 4011d, the cooking content is displayed, and the process proceeds to F in the figure or to step 4011e.
In step 4011e, check what the user is doing. If the application is terminated, it will be terminated.
On the other hand, in the case of change operation of display contents, manual input (button press, etc.), voice input ("Next", "Previous", etc.), proceed to step 4011f and check whether cooking is completed as a result of changing the display contents. do. Here, if Yes, the process proceeds to step 4011g, the user is notified of the end of cooking, and the process ends. If No, proceed to step 4011a.
(Recognition of Notification Sound of Microwave Oven) FIG. 156 is a diagram for explaining a process of recognizing the notification sound of a microwave oven in the present embodiment. Hereinafter, FIG. 156 will be described.
First, following E in the figure, in step 4012a, as parallel processing (processing g), sound collection from peripheral sound collecting devices and recognition of microwave oven notification sound are started.
Next, in step 4012f, (process i) confirmation of the mobile phone operation status is started as parallel processing.
Next, in step 4012g, (process j) user position tracking is started as a parallel process.
Next, in step 4012b, the recognition content is confirmed.
Here, when the notification sound of the button press is recognized, the process proceeds to step 4012c, the setting change is registered, and the process returns to step 4012b. If the user's operation is recognized, the process proceeds to F in FIG. 155. If you recognize (the sound of the end of operation or the sound of opening the door of the microwave oven) after the operation time has elapsed after putting the ingredients in the microwave and displaying that the operation start button is pressed, in step 4012e, (Process h ) Notify the user of the end of operation of the microwave oven, and proceed to G in Fig. 155. When the notification sound of the start of operation is recognized, the process proceeds to step 4012d, waits until the operation time elapses, proceeds to step 4012e, (process h) notifies the user of the end of the operation of the microwave oven, and G in FIG. 155. Proceed to.
(Process g: Sound collection from peripheral sound collecting devices and recognition of microwave oven notification sound) Fig. 157 shows the process of collecting sound from peripheral sound collecting devices and recognizing microwave oven notification sound in the present embodiment. It is a figure for demonstrating. Hereinafter, FIG. 157 will be described.
First, in step 4013a, a device (sound collecting device) that can communicate from a mobile phone and can collect sound is searched for.
Then, in step 4013b, check if you have found a sound collector.
Here, if No, the process ends. On the other hand, if Yes, the process proceeds to step 4013c, and the position information of the sound collector and the microphone characteristic information are acquired from the server.
Then, in step 4013d, check if the information exists on the server.
If Yes, proceed to step 4013r and check if the sound collecting device is installed close enough to the microwave oven to collect the notification sound.
Here, in the case of No in step 4013r, the process returns to step 4013a. If Yes, proceed to step 4013s and check if the arithmetic unit of the sound collecting device can recognize voice. If Yes in step 4013s, in step 4013u, information for recognizing the notification sound of the microwave oven is transmitted to the sound collecting device. Then, in step 4013v, the sound collecting device is started to collect sound and voice recognition, and the recognition result is transmitted to the mobile phone. Then, in step 4013q, the recognition process of the notification sound of the microwave oven is performed until the next cooking stage is performed, and the recognition result is transmitted to the mobile phone. On the other hand, in step 4013s, if No, the process proceeds to step 4013t, and the sound collecting device is started to collect sound and the sound collecting voice is transmitted to the mobile phone. Then, in step 4013j, the collected voice is transmitted to the mobile phone until the next cooking stage is performed, and the notification sound of the microwave oven is recognized by the mobile phone.
If the result is No in step 4013d, the process proceeds to step 4013e, and it is confirmed whether the arithmetic unit of the sound collecting device can recognize voice.
If Yes, the process proceeds to step 4013k, and the information for recognizing the notification sound of the microwave oven is transmitted to the sound collecting device. Then, in step 4013m, the sound collecting device is started to collect sound and voice recognition, and the recognition result is transmitted to the mobile phone. Then, in step 4013n, the notification sound of the microwave oven is sounded. Then, in step 4013p, it is confirmed whether or not the notification sound can be recognized by the sound collecting device. In step 4013p, if Yes, the process proceeds to 4013q, the notification sound of the microwave oven is recognized until the next cooking stage is performed, the recognition result is transmitted to the mobile phone, and the process returns to step 4013a. If No in step 4013p, return to step 4013a.
If the result is No in step 4013e, the process proceeds to step 4013f to cause the sound collecting device to start collecting sound and transmit the sound collecting voice to the mobile phone. Then, in step 4013g, the notification sound of the microwave oven is sounded. Then, in step 4013h, the recognition process is performed on the voice sent from the sound collecting device. Then, in step 4013i, it is confirmed whether or not the notification sound can be recognized. Here, if Yes, the process proceeds to 4013j, the collected voice is transmitted to the mobile phone until the next cooking stage is performed, the mobile phone recognizes the notification sound of the microwave oven, and the process returns to step 4013a. If No, return to step 4013a.
(Process h: Notifying the user of the end of operation of the microwave oven) FIG. 158 is a diagram for explaining the process of notifying the user of the end of operation of the microwave oven in the present embodiment. Hereinafter, FIG. 158 will be described.
First, in step 4013a, it is confirmed whether the mobile phone is in use or whether it can be determined from the sensor data that the mobile phone is moving. If Yes, the process proceeds to step 4014m, and the user is notified of the end of the microwave oven operation by using the screen display of the mobile phone, voice, vibration, or the like, and the process ends.
On the other hand, in step 4013a, if No, the process proceeds to step 4014b to search for the device being operated (device being operated by the user) on the PC or the like on which the user is logged in.
Next, in step 4014c, it is confirmed whether or not the device being operated by the user is found. If Yes, the user is notified of the end of the operation of the microwave oven by using the screen display of the device being operated by the user, and the operation is terminated.
On the other hand, in step 4014c, if No, the process proceeds to step 4014e to search for a device (imaging device) capable of communicating from a mobile phone and acquiring an image.
Next, in step 4014f, it is confirmed whether or not the imaging device has been found.
Here, if Yes, the process proceeds to step 4014p, the image pickup device is made to take an image, the user's face data is transmitted to the image pickup device, and the user's face recognition is performed. Alternatively, the captured image is sent to a mobile phone or a server, and the user's face is recognized at the destination of the image.
Then, in step 4014q, it is confirmed whether or not the user's face is recognized. If No, return to step 4014e. If Yes, proceed to step 4014r and check if the device that found the user (discovery device) has a display device or vocalization device. If Yes in step 4014r, the process proceeds to step 4014s, the user is notified of the end of microwave oven operation using the device attached to the device, and the process ends.
If the result is No in step 4014f, the process proceeds to step 4014g to search for a device (sound collecting device) that can communicate from a mobile phone and can collect sound.
If the result is No in step 4014h, the process proceeds to step 4014i to find a device that can specify the user's position by means of other device operations, walking vibration, or the like. Then, the process proceeds to step 4014m, and the user is notified of the end of the operation of the microwave oven by using the screen display of the mobile phone, voice, vibration, or the like, and the process ends.
If Yes in step 4014i, the process proceeds to step 4014r to check whether the device (discovery device) that found the user has a display device or a vocalization device. Here, in the case of No, the process proceeds to step 4014t, and the location information of the discovered device is acquired from the server.
Then, in step 4014u, a device (notification device) that is near the discovery device and has a display device and a vocalization device is searched for. Then, in step 4014v, the distance from the notification device to the user is taken into consideration, and the user is notified of the end of the operation of the microwave oven by the screen display or a voice of sufficient volume, and the process ends.
(Process i: Confirmation of mobile phone operation state) FIG. 159 is a diagram for explaining a process for confirming the mobile phone operation state in the present embodiment. Hereinafter, FIG. 159 will be described.
First, in step 4015a, the mobile phone is operating, the mobile phone is moving, the input / output device connected to the mobile phone has input / output, or video or music is being played, or the mobile phone. Check if the device near the phone is in operation, or if the camera or various sensors of the device near the mobile phone recognize the user.
Here, if Yes, the process proceeds to step 4015b, recognizing that the user's position is likely to be close to this mobile phone, and returns to step 4015a.
On the other hand, if No, the process proceeds to step 4015c, and the device located far from the mobile phone is being operated, or the user is recognized by the camera or various sensors of the device located far from the mobile phone, or charging is in progress. Check if.
If Yes in step 4015c, the process proceeds to step 4015d, recognizing that the user's position is likely to be far from this mobile phone, and returns to step 4015a. If No in step 4015c, the process returns to step 4015a.
(Process j: Tracking User Position) FIG. 160 is a diagram for explaining a process of tracking the user position in the present embodiment. Hereinafter, FIG. 160 will be described.
First, in step 4016a, it is confirmed whether or not the mobile phone is determined to be moved by the directional sensor, the position sensor, or the 9-axis sensor. The 9-axis sensor is a sensor consisting of at least one of an acceleration sensor, an angular velocity sensor, and a geomagnetic sensor.
If Yes in step 4016a, proceed to step 4016b, register the mobile phone and user's location in the DB, and return to step 4016a.
On the other hand, if No in step 4016a, the process proceeds to step 4016c, and a device (user) capable of communicating from a mobile phone and detecting the user's position or existence such as a camera, a microphone, or a motion sensor. Detecting device).
Then, in step 4016d, ask if you want to find a sound collector. In step 4016d, if No, return to step 4016a.
If yes in step 4016d, proceed to step 4016e to see if the user detector will detect the user. If No in step 4016e, return to step 4016a.
If Yes in step 4016e, proceed to step 4016f to send the user's detection to the mobile phone.
Next, in step 4016g, the existence of the user in the vicinity of the user detection device is registered in the DB.
Next, in step 4016h, if there is position information of the user detection device in the DB, it is acquired, the position of the user is specified, and the process returns to step 4016a.
FIG. 161 shows the user position based on the recognition result of the user position by recognizing the notification sound of the home appliance and causing the communicable electronic device to recognize the current position of the user (operator) while canceling the voice from the voice output device. It is a figure which showed that the device at a position close to is notified to a user. Further, FIG. 162 is a diagram showing the contents of a database held in a server, a mobile phone, or a microwave oven according to the present embodiment.
As shown in FIG. 162, the microwave table 4040a recognizes the model of the microwave, the data (speaker characteristics, modulation method, etc.) that specifies the sound that can be output, and the mobile phone for each mobile phone type. The notification sound data having characteristics that are easy to easily recognize and the notification sound data that is easy to recognize on average by a general mobile phone are stored in association with each other.
The mobile phone table 4040b holds each individual mobile phone, the model of the mobile phone, the user who uses the mobile phone, and the data indicating the position of the mobile phone in association with each other.
The mobile phone model table 4040c holds the mobile phone model in association with the sound collecting characteristics of the microphone or the like attached to the mobile phone type.
In the user voice characteristic table 4040d, the user and the acoustic characteristic of the user's voice are associated and held.
In the user keyword voice table 4040e, the voice waveform data when the user utters a keyword to be recognized by the mobile phone such as "next" and "return" is stored in association with the user. It should be noted that this data is not the voice waveform data itself, but may be analyzed and transformed into a form that is easy to handle.
In the user-owned device position table 4040f, the user, the device owned by the user, and the position data of the device are stored in association with each other.
In the user-owned device position table 4040g, the user, the device owned by the user, and the sound data such as the notification sound and the driving sound emitted by the device are stored in association with each other.
In the user position table 4040h, the data of the user and the user's position are stored in association with each other.
Further, in FIG. 163, the user cooks based on the cooking procedure displayed on the mobile phone in the present embodiment, and the user operates the display contents of the mobile phone by voices such as "next" and "return". It is a figure which shows that it does. FIG. 164 shows that the user is moving to another place while waiting for the end of the operation of the microwave oven in the present embodiment, while boiling the simmered food, and the like. It is a figure. FIG. 165 shows a device connected to a mobile phone via a network and capable of recognizing the user's position and the existence of the user, such as a camera, a microphone, and a motion sensor. It is a figure which shows that the instruction to detect a user is transmitted from the mobile phone. FIG. 166 is a diagram showing that, as an example of user detection, the camera attached to the television recognizes the user's face, and the motion sensor of the air conditioner recognizes the movement or existence of the user. It should be noted that this recognition process may be performed by a television or an air conditioner, or image data or the like may be transmitted to a mobile phone or a server and the recognition process may be performed at the destination. However, from the viewpoint of privacy protection, the user's data should not be sent to an external server.
FIG. 167 is a diagram showing that the device that has detected the user has detected the user, and that the relative position from the detected device to the user is transmitted to the mobile phone.
In this way, if the position information of the device that detected the user exists in the DB, it is possible to specify the position of the user.
FIG. 168 is a diagram showing that the mobile phone recognizes the operation end sound of the microwave oven in the present embodiment. FIG. 169 shows a mobile phone that recognizes the end of operation of a microwave oven, among the devices that detect the user, that has a screen display function and a voice output function (in this figure, the TV in front of the user). It is a figure which shows that the instruction to notify the user of the end of operation of a microwave oven is transmitted.
FIG. 170 is a diagram showing that the device that has received the above command notifies the user of the content of the notification (in this figure, the screen of the television indicates that the operation of the microwave oven has been completed). FIG. 171 is a diagram showing that the operation end sound of a microwave oven is recognized by a device that is connected to a mobile phone via a network, has a microphone, and is present in the vicinity of the microwave oven. FIG. 172 is a diagram showing that the device that recognizes the end of operation of the microwave oven notifies the mobile phone of the end of operation of the microwave oven. FIG. 173 shows that when the mobile phone receives the notification of the end of operation of the microwave oven, if the mobile phone is near the user, the user is notified of the end of operation of the microwave oven by using the screen display or voice output of the mobile phone. It is a figure which shows that it does.
FIG. 174 is a diagram showing that the user is notified of the end of operation of the microwave oven. Specifically, FIG. 174 shows the screen display function and the voice output function among the devices detecting the user if the mobile phone is not near the user when the mobile phone receives the notification of the end of operation of the microwave oven. (In this figure, the TV in front of the user) is sent a command to notify the user of the end of microwave oven operation, and the device that receives the command notifies the user of the end of microwave oven operation. It is a figure which shows that. When the mobile phone is connected to the charger, the mobile phone is often not near the microwave oven or near the user, indicating that the situation shown in this figure is likely to occur.
FIG. 175 is a diagram showing that the user who has been notified of the end of operation of the microwave oven moves to the kitchen. At this time, the mobile phone displays the following cooking contents. In addition, the mobile phone may recognize that the user has moved to the kitchen by voice or the like, and may start explaining the next cooking procedure at just the right time.
In FIG. 176, information such as the end of operation is transmitted from the microwave to the mobile phone by wireless communication, a notification command is given from the mobile phone to the TV that the user is watching, and the user is notified by the screen display or voice of the TV. It is a figure which shows what to do.
Communication between the information source device (microscope in this figure) and the mobile phone, and communication between the mobile phone and the device that notifies the user (television in this figure) are via home LAN or direct wireless communication. In particular, wireless communication in the 700MHz to 900MHz range can be used. Further, although the mobile phone is used as a hub here, another device having communication capability may be used instead of the mobile phone.
FIG. 177 is a diagram showing that information such as the end of operation is transmitted from a microwave oven to a television watched by the user by wireless communication, and the user is notified by using the screen display or voice of the television. This shows the operation when the mobile phone, which has played the role of a hub in FIG. 176, is not used.
FIG. 178 shows that when the air conditioner on the first floor gives some information notification, the information is transmitted to the air conditioner on the second floor, the information is transmitted from the air conditioner on the second floor to the mobile phone, and the mobile phone sends the information to the TV the user is watching. It is a figure which shows that the notification command is given, and the user is notified by the screen display of a television, or by voice. This is to send information to other devices that can communicate and communicate to the mobile phone when the information source device (air conditioner on the first floor in this figure) cannot directly communicate with the mobile phone that is the hub. Is shown.
FIG. 179 is a diagram showing that information is notified to a user at a remote location. Specifically, FIG. 179 shows that a mobile phone notified from a microwave oven via voice, light, wireless communication, etc. notifies a user at a remote location via the Internet or carrier communication. ing. FIG. 180 is a diagram showing that information is transmitted to a mobile phone via a personal computer or the like when direct communication cannot be performed from the microwave oven to the mobile phone serving as a hub. FIG. 181 is a diagram showing that information such as an operation command is transmitted from the mobile phone that has received the communication of FIG. 180 to the microwave oven by tracing the information communication path in the reverse direction.
The mobile phone may receive the information shown in FIG. 180 and automatically transmit the information, or may notify the user of the information and transmit the information of the operation performed by the user in response to the notification. ..
FIG. 182 is a diagram showing that information is notified to the user when the air conditioner, which is an information source device, cannot directly communicate with the mobile phone as a hub. Specifically, FIG. 182 shows the step of communication from an air conditioner, which is an information source device, to a mobile phone such as a personal computer with a circle 1 when direct communication to a mobile phone as a hub is not possible. Send information to the device that becomes, and send information from the personal computer to the mobile phone through the Internet or carrier communication network with circle 2 and circle 3, and the mobile phone automatically processes the information or holds the mobile phone. In response to the operation of the existing user, circle 4 and circle 5 send information to the personal computer via the Internet or carrier communication network, and circle 6 allows the personal computer to notify the user who wants to be notified (TV in this figure). A notification command is sent to the user, and circle 7 indicates that the user is notified of the information using the screen display or voice of the television.
Such a situation is likely to occur when the user who should receive the notification information from the air conditioner and the user who uses the mobile phone are different.
In this figure, the communication between the personal computer and the mobile phone is via the Internet or a carrier communication network, but communication may be performed by a home LAN or direct communication.
FIG. 183 is a diagram for explaining a system using a communication device using radio waves of 700 to 900 MHz. Specifically, in the configuration of FIG. 183, a system using a communication device (hereinafter referred to as G device) using a radio wave of 700 to 900 MHz (hereinafter referred to as G radio wave) is described, and an electron with a G device is described. Information is transmitted from the range to the mobile phone on the 3rd floor with the G device using G radio waves, and from the mobile phone on the 3rd floor with the G device to the mobile phone on the 2nd floor without the G device using the home network. It indicates that the information will be transmitted and the information will be notified to the user from the mobile phone on the second floor.
For registration / authentication of communication between devices with G device, a method using NFC attached to both devices can be considered. In addition, if one of the devices does not have NFC, as a registration mode, the output of G radio waves is lowered so that communication is possible only at a distance of about 10 to 20 cm, and both devices are brought closer to each other, and communication is successful. In this case, a method of registering / authenticating communication between G devices can be considered.
Further, the information source device (microwave oven in this figure) may be a device other than the microwave oven as long as it is equipped with the G device.
In addition, the device that relays the information source device and the information notification device (mobile phone on the 2nd floor in this figure) (mobile phone on the 3rd floor in this figure) should be portable if it has access to G radio waves and the home network. Instead of a telephone, a device such as a personal computer, an air conditioner, or a smart meter may be used.
Further, the information notification device may be a device such as a personal computer or a television instead of a mobile phone as long as it can access the home network and can notify the user by using a screen display, voice output, or the like.
FIG. 184 is a diagram showing that a mobile phone in a remote place notifies a user of information. Specifically, FIG. 184 shows that an air conditioner equipped with a G device transmits information to a mobile phone in the house equipped with the G device, and the information is transmitted from the mobile phone in the house to a mobile phone in a remote location through the Internet or a carrier communication network. It indicates that the information is transmitted and the information is notified to the user by a mobile phone in a remote place.
The information source device (air conditioner in this figure) may be a device other than a microwave oven as long as it is equipped with a G device.
In addition, if the device that relays the information source device and the information notification device (mobile phone in the remote location in this figure) (mobile phone in the house in this figure) can access the G radio wave and the Internet or carrier communication network, Instead of a mobile phone, a device such as a personal computer, an air conditioner, or a smart meter may be used.
The information notification device may be a device such as a personal computer or a television instead of a mobile phone as long as it can access the Internet and a carrier communication network and can notify the user by using a screen display or voice output. ..
FIG. 185 is a diagram showing that a mobile phone in a remote place notifies a user of information. Specifically, in FIG. 185, the TV with the G device recognizes the notification sound of the microwave without the G device, and the TV sends information to the mobile phone in the house with the G device via the G radio wave, and the inside of the house. It shows that information is transmitted from a mobile phone in a remote location to a mobile phone in a remote location via the Internet or a carrier communication network, and the mobile phone in the remote location notifies the user of the information.
The information source device (microwave oven in this figure) is the same for other devices, and the method of recognizing the notification of the information source device by the notification recognition device (television in this figure) is not voice but light emission state. The same applies even if it is the same.
The same applies to the notification recognition device even if it is another device with a G device. In addition, if the device that relays the notification recognition device and the information notification device (mobile phone in the remote location in this figure) (mobile phone in the house in this figure) can access the G radio wave and the Internet or carrier communication network, Instead of a mobile phone, a device such as a personal computer, an air conditioner, or a smart meter may be used.
The information notification device may be a device such as a personal computer or a television instead of a mobile phone as long as it can access the Internet and a carrier communication network and can notify the user by using a screen display or voice output. ..
In the same case as in FIG. 185, FIG. 186 shows a device for relaying a notification recognition device (television on the second floor in this figure) and an information notification device (mobile phone in a remote place in this figure) (inside the house in FIG. 185). It is a figure when the TV on the second floor plays the role of a relay device instead of the mobile phone.
As described above, the apparatus of this embodiment realizes the following functions.
-A function to learn the voice characteristics of the user through the use of the application-A function to find a sound collecting device that can collect the sound emitted from the mobile phone among the devices that can communicate from the mobile phone and have a sound collecting function-Mobile A function to find a sound collecting device that can collect sound emitted from an electronic device among devices that can communicate from a telephone and have a sound collecting function The sound collected by the sound collecting device can be used as it is, or the sound recognition result can be obtained. Function to send to mobile phone Function to analyze the characteristics of environmental sound and improve the accuracy of voice recognition Function to acquire the sound that can be output from the device owned by the user from the DB and improve the accuracy of voice recognition Mobile A function to find a sound output device that can be communicated from a telephone and has a sound output function, and a mobile phone or a sound collecting device can collect the sound emitted by the device Sound data output from the sound output device Function to cancel unnecessary sound from sound collection sound by acquiring and subtracting from sound collection sound in consideration of transmission characteristics Controls cooking procedure and cooking equipment that receives parameter input of cooking recipe and gives instructions to user Function to acquire control data from the server Function to set the notification sound emitted by the device based on the sound data that can be output by the device so that it can be easily recognized by mobile phones and sound collecting devices User's voice A function to improve the recognition accuracy of the user's sound by adjusting the recognition function based on the characteristics A function to recognize the user's sound using multiple sound collecting devices Notification of electronic devices using multiple sound collecting devices Sound recognition function-Get the necessary information from the electronic device and set it in the microwave to perform a series of operations with just one operation via the non-contact IC card of the mobile phone and the electronic device. Function Use a device that can communicate from a mobile phone and can detect the user, such as a camera, microphone, or human sensor, search for the user, send the user's current position to the mobile phone, or send it to the DB. Function to save Function to notify the user from a nearby device using the user position saved in the DB Mobile from the state of the mobile phone (operation state, sensor value, charging state, data link state, etc.) Near the phoneAbility to estimate if a user exists
In the processing of FIGS. 145 to 175, the same function can be realized even if the sound data is read as light emission data (frequency, luminance, etc.), the sound output is read as light emission, and the sound collection is read as light reception.
Further, although a microwave oven is taken as an example in this embodiment, the electronic device that emits the notification sound to be recognized is not a microwave oven, but a washing machine, a rice cooker, a vacuum cleaner, a refrigerator, an air purifier, a pot, and a dishwasher / dryer. , Air conditioners, personal computers, mobile phones, TVs, automobiles, telephones, mail receiving devices, etc. have the same effect.
Further, in the present embodiment, the microwave oven, the mobile phone, and the device for notifying the user such as a television are directly communicated with each other, but if the direct communication is inconvenient, another device is used. Communication may be performed indirectly through.
Further, in the present embodiment, communication mainly using a home LAN is assumed, but the same applies to direct wireless communication between devices, communication via the Internet, or a carrier communication network. The function is feasible.
Further, in the present embodiment, the location of the user is identified by a TV camera or the like by a simultaneous inquiry from the mobile phone, and encrypted and sent to the mobile phone of the user, so that personal information is leaked. It has the effect of preventing it. Even if there are multiple people in the house, by storing the data of the motion sensors of the air conditioner, air purifier, and refrigerator in the position management database of the mobile device, etc., the sensor can be used as the operator moves once. By tracking, the position of the operator can be estimated.
If the user has a mobile phone with a gyro or a directional meter, the specified position data may be registered in the user position database.
Further, when the operator puts down the mobile phone, the movement of the physical sensor is first stopped for a certain period of time, so that this can be detected. The next distance is detected by using motion sensors and button operations of home appliances and lighting equipment, cameras such as TVs, and microphones of mobile phones. Then, the location is registered in the user location database of the mobile phone or the server in the house.
As described above, according to the ninth embodiment, it is possible to realize an information communication device (recognition device) that enables communication between devices.
Specifically, the information communication device of the present embodiment searches for an electronic device (sound collecting device) having a sound collecting function among the electronic devices that can communicate from the operation terminal, and the sound collecting function of the sound collecting device. It may include a recognition device that recognizes the notification sound of another electronic device by using the above.
Here, the recognition device may be a recognition device that uses only the sound collecting function of the sound collecting device capable of collecting the signal sound emitted from the operation terminal.
Further, the information communication device of the present embodiment searches for an electronic device (voice output device) having a voice output function among the electronic devices that can communicate from the operation terminal, and the voice output device and the sound collecting device are used. Sound collection that analyzes the audio transmission characteristics between the two, acquires the output audio data of the audio output device, and cancels the audio output from the audio output device from the audio collection audio from the audio transmission characteristics and the output audio data. The device may be included.
Further, the information communication device of the present embodiment may include a recognition device for adjusting the notification sound of the electronic device for which the notification sound is desired to be recognized so as not to be buried in the environmental sound.
Further, the information communication device of the present embodiment holds the electronic device (owned electronic device) owned by the user, the sound data output by the owned electronic device, and the position data of the owned electronic device in a database, and holds the possessed electronic device. A recognition device for adjusting the notification sound of the electronic device to be recognized may be included so that the output voice and the notification sound of the electronic device for which the notification sound is to be recognized can be easily distinguished.
Here, the recognition device may be a recognition device that adjusts the voice recognition process so that the voice output by the possessed electronic device and the notification sound of the electronic device for which the notification sound is to be recognized can be easily distinguished.
Further, in the information communication device of the present embodiment, the positions of the operation terminal and the operator are close to each other by using the operation state of the operation terminal, the sensor value of the physical sensor, the data link state, and the charging state. It may include a recognition device that recognizes whether or not.
Here, the recognition device further utilizes the operation state of the electronic device communicable from the operation terminal, the camera, the microphone, the motion sensor, and the position data of the electronic device held in the database. It may be a recognition device that recognizes the position of the user.
Further, the recognition device is an electronic device (notification device) having a function of notifying the user by means such as screen display and voice output, which is stored in the database and the recognition result of the user position. It may be included in an information notification device that notifies a user of information by using the notification device capable of notifying the user by using the position data.
It should be noted that these general or specific embodiments may be realized by a system, a method, an integrated circuit, a computer program or a recording medium, and may be realized by any combination of a system, a method, an integrated circuit, a computer program or a recording medium. May be done.
(Embodiment 10) Currently, various simple authentication methods are being studied in wireless communication. For example, in the wireless LAN WPS established by the Wi-Fi Alliance, the push button method, PIN input method, NFC method, etc. are specified. Various simple authentication methods in wireless can determine whether the user using the device is trying to authenticate by limiting the time or identifying that they are at a distance where they can directly touch both devices. Judgment and authentication are performed.
However, the method of limiting the time is not safe when there is a malicious user at a certain close distance. In addition, in stationary equipment such as home appliances, it may be difficult or troublesome to touch them directly.
Therefore, in the present embodiment, by using the communication using visible light for wireless authentication, it is specified that the user who is trying to authenticate is surely in the room, and the wireless authentication of the home electric appliance is simplified. And explain how to do it safely.
FIG. 187 is a diagram showing an example of the environment in the house according to the present embodiment. FIG. 188 is a diagram showing an example of communication between a home electric appliance and a smartphone according to the present embodiment. FIG. 189 is a diagram showing a configuration of a transmitting side device according to the present embodiment. FIG. 190 is a diagram showing a configuration of a receiving side device according to the present embodiment. FIGS. 187 to 190 are the same drawings as those of FIGS. 124 to 127, and detailed description thereof will be omitted.
As the home environment, consider an environment that authenticates the tablet that the user has in the kitchen and the TV in the living room, as shown in FIG. 187. Both are terminals that can be connected to wireless LAN, and it is assumed that the WPS module is installed.
FIG. 191 is a sequence diagram in FIG. 187 when the transmitting side terminal (TV) performs wireless LAN authentication with the receiving side terminal (tablet) by using optical communication. Hereinafter, FIG. 191 will be described.
First, for example, the transmitting terminal as shown in FIG. 189 creates a random number (step 5001a). Then register with the WPS registrar (step 5001b). Further, the light emitting element is made to emit light according to the random number pattern registered in the registrar (step 5001c).
On the other hand, while the light emitting element of the transmitting side device emits light, for example, in the receiving side device as shown in FIG. 190, the camera is activated in the optical authentication mode. Here, the optical authentication mode is a mode in which it is possible to recognize that the light emitting element is shining for authentication, and refers to a moving image shooting mode in which shooting can be performed together with the cycle on the light emitting side.
That is, first, the user photographs the light emitting element of the transmitting terminal (step 5001d). Next, the receiving terminal receives a random number by shooting (step 5001e). Then, the receiving terminal that receives the random number inputs the random number as the PIN of WPS (step 5001f).
Here, the transmitting / receiving terminal whose PIN is shared by both parties performs the authentication process according to the WPS regulations (step 5001g).
Next, when authentication is complete, the sending terminal removes the random number from the registrar so that it does not accept authentication from multiple terminals (5001h).
This method is not limited to wireless LAN authentication, but can be applied to all wireless authentication methods using a shared key.
Further, this method is not limited to the wireless authentication method. For example, it can be applied to the authentication of applications installed between both TVs and tablets.
FIG. 192 is a sequence diagram in the case of performing authentication in the application according to the present embodiment. Hereinafter, FIG. 192 will be described.
First, the sender terminal creates a sender ID according to the state of the terminal (step 5002a). Here, the sender ID may be a random number or a key for encryption. Further, the terminal ID (MAC address, IP address) of the transmitting terminal may be included. Subsequently, the transmitting terminal emits light according to the pattern of the transmitting ID (step 5002b).
On the other hand, the receiving side device receives the transmitting side ID in the same procedure as in the case of wireless authentication (step 5002f). Then, when the receiving device receives the transmitting ID, it creates a receiving ID that can prove that the transmitting ID has been received (step 5002g). For example, it may be the terminal ID of the receiving terminal encrypted with the transmitting ID. It may also include the process ID and password of the application running on the receiving terminal. Subsequently, the receiving terminal wirelessly broadcasts the receiving ID (step 5002h). If the sender ID includes the terminal ID of the sender terminal, unicast may be used.
Next, the 5002c sender terminal that has received the receiver ID wirelessly performs authentication using the sender ID that is shared with the terminal that sent the received receiver ID (step 5002d).
FIG. 193 is a flowchart showing the operation of the transmitting terminal in the present embodiment. Hereinafter, FIG. 193 will be described.
First, the transmitting terminal emits an ID according to the state of the terminal (step 5003a).
Next, it emits light in a pattern according to the ID (step 5003b).
Next, check if there is a radio response corresponding to the emitted ID (step 5003c). If there is a response (Yes in step 5003c), the responding terminal is authenticated (step 5003d). If there is no response in step 5003c, the time-out time is waited (step 5003i), no response is displayed, and the process ends (step 5003j).
Next, check if the authentication process was successful in step 5003e, and if the authentication process is successful (Yes in step 5003e), if the emitted ID contains a command other than authentication (Yes in step 5003f). ), Process according to the command (step 5003g).
If authentication fails in step 5003e, an authentication error is displayed (step 5003h) and the process ends.
FIG. 194 is a flowchart showing the operation of the receiving terminal in the present embodiment. Hereinafter, FIG. 194 will be described.
First, the receiving terminal activates the camera in optical authentication mode (step 5004a).
Next, it is confirmed whether the light can be received in a specific pattern (step 5004b), and if it is confirmed (Yes in step 5004b), a receiver ID that can prove that the sender ID has been received is created (step 5004b). Step 5004c). If it cannot be confirmed (No in step 5004b), wait for the timeout time (Yes in step 5004i), display the timeout (Step 5004j), and end.
Next, check if the sender terminal contains the sender ID (step 5004k), and if so (Yes in step 5004k), unicast the receiver ID to that terminal (step 5004k). 5004d). On the other hand, if it is not included (No in step 5004k), it will be broadcast (step 5004l).
Next, the authentication process is started from the transmitting terminal side (step 5004e), and if the authentication process is successful (Yes in step 5004e), it is determined whether the received ID contains a command (step 5004f). .. If it is determined in step 5004f that it was included (YES in step 5004f), processing according to the ID is performed (step 5004g).
If authentication fails in step 5004e (No in step 5004e), an authentication error is displayed (step 5004h), and the process ends.
As described above, according to the present embodiment, by using the communication using visible light for wireless authentication, it is specified that the user who is trying to authenticate is surely in the room, and the wireless authentication of the home electric appliance is performed. It can be done easily and safely.
(Embodiment 11) In the above-described embodiment, the flow for exchanging data using NFC communication and high-speed wireless communication has been described, but the present invention is not limited to this. Of course, the present embodiment can have a flow as shown in FIGS. 195 to 197.
FIG. 195 is a sequence diagram in which the mobile AV terminal 1 in the present embodiment transmits data to the mobile AV terminal 2. Specifically, FIG. 195 shows a sequence diagram of data transmission / reception using NFC / wireless LAN wireless communication. Hereinafter, FIG. 195 will be described.
The mobile AV terminal 1 first displays the data to be transmitted to the mobile AV terminal 2 on the screen.
Here, when the mobile AV terminal 1 and the mobile AV terminal 2 perform NFC communication by contacting each other's devices, the mobile AV terminal 1 displays a confirmation screen on the screen as to whether or not data transmission is performed. This confirmation screen may ask the user to select "Yes / No" along with the text "Do you want to send data?", Or by touching the screen of the mobile AV terminal 1 again. The interface may be such that data transmission is started.
If the confirmation of data transmission intention is OK, the mobile AV terminal 1 and the mobile AV terminal 2 exchange information on the data to be transmitted and information for establishing high-speed wireless communication by NFC communication. The information of the data to be transmitted may be performed by wireless LAN communication. The information related to the establishment of wireless LAN communication may be a communication channel, SSID, or encryption key information, or a method of exchanging randomly generated ID information and establishing a secure communication path by this information. Is also good.
When the wireless LAN communication is established, the mobile AV terminal 1 and the mobile AV terminal 2 perform data communication by wireless LAN communication, and transmit the data to be transmitted by the mobile AV terminal 1 to the mobile AV terminal 2.
Next, with reference to FIGS. 196 and 197, the screen transitions of the mobile AV terminal 1 and the mobile AV terminal 2 will be mainly described. FIG. 196 is a screen transition diagram when the mobile AV terminal 1 in the present embodiment transmits data to the mobile AV terminal 2. FIG. 197 is a screen transition diagram when the mobile AV terminal 1 in the present embodiment transmits data to the mobile AV terminal 2.
In FIGS. 196 and 197, on the mobile AV terminal 1, the user first activates an application for playing a moving image and a still image. This app displays still images and video data in the mobile AV terminal 1.
Here, NFC communication is performed by quasi-contacting the mobile AV terminal 1 and the mobile AV terminal 2. This NFC communication is a process for starting the exchange of still image and video data on the mobile AV terminal 1.
First, when the mobile AV terminal 1 and the mobile AV terminal 2 recognize the start of data exchange by NFC communication, a confirmation screen asking whether data can be transmitted is displayed on the screen of the mobile AV terminal 1. As shown in FIG. 196, this confirmation screen may be an interface that allows the user to touch the screen at the start of data transmission, or may be an interface that allows the user to select whether or not to transmit data with Yes / No. If Yes in the data transmission start judgment, that is, when the mobile AV terminal 1 transmits data to the mobile AV terminal 2, the mobile AV terminal 1 starts high-speed wireless communication by wireless LAN with the data information to be exchanged with the mobile AV terminal 2. Send information about. The information of the data to be exchanged may be performed by using high-speed wireless communication.
Next, when the mobile AV terminal 1 and the mobile AV terminal 2 send and receive information regarding the start of high-speed wireless communication by wireless LAN, they perform a process for establishing a wireless LAN communication connection. This process includes exchange of password information, mutual SSID and terminal information, etc., which channel is used for communication, which is the parent terminal and which is the child terminal in terms of communication topology.
Next, when the wireless LAN communication connection is established, the mobile AV terminal 1 and the mobile AV terminal 2 transmit data by wireless LAN communication. During data transmission, the mobile AV terminal 1 displays the video playback screen as usual, and the mobile AV terminal 2 on the data receiving side displays a screen indicating that data is being received. This is because when the screen during data transmission is displayed for the mobile AV terminal 1, other processing cannot be performed. Therefore, the data transmission is performed in the background, which has the advantage of improving the convenience of the user. In addition, the mobile AV terminal 2 is receiving data, and by displaying a screen indicating that data is being received on the screen so that the received data can be displayed immediately, there is an advantage that the received data can be displayed immediately when the data reception is completed. There is.
Finally, the mobile AV terminal 2 displays the received data on the screen when the data reception is completed.
FIGS. 198 to 200 are schematic system diagrams when the mobile AV terminal 1 in the present embodiment is a digital camera.
As shown in FIG. 198, it goes without saying that the mobile phone of this implementation is applicable even if the mobile AV terminal 1 is a digital camera.
Further, when the mobile AV terminal 1 is a digital camera, in general, the digital camera has a means of Internet access by wireless LAN but does not have a means of Internet access by mobile communication in many cases.
Therefore, as shown in FIGS. 199 and 200, the digital camera (mobile AV terminal 1) transmits the image data taken by the wireless LAN to the photo sharing service in an environment where wireless LAN communication can be performed. In an environment where wireless LAN communication is not possible, first send data to mobile AV terminal 2 using wireless LAN, and mobile AV terminal 2 should be configured to send the received data as it is to the photo sharing service by mobile phone communication. Is desirable.
Since wireless LAN communication is faster than mobile phone communication, if wireless LAN communication is possible, it is possible to send photos to the photo sharing service at high speed by performing wireless LAN communication. In addition, since the service area of a mobile phone communication network is generally wider than that of a wireless LAN communication network, if there is no wireless LAN environment, the mobile AV terminal 2 is relayed and data is transmitted to the photo sharing service by mobile phone communication. By having a function that can be used, it will be possible to instantly send photos to a photo sharing service in various places.
As described above, according to the present embodiment, data can be exchanged using NFC communication and high-speed wireless communication.
Although the information communication device and the like according to one or more aspects of the present invention have been described above based on the embodiments, the present invention is not limited to the embodiments. As long as it does not deviate from the gist of the present invention, one or more of the present embodiments may be modified by those skilled in the art, or may be constructed by combining components in different embodiments. It may be included within the scope of the embodiment.
In each of the above embodiments, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.
(Embodiment 12) In the present embodiment, each application example using a receiver such as a smartphone in the above-described first to 11th embodiments and a transmitter for transmitting information as an LED blinking pattern will be described.
FIG. 201 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the transmitter 7001a, which is a signage of a restaurant, transmits the identification information (ID) of the transmitter 7001a to the receiver 7001b configured as, for example, a smartphone. The receiver 7001b acquires the information associated with the ID from the server and displays it. The information is, for example, the route to the restaurant, seat availability, and coupons.
FIG. 202 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the transmitter 7042b, which is a signage of a movie, transmits the identification information (ID) of the transmitter 7042b to the receiver 7042a configured as, for example, a smartphone. The receiver 7042a acquires the information associated with the ID from the server and displays it. The information is, for example, an image 7042c prompting a movie reservation, an image 7042d indicating a movie show time, an image 7042e indicating a seat availability, and an image 7042f notifying the reservation completion.
FIG. 203 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the transmitter 7043b, which is a signage of a drama, transmits the identification information (ID) of the transmitter 7043b to the receiver 7043a configured as, for example, a smartphone. Upon receiving the ID, the receiver 7043a acquires the information associated with the ID from the server and displays it. The information is, for example, the image 7043c prompting the recording reservation of the drama, the image 7043d prompting the selection of the recorder for recording the drama, and the image 7043e notifying the completion of the reservation.
FIG. 204 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the transmitter 7044d or 7044c, which is a signage configured as a roof sign of a store or a sign placed on the street, transmits the identification information (ID) of the transmitter 7044d or 7044c to a receiver 7044a configured as, for example, a smartphone. .. The receiver 7044a acquires the information associated with the ID from the server and displays it. The information is, for example, image 7044b showing the availability of the store, coupons, and the like.
FIG. 205 is a flowchart showing an example of the processing operation of the receiver and the transmitter in the twelfth embodiment. It should be noted that this flowchart corresponds to the application example shown in FIGS. 201 to 204.
First, the ID of the transmitter is associated with the information to be passed to the receiver that received the ID and stored in the server (7101a). Here, the information passed to the receiver includes the store name, product name, map information to the store, vacant seat information, coupon information, product inventory quantity, movie or drama screening time, and reservation information. It may also include information such as the URL of the server that makes the reservation or purchase.
Next, the ID is transmitted from the transmitter (S7101b). Point the camera of the receiver at the transmitter and receive the ID (S7101c).
The receiver sends the received ID to the server and stores the information related to the ID in the receiver (S7101d).
The receiver saves the terminal ID and user ID together in the server (S7101e). In addition, the receiver displays the information stored in the server for display on the receiver (S7101f).
The receiver adjusts the display contents based on the user profile stored in the receiver or server (S7101g). For example, control is performed such as changing the size of characters, hiding age-restricted content, and preferentially displaying content presumed to be preferred from past user behavior.
The receiver displays directions from the current location to the store or merchandise store (S7101h). The receiver acquires information from the server as appropriate, updates and displays vacant seat information and reservation information (S7101i). The receiver displays a button to save the acquired information and a button to cancel saving the displayed contents (S7101j).
The user taps the button to save the information acquired by the receiver (S7101k). The receiver saves the acquired information so that it can be redisplayed by the user's operation (S7101m). The store reader reads the information transmitted by the receiver (S7101n). Transmission methods include visible light communication, communication via wifi and bluetooth, and communication using a two-dimensional bar code. The transmission information may include a receiver ID and a user ID.
The store reader stores the read information and the store ID in the server (S7101p). The server stores the transmitter, the receiver, and the store in association with each other (S7101q). This makes it possible to analyze the advertising effectiveness of signage.
FIG. 206 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the transmitter 7002a, which is a signage indicating a plurality of stores, transmits the identification information (ID) of the transmitter 7002a to the receiver 7002b configured as, for example, a smartphone. When the receiver 7002b receives the ID, it acquires the information associated with the ID from the server and displays the same display content as the signage. When the user makes a tap selection or voice input to select a desired store, the receiver 7002b displays the contents of that store.
FIG. 207 is a flowchart showing an example of the processing operation of the receiver 7002b and the transmitter 7002a in the twelfth embodiment.
The ID of the transmitter 7002a is associated with the information passed to the receiver 7002b that received the ID and stored in the server (7102a). The information passed to the receiver 7002b includes the store name, product name, map information to the store, vacant seat information, coupon information, product inventory, movie and drama show times, reservation information, and so on. It may include information such as the URL of the server that makes the reservation or purchase. The positional relationship of the contents displayed on the transmitter 7002a is stored in the server.
The ID is transmitted from the transmitter 7002a such as signage (7102b). Point the camera of receiver 7002b at transmitter 7002a and receive the ID (7102c). Receiver 7002b sends the received ID to the server to get the information related to the ID (7102d). Receiver 7002b displays the information stored in the server for display on receiver 7002b (7102e). The image that is the information may be displayed on the receiver 7002b while maintaining the positional relationship of the images displayed on the transmitter 7002a.
The user selects the information displayed on the receiver 7002b by tapping the screen or specifying by voice (7102f). The receiver 7002b displays the details of the information specified by the user (7102g).
FIG. 208 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the transmitter 7003a, which is a signage indicating a plurality of stores, transmits the identification information (ID) of the transmitter 7003a to the receiver 7003b configured as, for example, a smartphone. When the receiver 7003b receives the ID, it acquires the information associated with the ID from the server, and the part of the signage display content near the center of the image taken by the camera of the receiver 7003b (for example, the closest part). ) Is displayed.
FIG. 209 is a flowchart showing an example of the processing operation of the receiver 7003b and the transmitter 7003a in the twelfth embodiment.
The ID of the transmitter 7003a is associated with the information passed to the receiver 7003b that received the ID and stored in the server (7103a). The information passed to the receiver 7003b includes store name, product name, map information to the store, vacant seat information, coupon information, product inventory, movie and drama screening time, reservation information, and so on. It may include information such as the URL of the server that makes the reservation or purchase. The positional relationship of the contents displayed on the transmitter 7003a is stored in the server.
The ID is transmitted from the transmitter 7003a such as signage (7103b). Point the camera of receiver 7003b at transmitter 7003a and receive the ID (7103c). Receiver 7003b sends the received ID to the server to get the information related to the ID (7103d). The receiver 7003b displays the content displayed in the signage that is positionally closest to the center or the designated part of the captured image (7103e).
FIG. 210 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the transmitter 7004a, which is a signage indicating a plurality of stores, transmits the identification information (ID) of the transmitter 7004a to the receiver 7004b configured as, for example, a smartphone. When the receiver 7004b receives the ID, it acquires the information associated with the ID from the server, and the contents of the signage display contents near the center of the image taken by the camera of the receiver 7004b (for example, a store). An image showing the contents of "B Cafe") is displayed. Here, when the user flicks to the left, the receiver 7004b displays an image showing the contents of the store "C bookstore" on the right side of the store "B cafe" in the signage. That is, the receiver 7004b displays an image having the same positional relationship as the source signage.
FIG. 211 is a flowchart showing an example of the processing operation of the receiver 7004b and the transmitter 7004a in the twelfth embodiment.
The ID of the transmitter 7004a is associated with the information passed to the receiver 7004b that received the ID and stored in the server (7104a). The information passed to the receiver 7004b includes store name, product name, map information to the store, vacant seat information, coupon information, product inventory, movie and drama screening time, reservation information, and so on. It may include information such as the URL of the server that makes the reservation or purchase. The positional relationship of the contents displayed on the transmitter 7004a is stored in the server.
The ID is transmitted from a transmitter 7004a such as signage (7104b). Point the camera of receiver 7004b at transmitter 7004a and receive the ID (7104c). Receiver 7004b sends the received ID to the server to get the information related to the ID (7104d). Display information stored on the server for display on receiver 7004b (7104e).
The user performs a flick operation on the receiver 7004b (7104f). The receiver 7004b changes the display contents so as to have the same positional relationship as the information displayed on the transmitter 7004a according to the user operation (7104g). For example, if the user flicks the screen to the left to display the right side of the currently displayed part, it will be displayed on the right side of the content currently displayed on the receiver 7004b in the display of the transmitter 7004a. The contents are displayed on the receiver 7004b.
FIG. 212 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the transmitter 7005a, which is a signage indicating a plurality of stores, transmits the identification information (ID) of the transmitter 7005a to the receiver 7005b configured as, for example, a smartphone. When the receiver 7005b receives the ID, the receiver 7005b acquires the information associated with the ID from the server, and the contents of the signage display contents near the center of the image taken by the camera of the receiver 7005b (for example, a store). An image showing the contents of "B Cafe") is displayed. Here, when the user taps the left edge (or left-pointing arrow) of the screen of the receiver 7005b, the receiver 7005b displays an image showing the contents of the store "A restaurant" on the left side of the store "B cafe" in the signage. do. Also, when the user taps the bottom edge (or down arrow) of the screen of the receiver 7005b, the receiver 7005b displays an image showing the contents of the store "E office" under the store "B cafe" on the signage. indicate. Also, when the user taps the right edge (or right-pointing arrow) of the screen of the receiver 7005b, the receiver 7005b displays an image showing the contents of the store "C bookstore" on the right side of the store "B cafe" in the signage. .. That is, the receiver 7004b displays an image having the same positional relationship as the source signage.
FIG. 213 is a flowchart showing an example of the processing operation of the receiver 7005b and the transmitter 7005a in the twelfth embodiment.
The ID of the transmitter 7005a is associated with the information passed to the receiver 7005b that received the ID and stored in the server (7105a). The information passed to the receiver 7005b includes store names, product names, map information for stores, vacant seat information, coupon information, product inventory, movie and drama screening times, reservation information, and so on. It may include information such as the URL of the server that makes the reservation or purchase. The positional relationship of the contents displayed on the transmitter 7005a is stored in the server.
The ID is transmitted from a transmitter 7005a such as signage (7105b). Point the camera of receiver 7005b at transmitter 7005a and receive the ID (7105c). Receiver 7005b sends the received ID to the server to get the information related to the ID (7105d). Display information stored on the server for display on receiver 7005b (7105e).
The user taps the edge part displayed on the receiver 7005b or the part indicating the up, down, left, and right directions displayed on the receiver 7005b (7105f). The receiver 7005b changes the display contents so as to have the same positional relationship as the information displayed on the transmitter 7005a according to the user operation (7105g). For example, when the user taps the right edge of the screen or the part indicating the right direction, the content displayed on the right side of the content currently displayed on the receiver 7005b in the display of the transmitter 7005a is changed to the receiver 7005b. Display.
FIG. 214 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment. Note that FIG. 214 is a view of the automobile from behind.
For example, a transmitter (car) 7006a having two tail lamps (light emitting part or light) of a car transmits the identification information (ID) of the transmitter 7006a to a receiver configured as, for example, a smartphone. When the receiver receives the ID, it acquires the information associated with the ID from the server. For example, the information can be the ID of the car or transmitter, the distance between the light emitting parts, the size of the light emitting part, the size of the car, the shape of the car, the weight of the car, the number of the car, the appearance in front, or the danger. This is information indicating the presence or absence of. The receiver may also obtain this information directly from the transmitter 7006a.
FIG. 215 is a flowchart showing an example of the processing operation of the receiver and the transmitter 7006a in the twelfth embodiment.
The ID of the transmitter 7006a is associated with the information to be passed to the receiver that received the ID and stored in the server (7106a). The information passed to the receiver includes the size of the light emitting part that becomes the transmitter 7006a, the distance between the light emitting parts, the shape of the object that has the transmitter 7006a as a component, the weight, the vehicle body number, etc. Information such as the number, the state of a place that is difficult to observe from the receiver, and the presence or absence of danger may be included.
Transmitter 7006a transmits the ID (7106b). The transmission content may include the URL of the server and information stored in the server.
The receiver receives information such as the transmitted ID (7106c). The receiver acquires the information associated with the received ID from the server (7106d). The receiver displays the received information and the information obtained from the server (7106e).
The receiver is a receiver and a light emitting part in the manner of triangulation from the size information of the light emitting part and the visible size of the imaged light emitting part, or from the distance information between the light emitting parts and the distance between the imaged light emitting parts. Calculate the distance to (7106f). The receiver warns of danger based on information such as the state of places that are difficult to observe from the receiver and the presence or absence of danger (7106g).
FIG. 216 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, a transmitter (car) 7007b having two tail lamps (light emitting part or light) of a car transmits information of the transmitter 7007b to a receiver 7007a configured as, for example, a transmitter / receiver in a parking lot. The information on the transmitter 7007b indicates the identification information (ID) of the transmitter 7007b, the number of the car, the size of the car, the shape of the car, or the weight of the car. Upon receiving the information, the receiver 7007a transmits parking availability, billing information, or parking position. The receiver 7007a may receive the ID and acquire information other than the ID from the server.
FIG. 217 is a flowchart showing an example of the processing operation of the receiver 7007a and the transmitter 7007b in the twelfth embodiment. Since the transmitter 7007b not only transmits but also receives, the transmitter 7007b includes an in-vehicle transmitter and an in-vehicle receiver.
The ID of the transmitter 7007b and the information to be passed to the receiver 7007a that received the ID are associated and stored in the server (parking lot management server) (7107a). The information passed to the receiver 7007a includes the shape and weight of the object whose component is the transmitter 7007b, the identification number such as the vehicle body number, the identification number of the user of the transmitter 7007b, and the information for payment. May be included.
The transmitter 7007b (vehicle-mounted transmitter) transmits an ID (7107b). The transmission content may include the URL of the server and the information stored in the server. The parking lot receiver 7007a (parking lot transmitter / receiver) transmits the received information to a server that manages the parking lot (parking lot management server) (7107c). The parking lot management server uses the ID of the transmitter 7007b as a key to acquire the information associated with the ID (7107d). The parking lot management server investigates the availability of parking lots (7107e).
The parking lot receiver 7007a (parking lot transmitter / receiver) transmits parking availability, parking location information, or the address of the server that holds this information (7107f). Alternatively, the parking lot management server sends this information to another server. The transmitter (vehicle-mounted receiver) 7007b receives the information transmitted above (7107g). Alternatively, the in-vehicle system acquires this information from another server.
The parking lot management server controls the parking lot to facilitate parking (7107h). For example, it controls a multi-story parking lot. The parking lot transmitter / receiver sends an ID (7107i). The in-vehicle receiver (transmitter 7007b) makes an inquiry to the parking lot management server based on the user information of the in-vehicle receiver and the received ID (7107j).
The parking lot management server charges according to the parking time, etc. (7107k). The parking lot management server controls the parking lot so that parked vehicles can be easily accessed (7107m). For example, it controls a multi-story parking lot. The in-vehicle receiver (transmitter 7007b) displays a map to the parking position and navigates from the current location (7107n).
FIG. 218 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the transmitter 7008a or 7008b, which is a signage configured as a roof sign of a store or a sign placed on the street, transmits the identification information (ID) of the transmitter 7008a or 7008b to a receiver 7008c configured as, for example, a smartphone. .. When the receiver 7008c receives the ID, it acquires the information associated with the ID from the server and displays it. The information is, for example, an image showing the availability of the store, a coupon, a two-dimensional bar code, and the like.
FIG. 219 is a flowchart showing an example of the processing operation of the receiver 7008c and the transmitter 7008a or 7008b in the twelfth embodiment. Hereinafter, among the transmitters 7008a and 7008b, the transmitter 7008a will be described as an example, but the processing operation of the transmitter 7008b is the same as the processing operation of the transmitter 7008a.
First, the ID of the transmitter 7008a and the information to be passed to the receiver 7008c that received the ID are associated and stored in the server (7108a). The information passed to the receiver 7008c includes store name, product name, map information to the store, vacant seat information, coupon information, product inventory, movie and drama screening time, reservation information, and so on. It may include information such as the URL of the server that makes the reservation or purchase.
The ID is transmitted from a transmitter 7008a such as signage (7108b). Point the camera of receiver 7008c at transmitter 7008a and receive the ID (7108c). The receiver 7008c sends the received ID to the server and stores the information related to the ID in the receiver 7008c (7108d). The receiver 7008c stores the terminal ID and user ID together in the server (7108e).
The receiver 7008c displays the information stored in the server for display on the receiver 7008c (7108f). The receiver 7008c displays directions from the current location to the store or merchandise store (7108g). The receiver 7008c acquires information from the server as appropriate, updates and displays vacant seat information and reservation information (7108h).
The receiver 7008c displays a button to reserve or order a seat or item (7108i). The user taps the reservation button or order button displayed on the receiver 7008c (7108j). The receiver 7008c sends reservation and order information to the server that manages them (7108k).
FIG. 220 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the receiver (terminal) 7009b configured as a smartphone is placed on a table in front of a seat in a store, for example. At this time, for example, the transmitter 7009a, which is a lighting device, transmits the identification information (ID) of the transmitter 7009a to the receiver 7009b. When the receiver 7009b receives the ID, it acquires the information associated with the ID from the server. Then, the receiver 7009b reserves the seat, confirms the provisional reservation, extends the reservation time, and the like.
FIG. 221 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
The receiver 7009b that has acquired the information from the server displays, for example, the availability of seats in the store and a button for selecting one of "payment", "extension", and "additional order".
FIG. 222 is a flowchart showing an example of the processing operation of the receiver 7009b and the transmitter 7009a in the twelfth embodiment.
The ID of the transmitter 7009a is associated with the information passed to the receiver 7009b that received the ID and stored in the server (7109a). The information passed to the receiver 7009b may include information on the position and shape of the transmitter 7009a. The ID is transmitted from the transmitter 7009a such as ceiling lighting (7109b).
The user places the receiver 7009b on a table or the like (7109c). The receiver 7009b recognizes that the receiver 7009b is placed on a table or the like from the information of the gyro sensor and the 9-axis sensor, and starts the reception process (7109d). Receiver 7009b determines which camera is pointing upwards from above the 9-axis sensor and uses that camera to receive the ID.
Point the camera of receiver 7009b at transmitter 7009a and receive the ID (7109e). Receiver 7009b sends the received ID to the server and stores the information related to the ID in receiver 7009b (7109f). Receiver 7009b estimates the position of receiver 7009b (7109g).
The receiver 7009b transmits the position of the receiver 7009b to the store management server (7109h). The store management server identifies the seat where the receiver 7009b is located (7109i). The store management server sends the seat number to receiver 7009b (7109j).
FIG. 223 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the transmitter 7011a, which is a ceiling lighting, transmits the identification information (ID) of the transmitter 7011a to the receiver 7011b configured as, for example, a smartphone. When the receiver 7011b receives the ID, it acquires the information associated with the ID from the server and estimates (confirms) its own position. Then, when the receiver 7011b is placed in the electronic device 7011c, the receiver 7011b functions as an operation terminal of the electronic device 7011c. This makes it possible to operate the electronic device 7011c with a rich interface such as a touch panel and audio output.
FIG. 224 is a flowchart showing an example of the processing operation of the receiver 7011b and the transmitter 7011a in the twelfth embodiment.
Store the location of the electronic device in the server (7110a). The ID of the electronic device, the model, the function to be possessed, and the interface information (screen, input / output voice, dialogue model) for operation may be stored in association with the position information.
The ID of the transmitter 7011a is associated with the information passed to the receiver 7011b that received the ID and stored in the server (7110b). The information passed to the receiver 7011b may include information on the position and shape of the transmitter 7011a.
The ID is transmitted from the transmitter 7011a such as ceiling lighting (7110c). Point the camera of receiver 7011b at transmitter 7011a and receive the ID (7110d). The receiver 7011b sends the received ID to the server and stores the information related to the ID in the receiver 7011b (7110e). Receiver 7011b estimates the position of receiver 7011b (7110f).
The user places the receiver 7011b on an electronic device (7110g). The receiver 7011b recognizes that the receiver 7011b has stopped from the information of the gyro sensor and the 9-axis sensor, and starts the following processing (7110h). If a certain period of time has passed since the receiver 7011b last estimated the position of the receiver 7011b, the receiver 7011b estimates its own position by the above method (7110i).
The receiver 7011b estimates the movement since the last self-position estimation from the information of the gyro sensor and the 9-axis sensor, and estimates the current position (7110j). The receiver 7011b acquires the information of the electronic device closest to the current position from the server (7110k). The receiver 7011b acquires the information of the electronic device from the electronic device via bluetooth or wifi (7110m). Alternatively, the information of the electronic device stored in the server is acquired.
Receiver 7011b displays electronic device information (7110n). The receiver 7011b accepts input as an operating terminal for electronic devices (7110p). The receiver 7011b transmits the operation information of the electronic device to the electronic device via bluetooth or wifi (7110q). Alternatively, it is sent to an electronic device via a server.
FIG. 225 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the camera of the receiver 7012a configured as a smartphone is aimed at the transmitter 7012b configured as an electronic device such as a television receiver (TV). At this time, the receiver 7012a receives the identification information (ID) of the transmitter 7043b transmitted from the transmitter 7043b. The receiver 7043a acquires the information associated with the ID from the server. As a result, the receiver 7012a functions as an operation terminal of the electronic device in the direction in which the camera is pointed. In other words, the receiver 7012a wirelessly connects to the transmitter 7012b via bluetooth, wifi, or the like.
FIG. 226 is a flowchart showing an example of the processing operation of the receiver 7012a and the transmitter 7012b in the twelfth embodiment.
The ID of the transmitter 7012b is associated with the information passed to the receiver 7012a that received the ID and stored in the server (7111a). The information passed to the receiver 7012a may include the ID of the electronic device, the model, the functions possessed, and the interface information for operation (screen, input / output voice, dialogue model).
The ID is transmitted from the transmitter 7012b mounted on or associated with the electronic device (7111b). Point the camera of receiver 7012a at transmitter 7012b and receive the ID (7111c). The receiver 7012a sends the received ID to the server and stores the information related to the ID in the receiver 7012a (7111d). The receiver 7012a acquires the information of the electronic device from the server by using the received ID as a key (7111e).
The receiver 7012a acquires the information of the electronic device from the electronic device via bluetooth or wifi (7111f). Alternatively, the information of the electronic device stored in the server is acquired. The receiver 7012a displays electronic device information (7111g).
The receiver 7012a accepts input as an operating terminal for electronic devices (7111h). The receiver 7012a transmits the operation information of the electronic device to the electronic device via bluetooth or wifi (7111i). Alternatively, it is sent to an electronic device via a server.
FIG. 227 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the receiver 7013b configured as a smartphone accepts a destination input by the user. Then, the camera of the receiver 7013b is pointed at the transmitter 7013a configured as a lighting device (light). At this time, the receiver 7013b receives the identification information (ID) of the transmitter 7013a transmitted from the transmitter 7013a. The receiver 7013b acquires the information associated with the ID from the server. Based on the acquired information, the receiver 7013b estimates (confirms) its own position. Then, the receiver 7013b navigates the user to the destination by voice or the like. If the user is visually impaired, the receiver 7013b reports the obstacle in detail to the user.
FIG. 228 is a flowchart showing an example of the processing operation of the receiver 7013b and the transmitter 7013a in the twelfth embodiment.
The user inputs the destination to receiver 7013b (7112a). The user points the receiver 7013b to the light (transmitter 7013a) (7112b). Even visually impaired users can point the receiver 7013b at it if they can recognize the strong light.
Receiver 7013b receives the signal superimposed on the light (7112c). Receiver 7013b acquires information from the server using the received signal as a key (7112d). Receiver 7013b gets a map from its current location to its destination from the server (7112e). Receiver 7013b displays a map and navigates from your current location to your destination (7112f).
FIG. 229 is a diagram showing a state of the receiver according to the twelfth embodiment.
For example, the receiver (terminal) 7014a configured as a smartphone is equipped with a face camera 7014b. Then, when the image pickup direction of the face camera 7014b is directed upward by a certain angle or more with respect to the ground plane, the receiver 7014a processes the signal reception by the face camera 7014b (receives the signal from the transmitter by image pickup). Processing) is performed. If the receiver 7014a also has a camera other than the face camera 7014b, the priority of the face camera 7014b is raised over the other cameras.
FIG. 230 is a flowchart showing an example of the processing operation of the receiver 7014a according to the twelfth embodiment.
It is determined whether the image pickup direction of the face camera 7014b points upward by a certain angle or more with respect to the ground plane (7113a). If the judgment result is true (Y), reception by the face camera 7014b is started (7113b). Alternatively, raise the priority of reception processing by the face camera 7014b. Then, after a certain period of time (7113c), reception by the face camera 7014b ends (7113d). Alternatively, lower the priority of reception processing by the face camera 7014b.
FIG. 231 is a diagram showing a state of the receiver according to the twelfth embodiment.
For example, the receiver (terminal) 7015a configured as a smartphone is equipped with an out-camera 7015b. Then, when the angle of the out-camera 7015b with respect to the ground plane in the imaging direction is equal to or less than a certain angle, the receiver 7014a processes the signal reception by the out-camera 7015b (the process of receiving the signal from the transmitter by imaging). Do it. If the receiver 7015a also has a camera other than the out-camera 7015b, the out-camera 7015b has a higher priority than the other cameras.
When the angle of the out-camera 7015b with respect to the ground plane in the imaging direction is equal to or less than a certain angle, the receiver 7015a is in a vertical state with respect to the ground plane of the surface of the receiver 7015a provided with the out-camera 7015b. The angle is more than a certain angle.
FIG. 232 is a flowchart showing an example of the processing operation of the receiver 7015a in the twelfth embodiment.
It is determined whether the image pickup direction of the out-camera 7015b is below a certain angle with respect to the ground plane (7114a). If the judgment result is true (Y), reception by the out-camera 7015b is started (7114b). Alternatively, raise the priority of reception processing by the out-camera 7015b. Then, after a certain period of time (7114c), reception by the out-camera 7015b ends (7114d). Alternatively, lower the priority of reception processing by the out-camera 7015b.
FIG. 233 is a diagram showing a state of the receiver according to the twelfth embodiment.
For example, the receiver (terminal) 7016a configured as a smartphone is equipped with an out-camera. Then, when the receiver 7016a is moved (protruded) in the imaging direction of the out-camera, the receiver 7016a performs a signal reception process (a process of receiving a signal from the transmitter by imaging) by the out-camera. If the receiver 7016a also has a camera other than the out-camera, the out-camera has a higher priority than the other cameras.
When the camera is moved in the imaging direction of the out-camera, the angle between the moving direction and the imaging direction (at the end of the movement) is equal to or less than a certain angle.
FIG. 234 is a flowchart showing an example of the processing operation of the receiver 7016a according to the twelfth embodiment.
The receiver 7016a is moved to determine whether the direction of movement and the imaging direction of the out-camera at the end of the movement are below a certain angle (7115a). If the judgment result is true (Y), reception by the out-camera is started (7115b). Alternatively, raise the priority of reception processing by the out-camera. Then, after a certain period of time (7115c), reception by the out-camera ends (7115d). Alternatively, lower the priority of reception processing by the out-camera.
FIG. 235 is a diagram showing a state of the receiver according to the twelfth embodiment.
For example, the receiver (terminal) 7017a configured as a smartphone is equipped with a predetermined camera. Then, the receiver 7017a performs a signal reception process (a process of receiving a signal from the transmitter by imaging) by the predetermined camera when the display operation corresponding to the predetermined camera or the pressing of the dedicated button is performed. Do it. If the receiver 7017a also has a camera other than the predetermined camera, the priority of the predetermined camera is higher than that of the other camera.
FIG. 236 is a flowchart showing an example of the processing operation of the receiver 7017a in the twelfth embodiment.
It is determined whether the display operation or the dedicated button is pressed for the receiver 7017a (7115h). If the judgment result is true (Y), reception by the camera corresponding to the display operation or the pressing of the dedicated button is started (7115i). Alternatively, raise the priority of reception processing by the camera. Then, after a certain period of time (7115j), the reception by the camera corresponding to the display operation or the pressing of the dedicated button is terminated (7115k). Alternatively, lower the priority of reception processing by the camera.
FIG. 237 is a diagram showing a state of the receiver according to the twelfth embodiment.
For example, the receiver (terminal) 7018a configured as a smartphone is equipped with a face camera 7018b. Then, in the receiver 7018a, the receiver 7014a is moving along a direction in which the image pickup direction of the face camera 7018b is directed upward by a certain angle or more with respect to the ground plane and is below a certain angle with respect to the ground plane. Occasionally, the face camera 7018b performs a signal reception process (a process of receiving a signal from the transmitter by imaging). If the receiver 7018a also has a camera other than the face camera 7018b, the face camera 7018b has a higher priority than the other cameras.
FIG. 238 is a flowchart showing an example of the processing operation of the receiver 7018a according to the twelfth embodiment.
It is determined whether the image pickup direction of the face camera 7018b is directed upward by a certain angle or more with respect to the ground plane and is translated by a certain angle or less with respect to the ground plane (7116a). If the judgment result is true (Y), reception by the face camera 7018b is started (7116b). Alternatively, raise the priority of reception processing by the face camera 7018b. Then, after a certain period of time (7116c), reception by the face camera 7018b ends (7116d). Alternatively, lower the priority of reception processing by the face camera 7018b.
FIG. 239 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the camera of the receiver 7019b configured as a smartphone is aimed at the transmitter 7019a configured as an electronic device such as a television receiver (TV). At this time, the receiver 7019b receives the identification information (ID) of the channel being viewed, which is transmitted from the transmitter 7019a (display of the transmitter 7019a). The receiver 7019b acquires the information associated with the ID from the server. As a result, the receiver 7019b displays the purchase page of the related product of the TV program or the related information of the TV program. Receiver 7019b also participates in television programs by voting or applying for gifts. The transmitter (TV) 7019a may include an address storage unit that stores the user's address, and may transmit information related to the address stored in the address storage unit. The receiver 7019b can acquire data from the server by sending the received ID and the time when the ID is received to the server, excluding the influence of the delay from the ID reception to the server access. The transmitter 7019a may acquire time information or an ID that changes depending on the time from the built-in clock or broadcast wave and transmit the information. As a result, the data set by the broadcaster can be sent from the server to the receiver regardless of the time setting of the receiver.
FIG. 240 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
The transmitter 7019a and the receiver 7019b may directly transmit and receive the information necessary for realizing the application example shown in FIG. 239, as shown in FIG. 240 (a).
Further, as shown in FIG. 240 (b), the transmitter 7019a transmits the ID of the channel being viewed to the receiver 7019b. The receiver 7019b acquires the information associated with the ID, that is, the information necessary for realizing the application example shown in FIG. 239 from the server.
Further, as shown in FIG. 240 (c), the transmitter 7019a may transmit the ID of the transmitter (TV) 7019a or the information necessary for wireless connection to the receiver 7019b. In this case, the receiver 7019b receives the ID or information, and based on the ID or information, inquires the transmitter 7019a or the recorder about the channel being watched. Further, the receiver 7019b acquires the information about the channel obtained by the inquiry, that is, the information necessary for realizing the application example shown in FIG. 239, from the server.
For example, the transmitter 7019a transmits an SSID (Servi ce Set Identifier), a password, an IP address, a device ID, or the like as information necessary for a wireless connection such as Wi-Fi or Bluetooth (registered trademark) . Upon receiving such information, the receiver 7019b makes a wireless connection with the transmitter 7019a based on the information. Then, the receiver 7019b acquires the information of the program being viewed by the user from the transmitter 7019a by wireless connection, and transmits the information of the program to the server. When the server receives the information of the program, the server transmits the content associated with the information of the program to the receiver 7019b. When the receiver 7019b acquires the content transmitted from the server, the receiver 7019b displays this content.
FIG. 241 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
Transmitter 7019a may include TV 2021b and recorder 2021a. In the transmitter 7019a, the recorder 2021a holds the identification information (ID) of the channel to be recorded and the recording time at the time of recording. Alternatively, the recorder 2021a acquires and retains the identification information (ID) of the channel to be recorded and the information associated with the recording time from the server. Then, at the time of reproduction, the TV 2021b transmits a part or all of the information stored in the recorder 2021a to the receiver 7019b. Also, at least one of the TV 2021b and the recorder 2021a may act as a server. When the recorder 2021a acts as a server, the recorder 2021a replaces the address of the server with the address of the recorder 2021a and causes the TV 202b to transmit the address to the receiver 7019b.
FIG. 242 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the camera of the receiver 7022c configured as a smartphone is aimed at the transmitter 7022b configured as an electronic device such as a television receiver (TV). At this time, the receiver 7022c receives the information transmitted from the transmitter 7022b (display of the transmitter 7022b). The receiver 7022c wirelessly communicates with the transmitter 7022b based on the information. Here, when the transmitter 7022b acquires information including an image to be displayed on the receiver 7022c from the server 7022a and transmits the information to the receiver 7022c, the transmitter 7022b uses the address of the server 7022a included in the information as the address of the transmitter 7022b. Rewrite to.
FIG. 243 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the recorder 7023b collectively acquires the information necessary for realizing the application example shown in FIG. 239 from the server 7023a when recording a television program.
When playing a television program, the recorder 7023b transmits the reproduction screen and the information necessary for realizing the application example shown in FIG. 239 to the television 7023c which is a transmitter. The television 7023c receives the playback screen and information, displays the playback image, and transmits the information from the display. For example, when the receiver 7023d configured as a smartphone receives the information, it wirelessly communicates with the television 7023c based on the information.
Alternatively, when playing a television program, the recorder 7023b transmits the playback screen and information necessary for wireless communication, such as the address of the recorder 7023b, to the television 7023c, which is a transmitter. The television 7023c receives the playback screen and information, displays the playback image, and transmits the information from the display. For example, when the receiver 7023d configured as a smartphone receives the information, it wirelessly communicates with the recorder 7023b based on the information.
FIG. 244 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the camera of the receiver 7045a configured as a smartphone is aimed at the transmitter 7045b configured as an electronic device such as a television receiver (TV). The transmitter 7045b displays images of TV programs such as song programs and transmits information from the display. At this time, the receiver 7045a receives the information transmitted from the transmitter 7045b (display of the transmitter 7045b). Then, the receiver 7045a displays a screen 7045c prompting the purchase of the song being viewed in the song program based on the information.
FIG. 245 is a flowchart showing an example of the processing operation of the receiver and the transmitter in the twelfth embodiment. It should be noted that this flowchart corresponds to the application example shown in FIGS. 239 to 244.
A transmitter mounted on a television or a recorder acquires information to be transmitted to a receiver as information related to a program being broadcast from a server (7117a). The transmitter superimposes the signal on the backlight of the display and transmits it (7117b). The signal to be transmitted may include the URL of the transmitter, the SSID of the transmitter, and the password for accessing the transmitter.
FIG. 246 is a flowchart showing an example of the processing operation of the receiver and the transmitter in the twelfth embodiment. It should be noted that this flowchart corresponds to the application example shown in FIGS. 239 to 244.
The receiver receives information from the display (7118a). Determine if the received information includes the channel information you are watching (7118b). If the judgment result is false (N), the channel information being viewed is acquired from the electronic device having the ID included in the received information (7118c).
On the other hand, if the above determination result is true (Y), the information related to the screen being viewed is acquired from the server (7118d). A television or recorder may act as a server. The receiver displays the information obtained from the server (7118e). The receiver adjusts the display contents based on the user profile stored in the receiver or server (7118f). For example, control is performed such as changing the size of characters, hiding age-restricted content, and preferentially displaying content presumed to be preferred from past user behavior.
FIG. 247 is a flowchart showing an example of the processing operation of the receiver and the transmitter in the twelfth embodiment. It should be noted that this flowchart corresponds to the application example shown in FIGS. 239 to 244.
The recorder acquires and saves information related to the program from the server at the same time as recording the program (7119a). If the related information changes with the time, save the time as well.
The recorder transmits the stored information to the display when playing back the recorded image (7119b). The access information (URL and password) of the server in the saved information may be replaced with the access information of the display.
The recorder transmits the stored information to the receiver when playing back the recorded image (7119c). The access information (URL and password) of the server in the saved information may be replaced with the access information of the recorder.
FIG. 248 is a diagram showing a change in brightness of the transmitter according to the twelfth embodiment.
The transmitter encodes the information transmitted to the receiver by varying the length of time between the sudden increase in brightness and the next sudden increase in brightness by sign (0 or 1). To become. As a result, it is possible to adjust the brightness perceived by humans by PWM (Pulse Width Modulation) control without changing the content of the transmitted information. The luminance waveform does not have to be an accurate square wave.
FIG. 249 is a flowchart showing an example of the processing operation of the receiver according to the twelfth embodiment. It should be noted that this flowchart shows the processing operation of the receiver corresponding to the transmitter with the luminance change shown in FIG. 248.
The receiver observes the brightness of the light emitted by the transmitter (7120a). Measure the time from when the brightness rises sharply before to when the brightness rises sharply next (7120b). Alternatively, the time from when the brightness drops sharply before to when the brightness drops sharply next is measured. The value of the signal is recognized by the above time (7120c). For example, when the above time is 300 microseconds or less, it is recognized as 0, and when it is more than 300 microseconds, it is recognized as 1.
FIG. 250 is a diagram showing a change in brightness of the transmitter according to the twelfth embodiment.
The transmitter expresses the starting point of the information transmitted to the receiver by changing the wavelength indicating the increase or decrease of the brightness. Alternatively, the transmitter superimposes one piece of information on another by changing its wavelength.
FIG. 251 is a flowchart showing an example of the processing operation of the receiver in the twelfth embodiment. It should be noted that this flowchart shows the processing operation of the receiver corresponding to the transmitter with the luminance change shown in FIG. 250.
The receiver observes the brightness of the light emitted by the transmitter (7121a). Find the minimum time width during a sudden change in brightness (7121b). Look for a brightness change width that is not an integral multiple of the above minimum value (7121c). The signal is analyzed with the luminance change width that is not an integral multiple as the starting point of the signal (7121d). The time width between the portions having a luminance change width that is not an integral multiple is obtained (7121e).
FIG. 252 is a diagram showing a change in brightness of the transmitter according to the twelfth embodiment.
The transmitter can adjust the brightness visible to the human eye by changing the brightness at intervals shorter than the exposure time of the receiver. In addition, it is possible to reset the accumulated brightness change with the passage of time.
FIG. 253 is a flowchart showing an example of the processing operation of the transmitter according to the twelfth embodiment. It should be noted that this flowchart shows the processing operation of the receiver corresponding to the transmitter with the luminance change shown in FIG. 252.
When the luminance or the current that controls the luminance falls below a certain value, the current is turned off and on with a time width sufficiently shorter than the exposure time of the receiver (S7125a). This makes it possible to return the current to the initial value and prevent the brightness of the light emitting unit from decreasing. In addition, when the luminance or the current that controls the luminance exceeds a certain value, the current is turned on and off in a time width sufficiently shorter than the exposure time of the receiver (S7125b). This makes it possible to return the current to the initial value and prevent the brightness of the light emitting unit from increasing.
FIG. 254 is a diagram showing a change in brightness of the transmitter according to the twelfth embodiment.
The transmitter expresses a different signal (information) by making the carrier frequency of the luminance different. The receiver can recognize the carrier frequency earlier than the signal content. Therefore, making the carrier frequencies different is suitable for expressing the contents to be recognized preferentially, such as the ID of the transmitter.
FIG. 255 is a flowchart showing an example of the processing operation of the receiver according to the twelfth embodiment. It should be noted that this flowchart shows the processing operation of the receiver corresponding to the transmitter with the luminance change shown in FIG. 254.
The receiver observes the brightness of the light emitted by the transmitter (7122a). Find the minimum time width during a sudden change in brightness (7122b). The above minimum value is recognized as a carrier frequency (7122c). Obtain information from the server using the carrier frequency as a key (7122d).
FIG. 256 is a flowchart showing an example of the processing operation of the receiver according to the twelfth embodiment. It should be noted that this flowchart shows the processing operation of the receiver corresponding to the transmitter with the luminance change shown in FIG. 254.
The receiver observes the brightness of the light emitted by the transmitter (7123a). The luminance change is Fourier transformed and the maximum component is recognized as the carrier frequency (7123b). Obtain information from the server using the carrier frequency as a key (7123c).
FIG. 257 is a flowchart showing an example of the processing operation of the transmitter according to the twelfth embodiment. It should be noted that this flowchart shows the processing operation of the transmitter accompanied by the change in luminance shown in FIG. 254.
The transmitted signal is expressed as a change in luminance (7124a). The luminance change is created so that the maximum component obtained by Fourier transforming the luminance change becomes the carrier frequency (7124b). The light emitting part is made to emit light according to the created brightness change (7124c).
FIG. 258 is a diagram showing a configuration example of the transmitter according to the twelfth embodiment.
The transmitter 7028a has a portion 7028b for transmitting the signal A, a portion 7028d for transmitting the signal B, and a portion 7028f for transmitting the signal C. In this way, the receiver can receive a plurality of signals at the same time because there is a portion that transmits different signals along the direction in which the image pickup unit (camera) of the receiver simultaneously exposes. In addition, a portion that does not emit a signal or a buffer portion 7028c, 7028e that emits a special signal may be provided between the respective portions 7028b, 7028d, and 7028f.
FIG. 259 is a diagram showing a configuration example of the transmitter according to the twelfth embodiment. The light emitting method according to the configuration of this transmitter is an extension of the light emitting method according to the configuration shown in FIG. 258.
The transmitter may be provided with a portion 7029a for transmitting each signal shown in FIG. 258 as shown in FIG. 259. As a result, even when the receiver is tilted, the image pickup unit (camera) of the receiver can simultaneously receive (image) many parts of the signals A, B, and C.
FIG. 260 is a diagram showing a configuration example of the transmitter according to the twelfth embodiment. The light emitting method according to the configuration of this transmitter is an extension of the light emitting method according to the configuration shown in FIG. 258.
The circular light emitting part of the transmitter includes a plurality of annular portions 7030a, 7030b, 7030c arranged concentrically to transmit each signal. Part 7030a transmits signal C, part 7030b transmits signal B, and part 7030c transmits signal A. In this way, when the light emitting part of the transmitter is circular, by arranging the parts that transmit each signal as described above, the receiver can receive many of the signals A, B, and C emitted from each part. Parts can be received (imaged) at the same time.
FIG. 261 is a flowchart showing an example of the processing operation of the receiver and the transmitter in the twelfth embodiment. It should be noted that this flowchart shows the processing operation of the transmitter and the receiver provided with the light emitting device shown in any of FIGS. 258 to 260.
The receiver measures the brightness at each position of the line that receives light at the same time (7126a). It receives signals at high speed by receiving signals transmitted separately in the direction perpendicular to the simultaneous light receiving line (7126b).
FIG. 262 is a diagram showing an example of display and photographing by the receiver and the transmitter in the twelfth embodiment.
The transmitter has multiple one-dimensional barcodes that are configured as uniform images in the direction perpendicular to the direction in which the image receiving unit (camera) of the receiver is simultaneously exposed, in frame 1 (7031a) and frame 2 (7031b), respectively. And displayed in order as frame 3 (7031c). The one-dimensional barcode is composed of a line (bar) along the direction perpendicular to the direction of simultaneous exposure. The receiver acquires the frame 1 (7031d) and the frame 2 (7031e) by capturing the image displayed on the transmitter as in each of the above embodiments. The receiver can recognize the continuously displayed one-dimensional barcodes in order by dividing the one-dimensional barcode at the part where the one-dimensional barcode bar is interrupted. In this case, it is not necessary to synchronize the timing of the display by the transmitter and the imaging by the receiver, and the receiver can recognize all the contents displayed by the transmitter. Further, the frame rate of the display by the transmitter may be faster than the frame rate of the image captured by the receiver. However, the display time of one frame displayed by the transmitter must be longer than the blanking time between frames imaged by the receiver.
FIG. 263 is a flowchart showing an example of the processing operation of the transmitter according to the twelfth embodiment. It should be noted that this flowchart shows the processing operation of the display device included in the transmitter that performs the display shown in FIG. 262.
The display device displays a one-dimensional barcode (7127a). The display device changes the barcode display at intervals longer than the blanking time of the receiver's imaging (7127b).
FIG. 264 is a flowchart showing an example of the processing operation of the receiver according to the twelfth embodiment. It should be noted that this flowchart shows the processing operation of the receiver that performs the photographing shown in FIG. 262.
The receiver captures the one-dimensional barcode displayed on the display device (7128a). When the barcode line is interrupted, the display device recognizes that the next barcode is displayed (7128b). By this method, the receiver can receive all the display contents without synchronizing the display and the imaging. In addition, a signal displayed at a frame rate faster than the image pickup frame rate of the receiver can also be received.
FIG. 265 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
For example, the transmitter 7032a, which is a lighting device, transmits the encrypted identification information (ID) of the transmitter 7032a. For example, when the receiver 7032b configured as a smartphone receives the encrypted ID, it sends the encrypted ID to the server 7032c. Upon receiving the encrypted ID, the server 7032c decrypts the encrypted ID. Alternatively, when the receiver 7032b receives the encrypted ID, it decrypts the encrypted ID and sends it to the server 7032c.
FIG. 266 is a flowchart showing an example of the processing operation of the receiver 7032b and the transmitter 7032a in the twelfth embodiment.
Transmitter 7032a retains partially or wholly encrypted information (7129a). The receiver 7032b receives the information transmitted from the transmitter 7032a and decodes the received information (7129b). Alternatively, the receiver 7032b sends the encrypted information to the server 7032c. If the as-encrypted information is sent, the server 7032c decrypts the as-encrypted information (7129c).
FIG. 267 is a diagram showing a state of the receiver according to the twelfth embodiment.
To make a call, the user listens to the receiver 7033a, which is configured as, for example, a smartphone. At this time, the illuminance sensor provided near the speaker of the receiver 7033a detects an illuminance value indicating low illuminance. As a result, the receiver 7033a estimates that the state of the receiver 7033a is in a call, and stops receiving information from the transmitter.
FIG. 268 is a flowchart showing an example of the processing operation of the receiver 7033a in the twelfth embodiment.
The receiver 7033a determines whether it is presumed to be in a call from the sensor value of the illuminance sensor (7130a). If the judgment result is true (Y), the receiver 7033a ends the reception by the face camera (7130b). Alternatively, the receiver 7033a lowers the priority of the reception process by the face camera.
FIG. 269 is a diagram showing a state of the receiver according to the twelfth embodiment.
For example, the receiver 7034a configured as a smartphone is provided with an illuminance sensor 7034b near a camera (for example, a face camera) which is an image pickup device for receiving (imaging) information from a transmitter. When the illuminance sensor 7034b detects an illuminance value indicating a low illuminance below a certain value, the receiver 7034a stops receiving information from the transmitter. Alternatively, if the receiver 7034a is equipped with another camera, the priority of the camera near the illuminance sensor 7034b (for example, a face camera) is lowered over the other cameras.
FIG. 270 is a flowchart showing an example of the receiver 7034a processing operation according to the twelfth embodiment.
The receiver 7034a determines whether the sensor value of the illuminance sensor 7034b is below a certain value (7131a). If the judgment result is true (Y), the receiver 7034a ends the reception by the face camera (7131b). Alternatively, the receiver 7034a lowers the priority of reception processing by the face camera.
FIG. 271 is a flowchart showing an example of the processing operation of the receiver according to the twelfth embodiment.
The receiver measures the change in illuminance from the sensor value of the illuminance sensor (7132a). The receiver receives the signal from the change in the illuminance value, just as the image pickup device (camera) measures the change in brightness and receives the signal (7132b). Since the illuminance sensor is cheaper than the image pickup device, the receiver can be manufactured at low cost.
FIG. 272 is a diagram showing an example of the wavelength of the transmitter according to the twelfth embodiment.
As shown in FIGS. 272 (a) and 272 (b), the transmitter expresses the information to be transmitted to the receiver by outputting the metallic lights 7037a and 7037b.
FIG. 273 is a flowchart showing an example of the processing operation of the receiver and the transmitter in the twelfth embodiment. It should be noted that this flowchart shows the processing operation of the transmitter and the receiver that output the light having the wavelength shown in FIG. 272.
The transmitter expresses a different signal with light (metameric light) that is perceived by humans as having the same color but has a different spectral distribution, and causes the light emitting part to emit light (7135a). The receiver measures the spectral distribution and receives the signal (7135b). According to this method, it is possible to transmit a signal without worrying about flicker.
FIG. 274 is a diagram showing a configuration example of a system including a receiver and a transmitter according to the twelfth embodiment.
This system includes an ID resolution server 7038a, a relay server 7038b, a receiver 7038c, a transmitter 7038d, and a transmitter-side controller 7038e.
FIG. 275 is a flowchart showing an example of the processing operation of the system according to the twelfth embodiment.
The ID resolution server 7038a stores the ID of the transmitter 7038d in association with the method in which the transmitter 7038e and the receiver 7038c communicate (7136a). The receiver 7038c receives the ID of the transmitter 7038d and acquires the communication method with the transmitter side control device 7038e from the ID resolution server 7038a (7136b). The receiver 7038c determines whether the receiver 7038c and the transmitter side controller 7038e can directly communicate with each other (7136c). If the determination result is false (N), the receiver 7038c communicates with the transmitter side controller 7038e via the relay server 7038b (7136d). On the other hand, if the determination result is true (Y), the receiver 7038c communicates directly with the transmitter side control device 7038e (7136e).
FIG. 276 is a diagram showing a configuration example of a system including a receiver and a transmitter according to the twelfth embodiment.
The system is equipped with a server 7039g, a store-side device 7039a and a mobile device 7039b. The store-side device 7039a includes a transmitter 7039c and an image pickup unit 7039d. The portable device 7039b includes a receiver 7039e and a display unit 7039f.
FIG. 277 is a flowchart showing an example of the processing operation of the system according to the twelfth embodiment.
The portable device 7039b displays information in the form of a two-dimensional bar code or the like on the display unit 7039f (7137a). The store-side device 7039a captures the content displayed on the display unit 7039f by the image pickup unit 7039d and acquires the information (7137b). The store-side device 7039a transmits some information from the transmitter 7039c (7137c).
The mobile device 7039b receives the transmitted information on the receiver 7039e (7137d). The mobile device 7039b changes the display content of the display unit 7039f based on the received information (7137e). The content to be displayed on the display unit 7039f may be determined by the mobile device 7039b, or may be determined by the server 7039g based on the received content.
The store-side device 7039a captures the content displayed on the display unit 7039f by the image pickup unit 7039d and acquires the information (7137f). The store-side device 7039a determines the consistency between the acquired information and the transmitted information (7137g). This judgment may be made by the store-side device 7039a or by the server 7039g. If consistent, the transaction completes successfully (7137h).
By this method, the coupon information displayed on the display unit 7039f can be copied and prevented from being used illegally. It is also possible to exchange encryption keys by this method.
FIG. 278 is a flowchart showing an example of the processing operation of the receiver according to the twelfth embodiment.
The receiver starts the reception process (7138a). The receiver sets the exposure time of the image pickup device (7138b). The receiver sets the gain of the imager (7138c). The receiver receives information from the brightness of the captured image (7138d).
FIG. 279 is a flowchart showing an example of the processing operation of the receiver according to the twelfth embodiment.
Set the exposure time (7139a). Determine if there is an API (Application Program Interface) that changes the exposure time (7139b). If the judgment result is false (N), point the image pickup device at a bright object such as a light source (7139c). Set the automatic exposure setting (7139d). If the change in the automatic exposure setting value becomes small enough, the exposure setting value is fixed (7139e).
On the other hand, if the above judgment result is true (Y), the setting of the exposure time is started using the API (7139f).
FIG. 280 is a diagram showing a configuration example of a system including a receiver and a transmitter according to the twelfth embodiment.
The system comprises a server 7036a, a receiver 7036b, and at least one transmitter 7036c. Receiver 7036b gets information from the server about at least one transmitter 7036c that is in the vicinity of receiver 7036b.
FIG. 281 is a flowchart showing an example of the processing operation of the receiver according to the twelfth embodiment.
The receiver 7036b estimates its own position from information such as GPS and base stations (7133a). The receiver 7036b sends the estimated self-position and the estimated error range to the server 7036a (7133b). The receiver 7036b acquires and stores the ID of the transmitter 7036c existing near the position of the receiver 7036b and the information associated with the ID from the server 7036a (7133c). Receiver 7036b receives the ID from transmitter 7036c (7133d).
Receiver 7036b determines if it remembers the information associated with the received ID (7133e). If the judgment result is false (N), the receiver 7036b acquires information from the server 7036a using the received ID as a key (7133f). The receiver 7036b estimates its own position from the information received from the server 7036a and the positional relationship of the transmitter 7036bc, and acquires the IDs of other transmitters 7036c in the vicinity and the information associated with the IDs from the server 7036a. And remember (7133g).
After the determination in step 7133e is true (Y), or after step 7133g is performed, the receiver 7036b displays the information associated with the received ID (7133h).
FIG. 282 is a diagram showing an application example of the receiver and the transmitter in the twelfth embodiment.
The building a (7040a) is provided with transmitters 7040c and 7040d configured as lighting equipment, for example, and the building b (7040b) is provided with transmitters 7040e and 7040f configured as lighting equipment, for example. Has been done. Transmitters 7040c and 7040e transmit signal A and transmitters 7040d and 7040f transmit signal B. For example, the receiver (terminal) 7040g configured as a smartphone receives a signal transmitted from any of the transmitters.
FIG. 283 is a flowchart showing an example of the processing operation of the receiver according to the twelfth embodiment.
Receiver 7040g detects that it has entered a building (7134a). The receiver 7040g transmits the estimated self-position, the estimation error range, the name of the building in which the receiver 7040g is estimated to exist, and the like to the server (7134b). The receiver 7040g acquires and stores the ID of the transmitter existing in the building in which the receiver 7040g is located and the information associated with the ID from the server (7134c). The receiver 7040g receives the ID from the transmitter (7134d).
The receiver 7040g determines if it remembers the information associated with the received ID (7134e). If the judgment result is false (N), the receiver 7040g acquires information from the server using the received ID as a key (7134f). The receiver 7040g acquires and stores the IDs of other transmitters existing in the same building as the building in which the transmitter receiving the ID exists and the information associated with the IDs from the server (7134g). ).
After the determination in step 7134e is true (Y) or after step S7134g is performed, the receiver 7040g displays the information associated with the received ID (7134h).
FIG. 284 is a diagram showing a configuration example of a system including a receiver and a transmitter according to the twelfth embodiment.
For example, transmitter 7041a configured as a luminaire transmits signal A, transmitter 7041b transmits signal B, transmitter 7041c transmits signal C, and transmitter 7041d transmits signal B. For example, the receiver (terminal) 7041e configured as a smartphone receives a signal transmitted from any of the transmitters. Here, the error range of the self-position of the receiver 7041e estimated based on the information (other means) such as GPS and the base station includes the transmitters 7041a, 7041b, 7041c.
FIG. 285 is a flowchart showing an example of the processing operation of the system according to the twelfth embodiment.
The receiver 7041e receives the ID from the transmitter (7140a). Receiver 7041e performs self-position estimation (7140b). The receiver 7041e determines whether the self-position estimation was successful (7140c). If the judgment result is false (N), the receiver 7041e displays a map or an input form and prompts the user to input the current position (7140d).
The receiver 7041e transmits the received ID, the estimated self-position, and the error range of the self-position estimation to the server (7140e).
The server determines whether there is only one transmitter that transmits the ID received by the receiver 7041e within the estimated error range (estimated error radius) from the estimated self-position of the receiver 7041e (7140f). If the determination result is false (N), the receiver 7041e repeatedly executes the process from step 7140d. On the other hand, if the judgment result is true (Y), the server sends the information associated with the transmitter to the receiver 7041e (7140 g).
FIG. 286 is a flowchart showing an example of the processing operation of the receiver according to the twelfth embodiment.
First, the receiver detects the light emitting device (transmitter) emitting the signal (7141a) and receives the signal (7141b). Then, the receiver displays the reception status, the received data amount, the transmission data amount, and the ratio of the received data amount to the transmission data amount (7141c).
The receiver then checks to see if it has received all the transmitted data (7141d). When it is confirmed that the reception has been received (Y of 7141d), the receiver cancels the reception processing (7141e) and displays the reception completion (7141f). In addition, the receiver sounds a notification sound (7141g) and vibrates (7141h).
If it is confirmed in step 7141d that it has not been received (N in 7141d), the receiver has determined whether a certain amount of time has passed since the transmitter framed out in the image obtained by the image pickup device (camera) of the receiver. Confirm (7141i). Here, when it is confirmed that a certain period of time has passed (Y of 7141i), the receiver discards the received data and stops the reception process (7141 m). In addition, the receiver sounds a notification sound (7141n) and vibrates (7141p).
If it is confirmed in step 7141i that a certain amount of time has not passed (N of 7141i), the receiver has either changed the sensor value obtained by the receiver's 9-axis sensor by a certain amount or more, or Check if it can be estimated that the receiver is pointed in the other direction (7141j). Here, if it is confirmed that the value has changed by a certain value or more, or it can be estimated that the direction has changed (Y of 7141i), the above-mentioned process of step 7141m or later is performed.
On the other hand, when it is confirmed that the size does not change beyond a certain value or it cannot be estimated that it is directed in another direction (N of 7141i), the receiver has a constant rhythm of the sensor value of the 9-axis sensor of the receiver. Check if it has changed in or if the receiver can be estimated to be shaken (7141k). Here, when it is confirmed that it can be estimated that the change or the shake is performed with a constant rhythm, the receiver performs the processing after step 7141m described above. On the other hand, if it is confirmed that it has not changed at a constant rhythm or cannot be estimated to be shaken (N of 7141k), the receiver repeatedly executes the process from step 7141b.
FIG. 287A is a diagram showing an example of the configuration of the transmitter according to the twelfth embodiment.
The transmitter 7046a includes a light emitting unit 7046b, a two-dimensional bar code 7046c, and an NFC chip 7046d. The light emitting unit 7046b transmits information common to at least one of the two-dimensional bar code 7046c and the NFC chip 7046d by the method of each of the above embodiments. Alternatively, the light emitting unit 7046b may transmit information different from at least one of the two-dimensional bar code 7046c and the NFC chip 7046d by the method of each of the above embodiments. In this case, the receiver may acquire information common to at least one of the two-dimensional bar code 7046c and the NFC chip 7046d from the server by using the content of the information transmitted from the light emitting unit 7046b as a key. Further, the receiver may perform common processing depending on whether the information is received from the light emitting unit 7046b or from at least one of the two-dimensional bar code 7046c and the NFC chip 7046d. In either case of the above, the receiver accesses a common server and displays common information.
FIG. 287B is a diagram showing another example of the transmitter configuration according to the twelfth embodiment.
The transmitter 7046e includes a light emitting unit 7046f, and causes the light emitting unit 7046f to display a two-dimensional bar code 7046g. That is, the light emitting unit 7046f has the respective functions of the light emitting unit 7046b and the two-dimensional bar code 7046c shown in FIG. 287A.
Here, the light emitting unit 7046b, 7046f may cause the receiver to estimate the distance from the receiver to the transmitters 7046a, 7046e by transmitting information indicating the size of the light emitting unit 7046b, 7046f. This allows the receiver to shoot the 2D barcode 7046c, 7046g more easily or clearly.
FIG. 288 is a flowchart showing an example of the processing operation of the receiver and the transmitters 7046a and 7046e in the twelfth embodiment. Hereinafter, among the transmitter 7046a and the transmitter 7046e, the transmitter 7046a will be described as an example, but the processing operation of the transmitter 7046e is the same as the processing operation of the transmitter 7046a.
The transmitter 7046a transmits information indicating the size of the light emitting unit 7046b (step 7142a). The maximum value of the distance between any two points in the light emitting unit 7046b is defined as the size of the light emitting unit 7046b. Since speed is important for this series of processing, the transmitter 7046a directly transmits information indicating the size of the light emitting unit 7046b of the transmitter 7046a, and the receiver does not use server communication to transmit this size. It is desirable to obtain information indicating that. In addition, it is desirable to transmit by a method that can be received quickly, such as the frequency of the change in brightness of the transmitter 7046a.
Next, the receiver receives the signal which is the above information and acquires the size of the light emitting unit 7046b of the transmitter 7046a (step 7142b). Then, the receiver determines the distance from the receiver to the light emitting unit 7046b based on the size of the light emitting unit 7046b, the size of the captured image of the light emitting unit 7046b, and the characteristics of the image pickup unit (camera) of the receiver. Calculate (step 7142c). Next, the receiver adjusts the focal length of the image pickup unit to the calculated distance and shoots (step 7142d). The receiver acquires the contents of the 2D barcode if it is taken (step 7142e).
(Embodiment 13) In the present embodiment, each application example using a receiver such as a smartphone in the above embodiments 1 to 12 and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described. do.
FIG. 289 is a flowchart showing an example of the processing operation regarding the receiver and the transmitter in the thirteenth embodiment.
At step 7201a, the transmitter emits a sound of a specific frequency or a sound that changes in a specific pattern (this frequency is collected by a general sound collector, which is difficult for humans to hear, for example, 2kHz to 20kHz). It is desirable that the sound has a frequency that can be sounded. Also, since a general sound collector has a sampling frequency of about 44.1 kHz, according to the sampling theorem, only half the frequency can be accurately recognized, but the transmitted signal. If you know, it is possible to estimate with high accuracy whether or not the signal is being collected. Therefore, you may use this property to use a signal with a frequency of 20 kHz or higher.) ..
At step 7201b, the user presses a button on the receiver to power it on from power off or sleep. At step 7201c, the receiver activates the sound collector. At step 7201d, the receiver collects the sound emitted by the transmitter. At step 7201e, the receiver notifies the user that there is a transmitter in the vicinity by displaying it on the screen, making a sound, or vibrating it. At step 7201f, the receiver starts and ends reception.
FIG. 290 is a flowchart showing an example of processing operations related to the receiver and the transmitter in the thirteenth embodiment.
At step 7202a, the user presses a button on the receiver to power it on from power off or sleep. At step 7202b, the receiver activates the illuminance sensor. At step 7202c, the receiver recognizes the change in illuminance from the illuminance sensor. At step 7202d, the receiver receives the transmit signal from the illuminance sensor. At step 7202e, the receiver notifies the user that there is a transmitter in the vicinity by displaying it on the screen, making a sound, or vibrating it. At step 7202f, the receiver starts and ends reception.
FIG. 291 is a flowchart showing an example of processing operations related to the receiver and the transmitter in the thirteenth embodiment.
At step 7203a, the user operates the receiver to initiate reception. Alternatively, the receiver automatically starts receiving with some trigger. In step 7203b, the average luminance of the entire screen or the luminance of the maximum luminance point preferentially receives the bright image pickup unit and ends.
FIG. 292 is a flowchart showing an example of processing operations related to the receiver and the transmitter in the thirteenth embodiment.
In step 7204a, the image pickup unit does not image the simultaneous image pickup line or pixel in which the transmitter is not shown in the simultaneous image pickup line or pixel, so that the simultaneous image pickup line or pixel in which the transmitter is shown is photographed at high speed. .. At step 7204b, the receiver detects the movement and camera shake of the receiver with a gyro and a 9-axis sensor, adjusts the transmitter so that it is always visible by electronic correction, and ends.
FIG. 293 is a flowchart showing an example of processing operations related to the receiver and the transmitter in the thirteenth embodiment.
At step 7205a, the receiver displays the 2D barcode A. At step 7205b, the transmitter reads the 2D barcode A. At step 7205c, the transmitter sends a display change command. At step 7205d, the receiver displays the 2D barcode B. At step 7205e, the transmitter reads the 2D barcode B and exits.
FIG. 294 is a diagram showing an application example of the transmitter according to the thirteenth embodiment.
The transmitter 7211a comprises a mark 7211b indicating that it is a transmitter. Humans cannot distinguish between transmitted signals and normal light, but this allows them to recognize that the 7211a is a transmitter. Similarly, the transmitter 7211c comprises a mark 7211d indicating that it is a transmitter. Similarly, the transmitter 7211e displays the mark 7211f indicating that it is a transmitter only while transmitting a signal.
FIG. 295 is a diagram showing an application example of the transmitter according to the thirteenth embodiment.
The transmitter 7212a of a television or the like transmits a signal by changing the brightness of the backlight or the screen 7212b. A transmitter 7212c such as a television transmits a signal by changing the brightness of a part other than the screen, such as a bezel 7212d or a logo mark.
FIG. 296 is a diagram showing an application example of the transmitter according to the thirteenth embodiment.
The transmitter 7213a, such as a television, transmits a signal on the screen 7213b when displaying emergency news, subtitles, or an on-screen display 7213c. In addition, the 7213c can make it easier for the receiver to receive the signal by using dark characters and a bright background and displaying them in a large size in the horizontal direction of the screen.
FIG. 297 is a diagram showing an application example of the transmitter and the receiver in the thirteenth embodiment.
When the user operates the remote controller 7214a such as a receiver or a television, the 7214a transmits a start signal to the transmitter 7214b, and the transmitter 7214b transmits a signal for a certain period of time after receiving the start signal. The transmitter 7214b displays a display 7214c indicating that transmission is in progress. This makes it easier for the receiver to receive the signal even when the original display of the television is dark. The display 7214c has many bright areas, and the wider it is horizontally, the easier it is for the receiver to receive the signal.
The transmitter 7214b may include a region 7214c for signal transmission in addition to the region for displaying a television image. The transmitter 7214b may recognize the movement of the user or the movement of the remote controller 7214a by the camera 7214d or the microphone 7214e, and may start signal transmission.
FIG. 298 is a diagram showing an application example of the transmitter and the receiver in the thirteenth embodiment.
The transmitters 7215a and 7215b transmit the ID number of the transmitter. The transmitter ID may be completely unique, or it may be a unique ID in an area, a building, or a room. In the latter case, it is desirable that the same ID does not exist within several tens of meters. The receiver 7215c transmits the received ID to the server 7215d. The position information of the 7215c recognized by the position sensor such as GPS, the terminal ID, the user ID, the session ID, etc. of the 7215c may be transmitted to the server at the same time.
The database 7215e is associated with another ID, transmitter location information (latitude, longitude, altitude, room number), transmitter model number, shape, size, characters, images, etc. in association with the ID transmitted by the transmitter. It holds content such as video and music, commands and programs to be executed by the receiver, URLs of other servers, transmitter owner information, ID registration date and expiration date, and so on.
The server 7215d reads the information associated with the received ID from the database and sends it to the receiver 7215c. The receiver 7215c performs processing such as displaying the received information, accessing another server based on the received information, and executing the received instruction.
FIG. 299 is a diagram showing an application example of the transmitter and the receiver in the thirteenth embodiment.
As in the case of FIG. 298, the transmitters 7216a and 7216b transmit the transmitter ID1. The receiver 7216c transmits the received ID1 to the server A7216d. Server A sends the information (URL, password, etc.) associated with ID1 to access Server B, which is different from ID2. The receiver 7216c transmits ID2 to the server B7216f. The server B7216f sends the information associated with ID2 to the receiver 7216c and performs the processing associated with ID2.
FIG. 300 is a diagram showing an application example of the transmitter and the receiver in the thirteenth embodiment.
As in the case of FIG. 298, the transmitters 7217a and 7217b transmit the transmitter ID1. The receiver 7217c transmits the received ID1 to the server A7217d. Server A sends the information associated with ID1 and the randomly generated key information to server B. The key information may be generated by the server B and sent to the server A. Server A sends the key information and information for accessing server B (URL, password, etc.) to the receiver. The receiver 7217c transmits the key information to the server B7217f. The server B7217f sends the information associated with ID2 to the receiver 7217c and performs the processing associated with ID2.
FIG. 301A is a diagram showing an example of a transmission signal according to the thirteenth embodiment.
The signal is header part 7218a, data part 7218b, padding part 7218c, End of It consists of the Data part 7218e. The signal transmits the same data repeatedly for 1/15 second. This makes it possible to decode the signal even if only one copy of the signal is received. The receiver extracts the header part from the received signal and decodes the data using the part between the two header parts as the data part. If the data part of one frame is shortened, decoding becomes possible even if the transmitter is small in the image pickup part of the receiver, and if the data part is lengthened, the communication speed can be increased. By repeating the same data for 1/15 second, the receiver that shoots 30 frames per second can reliably capture the signal of the data section even if there is blanking. Further, since the same signal is received in either of the adjacent frames, it can be used for checking the reception result. In addition, signals can be received even when non-continuous frames are not processed due to the operation of other applications, or even if the receiver can shoot only 15 frames per second. Further, since the closer to the header portion, the easier it is to receive the data, it is possible to arrange the data with high importance in the portion closer to the header portion.
FIG. 301B is a diagram showing another example of the transmission signal in the thirteenth embodiment.
The signal is composed of a header section 7218a, a data section 7218b, a padding section 7218c, and an End of Data section 7218e, as in the example of FIG. 301A. The signal transmits the same data repeatedly for 1/30 seconds. This makes it possible to decode the signal even if only one copy of the signal is received. If the data unit is shortened, decoding becomes possible even if the transmitter appears small in the image pickup unit of the receiver, and if the data unit is lengthened, the communication speed can be increased. By repeating the same data for 1/30 second, the receiver that shoots 30 frames per second can reliably capture the signal of the data section even if there is blanking. Further, since the same signal is received in either of the adjacent frames, it can be used for checking the reception result. Further, since the closer to the header portion, the easier it is to receive the data, it is possible to arrange the data with high importance in the portion closer to the header portion.
FIG. 302 is a diagram showing an example of a transmission signal according to the thirteenth embodiment.
The 7219a modulation method, which modulates a 2-bit signal into a 5-bit signal, is inferior in modulation efficiency to a modulation method such as 2200.2a, which modulates a 2-bit signal into a 4-bit signal, but in the same format as the data portion. Since the header pattern can be expressed, flicker can be suppressed as compared with inserting a header pattern of a different format. End of Data may be expressed by using a header in the data part.
FIG. 303A is a diagram showing an example of a transmission signal according to the thirteenth embodiment.
The signal is composed of a data unit 7220a, a buffer unit 7220b, and an End of Data unit 7220d. The buffer part may not be provided. The signal transmits the same data repeatedly for 1/15 second. When using FM modulation or the like that transmits a signal according to the emission frequency, a header such as 7218a becomes unnecessary.
FIG. 303B is a diagram showing another example of the transmission signal in the thirteenth embodiment.
The signal is composed of a data unit 7220a, a buffer unit 7220b, and an End of Data unit 7220d, as in the example of FIG. 303A. The buffer part may not be provided. The signal transmits the same data repeatedly for 1/30 seconds. When using FM modulation or the like that transmits a signal according to the emission frequency, a header such as 7218a becomes unnecessary.
FIG. 304 is a diagram showing an example of a transmission signal according to the thirteenth embodiment.
Assign signals by frequency. Since the receiver obtains the frequency from the cycle of the signal, it is possible to reduce the reception error by allocating the signal so that the reciprocal or logarithm of the frequency is evenly spaced rather than allocating the frequency to the signal at equal intervals. When the light that is transmitting data 1 and data 2 is captured in one screen by the image pickup unit of the receiver, the frequency of data 1 and data 2 can be obtained by Fourier-converting the luminance value in the direction perpendicular to the exposure line. In addition, a weaker peak appears than when the light transmitting one data is imaged.
According to this method, even when light that is sequentially transmitted at multiple frequencies is imaged on one screen, the transmission frequency can be analyzed, and it takes less than 1/15 seconds or 1/30 seconds. It can be received even if the frequency of the transmission signal is changed.
In order to recognize the order of the transmitted signals, the Fourier transform may be performed in a range shorter than one screen. Further, the captured screens may be connected and the Fourier transform may be performed in a range longer than one screen. In this case, the brightness value of the blanking time of imaging should be treated as unknown.
FIGS. 305A and 305B are diagrams showing an example of a transmission signal according to the thirteenth embodiment.
When the frequency of the transmitted signal is 200 Hz or less, it seems to be blinking to humans, but when the frequency is higher than that, it seems to shine continuously. The camera shoots blinks at frequencies up to about 500Hz (1kHz depending on the conditions). Therefore, it is desirable that the signal frequency (carrier frequency) is 1 kHz or higher. If the effect of flickering on the camera is low, the signal frequency may be 200Hz or higher. Harmonic noise of the lighting device becomes large at 20 kHz or more, so to avoid this, the signal frequency should be 20 kHz or less. Also, in the range of 500Hz to 3kHz, sound is generated due to the vibration of the coil, so it is necessary to take measures such as setting the signal frequency to 3kHz or higher or fixing the coil. When the signal frequency is 1 kHz (1 millisecond cycle), in order to recognize this signal asynchronously, it is necessary to reduce the exposure time of the image pickup device to 0.5 ms (= 1/2000 sec) or less, which is half. When frequency modulation is used as the signal modulation method, the exposure time of the image pickup apparatus must be similarly set to half or less of the signal cycle according to the sampling theorem. However, in the case of the modulation method that expresses the value by the frequency itself as shown in Fig. 304, the frequency can be estimated from the signal values at multiple time points, so the exposure time of the image pickup device is about 4 times or less of the signal cycle. good.
FIG. 306 is a diagram showing an application example of the transmitter according to the thirteenth embodiment.
Transmitter 7223a, such as lighting, transmits an ID. The receiver 7223b such as a personal computer receives the ID and sends the ID and the file 7223e to the server 7223d. The server 7223d stores the file 7223e in association with the ID, and grants access permission to the file to the personal computer that has sent the same ID. At this time, a plurality of access controls may be performed depending on the ID, such as read-only permission and read / write permission. The receiver 7223c such as a personal computer receives the ID, sends the ID to the server 7223d, and accesses the file 7223e on the server. The server 7223d deletes the file or initializes the access control when a certain period of time has passed since the last access to the file or when the personal computer 7223b sends a different ID. The personal computer 7223b or the personal computer 7223c may send the ID.
FIG. 307 is a diagram showing an application example of the transmitter according to the thirteenth embodiment.
The transmitter 7224b registers its own ID information in the server 7224d. The receiver 7224a displays coupons, admission tickets, member information, and prepaid information on the screen. Transmitter 7224b transmits the ID. The receiver 7224a receives the ID and transmits the received ID, the user ID, the terminal ID, and the information displayed on the screen to the server 7224d. The server 7224d confirms that the content displayed by the receiver 7224a is valid, and sends the result to the display device 7224c. Note that the server 7224d may transmit the key information that changes with time to the transmitter 7224b, and the transmitter 7224b may transmit the key information. The server 7224d may be mounted as the same device as the transmitter 7224b and the display device 7224c. The receiver 7224a displays coupons, admission tickets, member information, and prepaid information on the screen in the form of a two-dimensional bar code, and in the method of reading it, it can be easily disguised by displaying an image that is a copy of the screen. By using this method, it is possible to prevent camouflage by copying the screen.
FIGS. 308 to 310 are diagrams for explaining the image pickup device according to the thirteenth embodiment.
FIG. 308 shows a front view of the image pickup device 800 of the present invention. As described with reference to the figures of the previous embodiment, in order to improve the optical communication speed of the present invention, as shown in FIG. 310, the scanning line of the 830a in the region is tracked while the optical signal generation unit 830 is tracked. For example, if only the data of the line between n = 4 and n = 7 is sent to the vertical access means 802 as a scanning line selection signal, repeatedly scanned, and acquired, the data is continuously scanned as shown in the lower part of FIG. 310. The optical signal of the present invention can be taken out. Specifically, continuous signals with 4, 5, 6, 7 and blank periods, 4, 5, 6, 7 blank periods are obtained. In the current image sensor process, this blanking can be kept within 2 μs. If it is set to 2 μs or less, one frame is 33 ms at 30 fps and 33 μs / line is 1000 lines, so the data can be demodulated almost continuously.
In the present invention, in the rolling shutter type image sensor (image sensor), the shutter speed is first increased to display the line of the present invention, and then a signal is obtained. After that, the image 830 of the light source is photographed by the camera. It moves up, down, left and right due to the camera shake of the person. Then, the image 830 will be a part of the line n = 4 ~ 7. The signal is interrupted and an error occurs. Therefore, first, the image 830 is fixed when the image is corrected by using the image stabilization detection correction means 832. Alternatively / /, the line number n of the image 830 is specified by using the means 834 for detecting the line number at the position of the image 830, and the vertical access means is controlled by the line selection unit 835 to control the desired line n (for example,). , N = 7 ~ 10), the image 830 is obtained and a continuous signal is obtained, so that the data has few errors and high-speed data reception is possible.
Returning to FIG. 308, the image sensor 800 will be described. The horizontal pixels are a to k, and can be accessed by the horizontal access means 801. The pixels in the vertical direction have 12 columns from n = 1 to n = 12, and are read out for each column from 803a to 803n, read out to the line memory 805 at the time of reading, and output from the output unit 808.
As shown in FIG. 309, in the present invention, first, in the normal shooting mode, the reading is sequentially performed as shown in (a). A blank period 821 is provided between the normal frames, during which various adjustments related to video signals such as colors are performed.
Although it depends on the image sensor, it is not possible to obtain a signal during the 5% to 20% time zone. Since the reception pattern peculiar to the present invention cannot be obtained, when the data signal reception mode is set in step 820c, first, the shutter speed is increased and the gain is increased to receive the data. If Yes, the blank period 821 is shortened to the blank period 821a by canceling a part of the video shooting work such as the color, brightness, and sensitivity. As a result, the blank period 821a can be reduced to 2 μs or less in the current process by shortening the omission of adjustment work, so that the burst error of the input signal can be greatly reduced, and the transmission speed can be greatly increased. be able to.
Next, as shown in FIG. 310, when the video 830 is captured only part of the image, information on lines other than n = 4 to 8 cannot be obtained, resulting in a large burst error, a decrease in reception efficiency, and a transmission amount. It drops to a large extent.
The position and size of the image 830 are detected by the image position detecting means 834 in FIG. 310, and if the image is small, the line (n = 4 to 7) in which the image 830 is taken by switching to the high-speed reading mode in step 820d. ) Only scan. As shown in (c), the line signals 803d, 803e, 803f, and 803g are repeatedly read many times, and the pattern peculiar to the present invention is seamlessly read. Therefore, continuous data reception with almost no burst error is possible, and a large amount of data rate improvement is possible.
Specifically, if the carrier wave is 4.8 KHz with the current image sensor, a transmission rate of about 2400 bps can be obtained. In the future, as the speed of the image sensor increases, a transmission rate of several tens of kbps will be obtained.
When the data reading is completed in step 820e, the shutter speed is slowed down, the blank period is lengthened, and the normal shooting mode of (a) is returned.
By shortening the blank period and repeatedly reading the specific line, the synchronization signal and the address can be read reliably, and the transmission speed of the pattern transmission method of the present invention can be greatly increased.
(Modified Example) Here, a modified example or a supplement for each of the above embodiments will be described.
FIG. 311A is a flowchart showing the processing operation of the receiving device (imaging device). Note that FIG. 311A shows a more detailed processing operation than the processing operation of FIG. 51.
Here, the image pickup unit of the receiver is not a method of exposing all the light receiving elements at the same time (global shutter method), but a method of sequentially exposing a part of the light receiving elements at different times (rolling shutter method, focal plane shutter method). ) Is used. The "exposure" used in the description of the present invention is an exposure method in which the time during which light is applied to the image sensor by a physical shutter is controlled, and an exposure method in which only the output of the image sensor within a specific time is taken out by an electronic shutter. And include.
First, in step 7340a, if the imaging mode is the global shutter mode, the mode is changed to the rolling shutter mode. Next, in step 7340b, the shutter speed is such that the moving average brightness for a time width of 5 milliseconds or more does not change, and the emission line is captured when a subject whose brightness is changing is captured in the region of 5 milliseconds or less. To set.
In step 7340c, the sensitivity of the light receiving element is set so that the difference between the bright portion and the dark portion of the emission line becomes large. Next, in step 7340d, the imaging mode is set to the macro imaging mode. Alternatively, set the focal length shorter than focusing on the transmitter. As a result, the transmitter is blurred and a large image is taken, so that the number of exposure lines on which the emission line is imaged can be increased.
At step 7340e, observe the change in brightness of the emission line in the direction perpendicular to the exposure line. Next, in step 7340f, the interval of the portion where the brightness is suddenly higher than that of other places or the interval of the portion where the brightness is suddenly lowered is obtained, and the transmission signal is read from this interval. Alternatively, the period of change in luminance is obtained, and the transmission signal is read from that period.
FIG. 311B is a diagram showing a comparison between an image obtained in the normal imaging mode and an image obtained in the macro imaging mode. As shown in FIG. 311B, the image 7307b obtained by imaging a light emitting subject in the macro imaging mode contains more bright regions than the image 7307a obtained by imaging the same subject in the normal imaging mode. As a result, in the macro imaging mode, it is possible to increase the number of exposure lines capable of generating emission lines for the subject.
FIG. 312 is a diagram showing a display device that displays an image or the like.
For example, the display device 7300a provided with a liquid crystal display or the like displays an image in the image area 7300b and displays various information in the information display area 7300c. The display device 7300a is configured as a transmitter (transmitter), and transmits a signal by changing the brightness of the backlight.
FIG. 313 is a diagram showing an example of the processing operation of the display device 7300a.
First, in step 7350a, the signal transmission mode is set. Next, in step 7350b, a signal is transmitted by changing the brightness of the backlight in the information display area 7300c.
FIG. 314 is a diagram showing an example of a portion of the display device 7300a for transmitting a signal.
The display device 7300a transmits a signal by changing the brightness of the part where the backlight is turned on (7301d, 7301f, 7301g, 7301i), and signals from other parts (7301c, 7301e, 7301h, 7301j). Do not send.
FIG. 315 is a diagram showing another example of the processing operation of the display device 7300a.
First, in step 7351a, the signal transmission mode is set. Next, in step 7351b, when the backlight is turned off at the time of screen switching in order to improve the dynamic resolution, the signal is transmitted only in the part where the backlight is turned on. Then, in step 7351c, no signal is transmitted while the backlight of all parts of the screen is turned off.
FIG. 316 is a diagram showing another example of a part of the display device 7300a that transmits a signal.
The display device 7300a turns off the backlight control for improving the dynamic resolution of each part (7302b, 7302e, 7302g, 7202j), and transmits a signal from these parts. On the other hand, the display device 7300a turns on the backlight control for improving the dynamic resolution of each other part (7302c, 7302d, 7302h, 7301i).
FIG. 317 is a diagram showing still another example of the processing operation by the display device.
First, in step 7352a, the signal transmission mode is set. Next, in step 7352b, the backlight control for improving the dynamic resolution of a part of the screen (7302b, 7302e, 7302g, 7202j) is turned off, and the signal is transmitted from that part.
Then, in step 7352c, the average brightness of the backlight is adjusted so that the brightness of the portion transmitting the signal and the average brightness of the backlight of the portion not transmitting the signal are equal to each other. This adjustment may be performed by adjusting the ratio of the blinking time of the backlight at the time of signal transmission, or by adjusting the maximum brightness of the backlight.
FIG. 318 is a diagram showing a configuration of a communication system including a transmitter and a receiver.
This communication system includes transmitters 7303a, 7303b, a control device 7303c, a network 7303d, an ID management server 7303e, a wireless access point 7303f, and a receiver 7303g, 7303h.
FIG. 319 is a flowchart showing the processing operation of the communication system of FIG. 318.
First, in step 7353a, the transmitter ID and wireless access point 7303f information (SSID, password, wireless access point ID, radio frequency, access point location information, connectable location information, etc.) are sent to the ID management server 7303e. ) And the information of the controller 7303c (IP address, etc.) are stored in association with each other. Then, in step 7353b, the transmitter 7303a or 7303b transmits the ID of the transmitter 7303a or 7303b. The transmitter 7303a or 7303b may further transmit the information of the wireless access point 7303f and the information of the control device 7303c. Then, in step 7353c, the receiver 7303g or 7303h receives the ID of the transmitter 7303a or 7303b, and acquires the information of the wireless access point 7303f and the information of the control device 7303c from the ID management server 7303e. Alternatively, the receiver 7303g or 7303h receives the ID of the transmitter 7303a or 7303b and the information of the wireless access point 7303f.
Then in step 7353d, the transmitter 7303a or 7303b connects to the wireless access point 7303f. Then, in step 7353e, the transmitter 7303a or 7303b transmits the address of the ID management server 7303e on the network, the command to the ID management server 7303e, and the ID of the transmitter 7303a or 7303b to the control device 7303c.
Further, in step 7353f, the control device 7303c transmits the reception ID to the receiver 7303g or 7303h. Next, in step 7353g, the control device 7303c issues a command to the ID management server 7303e on the network and obtains a response. At this time, the control device 7303c operates as a proxy server.
Further, in step 7353h, the control device 7303c transmits the response and the reception ID from the transmitter 7303a or 7303b indicated by the transmitter ID. This transmission may be repeated until a reception completion notification is sent from the receiver 7303g or 7303h, or until a certain period of time has elapsed.
Then, in step 7353i, the receiver 7303g or 7303h receives the response. Further, in step 7353j, the receiver 7303g or 7303h transmits the reception ID to the control device 7303c to notify the completion of reception.
Then, in step 7353k, if the receiver 7303g or 7303h is in a position where the signal of the transmitter 7303a or 7303b cannot be received, the controller 7303c is notified to return a response via the wireless access point 7303f. You may.
FIG. 320 is a diagram showing a modified example of signal transmission in each of the above embodiments.
In the receiving method of the present invention, the signal transmission efficiency is higher as the light emitting part of the transmitter is larger in the image pickup unit of the receiver. Transmission efficiency is poor. Therefore, the efficiency of signal transmission can be improved by shining the illumination light of the transmitter 7313a on a wall, ceiling, floor, lighting umbrella, etc., and imaging the reflected light 7313b with the receiver 7313c.
FIG. 321 is a diagram showing a modified example of signal transmission in each of the above embodiments.
The transmitter 7314d projects an illumination light including a transmission signal on the exhibit 7314a, and the reflected light 7314b is imaged by the receiver 7314c to transmit a signal.
FIG. 322 is a diagram showing a modified example of signal transmission in each of the above embodiments.
The signal transmitted by the transmitter 7315a is received by the receiver 7315b equipped with an illuminance sensor. Since the receiver 7315b receives the signal using the illuminance sensor instead of the image sensor, it has low power consumption, is suitable for constantly receiving the signal, is lightweight, and can be manufactured at low cost.
The receiver 7315b is configured as part of eyeglasses, earrings, hair ornaments, watches, hearing aids, necklaces, wands, wheelbarrows, and shopping carts. The receiver 7315b displays video, reproduces audio, and vibrates according to the received signal. In addition, the receiver 7315b transmits the received signal to the mobile information terminal 7315c via a wireless or wired transmission line.
FIG. 323A is a diagram showing a modified example of signal transmission in each of the above embodiments.
The projector 7316a transmits a signal using the projected light as a transmission signal. The receiver 7316c receives the signal by capturing the reflected light from the screen 7316b. The receiver 7316c displays the content projected by the projector 7316a and its incidental information on the screen 7316d. The content displayed on the screen 7316d may be transmitted as a transmission signal, or may be acquired from the server 7316e based on the ID included in the transmission signal.
FIG. 323B is a diagram showing a modified example of signal transmission in each of the above embodiments.
The receiver 7317b receives the signal transmitted from the transmitter 7317a. The receiver 7317b transmits voice to the earphone or the hearing aid 7317c registered in the receiver 7317b. If the user profile registered in the receiver 7317b contains a visually impaired person, the receiver 7317b transmits a commentary voice for the visually impaired person to the earphone 7317c.
FIG. 323C is a diagram showing a modified example of signal transmission in each of the above embodiments.
The signal transmitted from the transmitters 7318a and 7318b is received by the receiver 7318c. The receiver 7318c may receive the signal using the illuminance sensor. The receiver 7318c is equipped with a highly directional illuminance sensor, so that the direction in which the transmitter is located can be estimated accurately. Further, the receiver 7318c can widen the range in which the transmission signal can be received by providing the receiver 7318c with a plurality of illuminance sensors. The receiver 7318c transmits the received signal to the earphone 7318d and the head-mounted display 7318e.
FIG. 323D is a flowchart showing the processing operation of a communication system including a display or a projector and a receiver. It should be noted that this flowchart shows the processing operation corresponding to the example of signal transmission shown in FIGS. 323A to 323C.
First, in step 7357a, the transmitter ID, the display content ID, and the content to be displayed on the display or the projector are associated and recorded in the ID management server. Then, in step 7357b, the transmitter displays the content on the display or projector and transmits the signal using the backlight of the display or the projected light of the projector. The transmission signal may include the ID of the transmitter, the display content ID, the URL in which the display content is stored, and the display content itself.
Further, in step 7357c, the receiver receives the transmission signal. Then, in step 7357d, the receiver acquires the content displayed on the display or the projector by the transmitter based on the received signal.
Next, in step 7357e, if a user profile is set for the receiver, the content suitable for the profile is acquired. For example, if a profile of poor hearing is set, subtitle data and audio content for hand playback are acquired, and if a profile of poor visual acuity is set, content for audio commentary is acquired.
Further, in step 7357f, the receiver displays the acquired image content on the display of the receiver, and reproduces the acquired audio content from the speaker, earphone, or hearing aid of the receiver.
FIG. 324 is a diagram showing an example of a transmission signal according to the twelfth embodiment. note that. FIG. 324 is a detailed description of the transmitted signal of FIG. 250.
When the transmission signal is encoded by the method shown in FIGS. 7 to 87, 302, the receiver can decode the transmission signal by detecting the points 7308c, 7308d, and 7308e where the luminance value suddenly increases. At this time, the transmission signals 7308a and 7308b are equivalent and represent the same signal.
Therefore, the average brightness can be changed by adjusting the time for lowering the brightness as in the transmission signals 7308a and 7308b. When it is necessary to change the brightness of the transmitter, the brightness can be adjusted without changing the content of the transmission signal by adjusting the average brightness in this way.
FIG. 325 is a diagram showing an example of a transmission signal according to the first embodiment. Note that FIG. 325 is a detailed description of the transmission signal of FIG.
The transmission signals 7309a and 7309b can be regarded as equivalent to the transmission signal 7309c when the average luminance of about 7309d is taken. By changing the brightness in a time width that cannot be observed by other receivers, such as the transmission signals 7309a and 7309b, another signal can be superimposed.
FIG. 326 is a diagram showing another example of the transmission signal in the first embodiment. Note that FIG. 326 is a detailed description of the transmission signal of FIG.
Another signal is superimposed by adding the luminance change in the time width that cannot be observed by other receivers to the transmission signal 7310a to obtain 7310c. When the signal cannot be superimposed in the section where the brightness is lowered in the transmission signal 7310a, the high-speed modulation signal can be transmitted intermittently by adding the start signal and the end signal such as 7310e to the high-speed modulation portion.
FIG. 327A is a diagram showing an example of an image pickup device of the receiver in each of the above embodiments.
Since many image sensors are arranged like the 7311a, the transmitter cannot be imaged while the optical black is being imaged. However, by arranging the image sensor as in 7311b, the transmitter can be imaged for a longer time.
FIG. 327B is a diagram showing a configuration example of an internal circuit of the image pickup device of the receiver in each of the above embodiments.
The image pickup apparatus 7319a includes a shutter mode change unit 7319b that switches between a global shutter mode and a rolling shutter mode. The receiver changes the shutter mode to the rolling shutter mode when starting reception, and changes the shutter mode to global shutter mode at the end of reception, or returns to the setting before the start of reception.
FIG. 327C is a diagram showing an example of a transmission signal in each of the above embodiments.
If the carrier wave is 1 kHz as the frequency at which the camera does not show flicker, one slot is 1 millisecond (7320a). At this time, in the modulation method (4PPM modulation) shown in FIG. 8, the average of one symbol (4 slots) is 75% (7320b). The range of the moving average for 4 milliseconds is 75% ± modulation / 4. The smaller the degree of modulation, the smaller the flicker. When considering one symbol as one cycle, if the frequency at which humans do not feel flicker is 200 Hz or higher, the carrier wave is 800 Hz or higher, and if the frequency at which flicker does not appear on the camera is 1 kHz or higher, the carrier wave is 4 kHz. That is all.
Similarly, when the carrier wave is 1 kHz, the average of 1 symbol (5 slots) is 80% in the modulation method (5PPM modulation) shown in FIG. 302 (7320c). The range of the moving average for 5 milliseconds is 80% ± modulation degree / 5. The smaller the degree of modulation, the smaller the flicker. When considering one symbol as one cycle, if the frequency at which humans do not feel flicker is 200 Hz or higher, the carrier wave is 1 kHz or higher, and if the frequency at which flicker does not appear on the camera is 1 kHz or higher, the carrier wave is 5 kHz. That is all.
FIG. 327D is an example of a transmission signal in each of the above embodiments.
The header pattern is different from the pattern representing the data, and the average brightness must be equal to the pattern representing the data in order to eliminate flicker. There are patterns such as 7321b, 7321c, 7321d, and 7321e as patterns in which the average brightness becomes equal to the data pattern of the modulation method of 2200.2a. If the brightness value can be controlled in stages, 7321b is desirable. When the change in brightness is sufficiently fast compared to the exposure time of the image pickup device of the receiver, such as 7321e, it is observed by the receiver as 7321b. In the modulation method of 7219a, the modulation method is defined including the header pattern.
FIG. 328A is a diagram for explaining an imaging mode of the receiver.
In the normal imaging mode, the receiver acquires the image 7304a by performing imaging using all the exposure lines (imaging lines) included in the image sensor. For example, all exposure lines are 3000. Then, the receiver acquires one image between the times t1 and t4 and further acquires one image between the times t5 and t8 by the image pickup.
Here, if a subject, which is a transmitter, is shown only in a part of the image, the receiver may not receive the signal from the subject. For example, if only the 1001st to 2000th exposure lines capture the subject and the other exposure lines do not capture the subject, then the 1001st to 2000th exposure lines are not exposed, that is, the 1st to 2000th exposure lines. When the 1000th exposure line is exposed (time t1 ~ t2, t5 ~ t6) and when the 2001th ~ 3000th exposure line is exposed (time t3 ~ t4, t7 ~ t8), The signal from the subject cannot be received.
Therefore, when the imaging mode is switched from the normal imaging mode to the special imaging mode A, the receiver uses only the exposure line that captures the subject among all the exposure lines for imaging. That is, the receiver uses only the 1001st to 2000th exposure lines for imaging between the times t1 to t4 and between the times t5 and t8. Further, in this special imaging mode A, the 1001st to 2000th exposure lines are uniformly exposed only once in order over the entire imaging time of one frame at time t1 to t4 or time t5 to t8. NS. This makes it possible to prevent the signal from being missed from the subject.
FIG. 328B is a flowchart showing a processing operation using the special imaging mode A of the receiver.
First, in step 7354a, the portion where the emission line is imaged is found from the captured image. Then, in step 7354b, turn on the image stabilization function.
Then, in step 7354c, the mode is switched to the special imaging mode A in which imaging is performed using only the pixels of the exposure line on which the emission line is captured. In the special imaging mode A, the time from the start of exposing an exposure line to the start of imaging the next exposure line is set to be evenly spaced in time while one image is captured (for example, at times t1 to t4). Set the exposure time of each exposure line. It should be noted that the pixels in the direction perpendicular to the exposure line may be thinned out for imaging.
As a result, the number of frames output from the image pickup unit of the receiver is the same as in the normal image pickup mode. Therefore, in this special image pickup mode A, a receiver having a low-performance processing device or a processing device that also performs other processing is performed. Suitable for receivers with.
Further, in step 7354d, the area to be imaged in the special imaging mode A is specified. Here, by designating a region narrower than the region where the emission line is captured as the region to be imaged, the emission line can be continuously imaged even if the imaging direction is changed due to camera shake or the like.
Then, in step 7354e, the movement of the captured image is detected. By moving the area to be imaged in the moving direction, it is possible to continue to image the emission line even if the position of the captured image changes. Next, in step 7354f, the information transmitted from the emission line pattern is acquired.
FIG. 329A is a diagram for explaining other imaging modes of the receiver.
When the imaging mode is switched from the normal imaging mode to the special imaging mode B, the receiver uses only the exposure line that captures the subject among all the exposure lines for imaging. That is, the receiver uses only the 1001st to 2000th exposure lines for imaging between the times t1 to t4 and between the times t5 and t8. Further, in this special imaging mode B, the 1001st to 2000th exposure lines are sequentially exposed multiple times over the entire imaging time of one frame at time t1 to t4 or time t5 to t8. NS. This makes it possible to prevent the signal from being missed from the subject.
FIG. 329B is a flowchart showing a processing operation using the special imaging mode B of the receiver.
First, in step 7355a, the portion where the emission line is imaged is found from the captured image. Then, in step 7355b, turn on the image stabilization function.
Then, in step 7355c, the mode is switched to the special imaging mode B in which imaging is performed using only the pixels of the exposure line on which the emission line is captured. In the special imaging mode B, imaging is performed at high speed by imaging only the area where the emission line is imaged. It should be noted that the pixels in the direction perpendicular to the exposure line may be thinned out for imaging.
Next, in step 7355d, the area to be imaged in the special imaging mode B is specified. Here, by designating a region narrower than the region where the emission line is captured as the region to be imaged, the emission line can be continuously imaged even if the imaging direction is changed due to camera shake or the like.
Then, in step 7355e, the movement of the captured image is detected. By moving the area to be imaged in the moving direction, it is possible to continue to image the emission line even if the position of the captured image changes. Next, in step 7355f, the information transmitted from the emission line pattern is acquired.
FIG. 330A is a diagram for explaining still another imaging mode of the receiver.
When the imaging mode is switched from the normal imaging mode to the special imaging mode C, the receiver uses only the exposure line that captures the subject among all the exposure lines for imaging. That is, the receiver uses only the 1001st to 2000th exposure lines for imaging between the times t1 to t4 and between the times t5 and t8. Further, in this special imaging mode C, the 1001st to 2000th exposure lines are sequentially exposed multiple times over the entire imaging time of one frame at time t1 to t4 or time t5 to t8. NS. Further, in this special imaging mode C, the plurality of images obtained by executing the multiple images are not output individually, and one image including the plurality of images (the same as the image generated in the normal imaging mode). Size image) is output. This makes it possible to prevent the signal from being missed from the subject.
FIG. 330B is a flowchart showing a processing operation using the special imaging mode C of the receiver.
First, in step 7356a, the portion where the emission line is imaged is found from the captured image. Then, in step 7356b, turn on the image stabilization function.
Then, in step 7356c, the mode is switched to the special imaging mode C in which imaging is performed using only the pixels of the exposure line on which the emission line is captured. The special imaging mode C is a method of creating a single image by imaging only the region where the emission line is captured, ignoring the original pixel arrangement, and arranging the imaging results. It should be noted that the pixels in the direction perpendicular to the exposure line may be thinned out for imaging.
As a result, the number of frames output from the image pickup unit of the receiver is the same as in the normal image pickup mode. Therefore, this special image pickup mode C is a receiver having a low-performance processing device or a processing device that also performs other processing. Suitable for receivers with.
Further, in step 7356d, the area to be imaged in the special image pickup mode C is specified. Here, by designating a region narrower than the region where the emission line is captured as the region to be imaged, the emission line can be continuously imaged even if the imaging direction is changed due to camera shake or the like.
Then, in step 7356e, the movement of the captured image is detected. By moving the area to be imaged in the moving direction, it is possible to continue to image the emission line even if the position of the captured image changes. Next, in step 7356f, the information transmitted from the emission line pattern is acquired.
Although the information and communication methods according to one or more embodiments have been described above based on the embodiments, the present invention is not limited to the embodiments. As long as it does not deviate from the gist of the present invention, a form in which various modifications conceived by those skilled in the art are applied to the present embodiment or a form constructed by combining components in different embodiments is also within the scope of one or a plurality of embodiments. May be included within.
FIG. 331A is a flowchart of the information communication method according to one aspect of the present invention.
The information communication method according to one aspect of the present invention is an information communication method for acquiring information from a subject, and includes steps SA11, SA12, and SA13.
That is, in this information communication method, in the image obtained by imaging the subject by the image sensor, a bright line corresponding to the exposure line included in the image sensor is generated according to the change in the brightness of the subject. An exposure time setting step (SA11) for setting an exposure time and an imaging step (SA12) in which the image sensor acquires an image including the emission line by taking an image of the subject whose brightness changes at the set exposure time. ), And an information acquisition step (SA13) for acquiring information by demodulating the data specified by the emission line pattern included in the acquired image.
FIG. 331B is a block diagram of an information communication device according to an aspect of the present invention.
The information communication device A10 according to one aspect of the present invention is an information communication device that acquires information from a subject, and includes components A11, A12, and A13.
That is, the information communication device A10 is such that the image obtained by the image pickup of the subject by the image sensor has a bright line corresponding to the exposure line included in the image sensor according to the change in the brightness of the subject. The exposure time setting unit A11 that sets the exposure time of the image sensor, and the image sensor A12 that acquires an image including the emission line by imaging the subject whose brightness changes at the set exposure time. It is provided with a demodizing unit A13 for acquiring information by demolishing data specified by the pattern of the emission line included in the acquired image.
FIG. 331C is a flowchart of the information communication method according to one aspect of the present invention.
The information communication method according to one aspect of the present invention is an information communication method for acquiring information from a subject, and includes steps SA21 to SA26.
That is, in this information communication method, from the first imaging step (SA21) of acquiring a first image by imaging the subject using an image sensor having a plurality of exposure lines, and the first image. , A detection step (SA22) for detecting the range in which the subject is imaged, and a determination step (SA23) for determining a predetermined exposure line for imaging the range in which the subject is imaged among the plurality of exposure lines. The exposure that sets the exposure time of the image sensor so that the emission line corresponding to the predetermined exposure line is generated in the second image acquired by using the predetermined exposure line according to the change in the brightness of the subject. A second image including the emission line is acquired by taking an image of the subject whose brightness changes with the time setting step (SA24) and the predetermined exposure line at the set exposure time. It includes an imaging step (SA25) and an information acquisition step (SA26) for acquiring information by demolishing the data specified by the emission line pattern included in the acquired second image.
FIG. 331D is a block diagram of an information communication device according to an aspect of the present invention.
The information communication device A20 according to one aspect of the present invention is an information communication device that acquires information from a subject, and includes components A21 to A26.
That is, the information communication device A20 is based on the first image acquisition unit A21 that acquires a first image by taking an image of the subject using an image sensor having a plurality of exposure lines, and the first image. An exposure range detection unit A22 that detects the range in which the subject is imaged, and an exposure line determination unit A23 that determines a predetermined exposure line that captures the range in which the subject is imaged among the plurality of exposure lines. The exposure that sets the exposure time of the image sensor so that the emission line corresponding to the predetermined exposure line is generated in the second image acquired by using the predetermined exposure line according to the change in the brightness of the subject. A second image acquisition that acquires a second image including the emission line by taking an image of the subject whose brightness changes at the set exposure time using the time setting unit A24 and the predetermined exposure line. It includes a unit A25 and a demodulation unit A26 that acquires information by demolishing the data specified by the pattern of the emission line included in the acquired second image.
The above-mentioned bright line pattern is synonymous with the difference in the interval between the bright lines.
FIG. 332 is a diagram showing an example of an image obtained by the information communication method according to one aspect of the present invention.
For example, the exposure time is set shorter than 10 milliseconds for a subject whose brightness changes at a frequency of 200 Hz or higher. Here, each of the plurality of exposure lines included in the image sensor is sequentially exposed at different timings. In such a case, as shown in FIG. 332, some emission lines are generated in the image obtained by the image sensor. That is, the image contains emission lines parallel to the exposure line. Further, in the information acquisition step (SA13 or SA26), among the patterns of the emission lines, the data specified by the pattern in the direction perpendicular to the exposure line is demodulated.
In the information communication method and the information communication device A10 shown by FIGS. 331A and 331B, the information transmitted by the change in the brightness of the subject is acquired by the exposure of the exposure line of the image sensor, and therefore, for example, wireless communication is performed. It is possible to enable communication between various devices without requiring a special communication device for the purpose. Further, in the information communication method and information communication device A20 shown by FIGS. 331C and 331D, of all the exposure lines included in the image sensor, only the exposure line that captures the subject is the second image including the emission line. Since it is used for acquisition, it is possible to omit processing for an exposure line that does not capture the subject, improve the efficiency of information acquisition, and prevent information from being missed from the subject.
FIG. 333A is a flowchart of an information communication method according to another aspect of the present invention.
The information communication method according to another aspect of the present invention is an information communication method for transmitting a signal by changing the luminance, and includes steps SB11, SB12 and SB13.
That is, in this information communication method, the transmission is performed by a determination step (SB11) of determining a pattern of luminance change by modulating a signal to be transmitted and a luminous body changing the luminance according to the determined pattern. The first transmission step (SB12) for transmitting the signal of interest and the luminance change according to the same pattern as the determined pattern within 33 milliseconds after the light emitter transmits the signal of the transmission target. This includes a second transmission step (SB13) of transmitting the same signal as the signal to be transmitted. Then, in the determination step (SB11), the pattern is determined so that each luminance value when moving averaged with a width of 5 milliseconds or more with respect to the luminance change falls within a predetermined range.
FIG. 333B is a block diagram of an information communication device according to another aspect of the present invention.
The information communication device B10 according to another aspect of the present invention is an information communication device that transmits a signal by changing the luminance, and includes components B11 and B12.
That is, the information communication device B10 has a luminance change pattern determination unit B11 that determines a pattern of luminance change by modulating the signal of the transmission target, and the luminance change pattern B11 that determines the luminance change pattern according to the determined pattern of the transmission target. Light emission that transmits a signal and transmits the same signal as the signal to be transmitted by changing the luminance according to the same pattern as the determined pattern within 33 milliseconds after transmitting the signal to be transmitted. Equipped with body B12. Then, the luminance change pattern determination unit B11 determines the pattern so that each luminance value when the moving average is performed with a width of 5 milliseconds or more with respect to the luminance change falls within a predetermined range.
In such an information communication method and information communication device B10 shown by FIGS. 333A and 333B, each luminance value when moving averaged with a width of 5 milliseconds or more with respect to a change in luminance is within a predetermined range. Since the pattern of the luminance change is determined, the signal can be transmitted by the luminance change, although it is possible to prevent the person from feeling flicker. Furthermore, since the same signal is transmitted within 33 milliseconds, even if the receiver receiving the signal has a blanking, the signal can be reliably transmitted to that receiver.
FIG. 334A is a flowchart of an information communication method according to still another aspect of the present invention.
The information communication method according to still another aspect of the present invention is an information communication method for transmitting a signal by changing the luminance, and includes steps SC11, SC12, SC13 and SC14.
That is, in this information communication method, a determination step (SC11) in which a plurality of frequencies are determined by modulating a signal to be transmitted and a fixed frequency of any one of the plurality of determined frequencies are determined by the light emitter. The transmission step (SC12) for transmitting a signal by changing the brightness according to the frequency, and the frequency used for the change in brightness are sequentially changed to other frequencies among a plurality of determined frequencies in a cycle of 33 milliseconds or more. Includes change step (SC14). After the transmission step SC12 is executed, it is determined whether or not all the determined frequencies are used for the luminance change (SC13), and when it is determined that all the frequencies are not used (SC13). In N), update step SC14 may be executed. Further, in the transmission step (SC12), the luminous body changes the luminance so that each luminance value when the moving average is performed with a width of 5 milliseconds or more with respect to the luminance change falls within a predetermined range.
FIG. 334B is a block diagram of an information communication device according to still another aspect of the present invention.
The information communication device C10 according to still another aspect of the present invention is an information communication device that transmits a signal by changing the brightness, and includes components C11, C12, and C13.
That is, the information communication device C10 changes the brightness according to a certain frequency of the frequency determination unit C11 that determines a plurality of frequencies by modulating the signal to be transmitted and one of the determined plurality of frequencies. The illuminant C13 that transmits a signal by doing so, and the frequency changing unit C12 that sequentially changes the frequency used for the brightness change to another frequency among the determined plurality of frequencies in a cycle of 33 milliseconds or more. Be prepared. Further, the luminous body C13 changes the luminance so that each luminance value when the moving average is performed with a width of 5 milliseconds or more with respect to the luminance change falls within a predetermined range.
In the information communication method and the information communication device C10 shown by FIGS. 334A and 334B, each luminance value when moving averaged with a width of 5 milliseconds or more with respect to the luminance change is within a predetermined range. Since the pattern of the luminance change is determined, the signal can be transmitted by the luminance change, although it is possible to prevent the person from feeling flicker. In addition, many FM-modulated signals can be transmitted.
Further, the information communication device includes an information management unit that manages device information including its own unique ID and device status information, a light emitting element, and an optical transmission unit that transmits information as a blinking pattern of the light emitting element. When the internal state of the device is changed, the optical transmission unit may convert the device information into a blinking pattern of light and transmit the information.
Further, the information communication device further includes a start history management unit for storing sensed information inside the device, which indicates its own start state or user's usage history, and the optical transmission unit generates a clock to be used. The performance information registered in advance of the device may be acquired and the transmission speed may be changed.
Further, the light emitting element includes first and second light emitting elements, and the second light emitting element is arranged around the first light emitting element for transmitting information by blinking light. The second light emitting element may emit light between the end and the start of the information transmission when the information transmission by the blinking of the first light emitting element is repeated a certain number of times.
Further, the information communication device utilizes the difference in the exposure time between the image pickup unit that exposes each image pickup element with a time difference and the exposure time of each image pickup element to change the time average brightness of the object to be imaged by 1 millisecond or less. May be provided with a signal analysis unit that reads from one captured image.
Further, the time average brightness may be a time average brightness of 1 / 30,000 second or more.
The information communication device may further modulate the transmission information into a light emission pattern and transmit the information according to the light emission pattern.
The information communication device may express the transmission signal by a change in the time average brightness of 1 millisecond or less, and may change the brightness of the light emitting unit so that the time average brightness of 60 milliseconds or more is uniform. ..
The information communication device may express the transmission signal by a change in time average brightness of 1 / 30,000 second or more.
Further, the common portion between the transmission signal and the signal expressed by the time average brightness of the adjacent information communication device of the same type is that the light emitting unit emits light at the same timing as the light emitting unit of the information communication device of the same type. It may be transmitted.
Further, the non-common part between the transmission signal and the signal expressed by the time average brightness of the adjacent information communication device of the same type is a time zone in which the information communication device of the same type does not express the signal by the time average brightness. In addition, it may be expressed by the time average brightness of the light emitting unit.
Further, the information communication device includes a first light emitting unit that expresses the transmission signal by a change in time average brightness, and a second light emitting unit that does not express the transmission signal by a change in time average brightness. , The signal may be transmitted depending on the positional relationship between the first light emitting unit and the second light emitting unit.
Further, the centralized control device may include a control unit that centrally controls any of the above-mentioned information and communication devices.
Further, the building may be provided with any of the above-mentioned information and communication devices or the above-mentioned centralized control device.
Further, the train may be equipped with any of the above-mentioned information and communication devices or the above-mentioned centralized control device.
Further, the image pickup device is an image pickup device that captures a two-dimensional image, and may be used by exposing only an arbitrary image pickup element to take an image at a higher speed than exposing all the image pickup elements to take an image.
Further, the arbitrary image sensor may be an image sensor that images a pixel having the largest change in time average brightness of 1 millisecond or less, or a row of image sensors including the image sensor.
In each of the above embodiments, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. For example, the program causes the computer to execute the information communication method shown by the flowchart of any one of FIGS. 331A, 331C, 333A and 334A.
(Summary of Each of the above Embodiments and Modifications thereof) The information communication method according to one aspect of the present invention is an information communication method for acquiring information from a subject, using an image sensor having a plurality of exposure lines. A first imaging step of acquiring a first image by imaging the subject, a detection step of detecting a range in which the subject is imaged from the first image, and a plurality of exposure lines. Among them, a determination step for determining a predetermined exposure line for capturing the range in which the subject is imaged, and a second image acquired by using the predetermined exposure line have bright lines corresponding to the predetermined exposure line. Using the exposure time setting step of setting the exposure time of the image sensor and the predetermined exposure line so as to occur in response to the change in the brightness of the subject, the subject whose brightness changes is set to the exposure time. Information is obtained by a second imaging step of acquiring a second image including the emission line by imaging and by demolishing the data specified by the pattern of the emission line included in the acquired second image. Including the information acquisition step to be acquired.
As a result, the information transmitted by the change in the brightness of the subject is acquired by the exposure of the exposure line of the image sensor, so that, for example, no special communication device for performing wireless communication is required, and the information is between various devices. Communication can be enabled. Furthermore, of all the exposure lines included in the image sensor, only the exposure line that captures the subject is used to acquire the second image including the emission line, so the processing for the exposure line that does not capture the subject is omitted. This makes it possible to improve the efficiency of information acquisition and prevent information from being missed from the subject. The exposure line is a column or row included in the image sensor and composed of a plurality of pixels to be exposed at the same time. Further, the emission line is a line included in the captured image shown by, for example, FIG.
Further, the predetermined exposure line includes only the exposure line that captures the range in which the subject is imaged among the plurality of exposure lines, and does not include the exposure line that captures the range in which the subject is not imaged. You may.
As a result, the efficiency of information acquisition can be increased more reliably, and information on the subject can be prevented from being missed.
Further, in the second imaging step, a second imaging time obtained by equally dividing the first imaging time for acquiring the first image by the number of lines included in the predetermined exposure line is acquired. The second imaging time may be used as the imaging time of each exposure line included in the predetermined exposure line.
Thereby, as shown in FIGS. 328A and 328B, for example, information can be appropriately acquired from the subject which is the transmitter.
Further, in the second imaging step, the imaging time of each exposure line of the image sensor in the first imaging step may be the imaging time of each exposure line included in the predetermined exposure line.
Thereby, as shown in FIGS. 329A and 329B, for example, information can be appropriately acquired from the subject which is the transmitter.
Further, in the second imaging step, a plurality of the second images captured by the predetermined exposure line are integrated to generate a third image having the same image size as the first image. In the information acquisition step, information may be acquired by demodulating the data specified by the emission line pattern included in the third image.
Thereby, as shown in FIGS. 330A and 330B, for example, information can be appropriately acquired from the subject which is the transmitter.
Further, in the determination step, an exposure line that captures a narrower range than the range in which the subject is imaged may be determined as the predetermined exposure line among the plurality of exposure lines.
As a result, as shown in FIGS. 328B, 329B and 330B, for example, information can be appropriately acquired from the subject which is the transmitter without being affected by camera shake or the like.
Further, the first mode in which the subject is imaged using all the exposure lines among the plurality of exposure lines of the image sensor and the predetermined exposure line among the plurality of exposure lines of the image sensor are used. It may be possible to switch between the second mode for capturing the subject and the second mode.
As a result, information can be appropriately acquired from the subject, which is the transmitter, by switching.
The information communication method according to one aspect of the present invention is an information communication method for acquiring information from a subject, and an image obtained by imaging the subject by an image sensor has a bright line corresponding to an exposure line included in the image sensor. The exposure time setting step for setting the exposure time of the image sensor and the image sensor taking an image of the subject whose brightness changes at the set exposure time so that The present invention includes an imaging step of acquiring an image including the emission line, and an information acquisition step of acquiring information by demolishing the data specified by the pattern of the emission line included in the acquired image.
As a result, the information transmitted by the change in the brightness of the subject is acquired by the exposure of the exposure line of the image sensor, so that, for example, no special communication device for performing wireless communication is required, and the information is between various devices. Communication can be enabled. The exposure line is a column or row included in the image sensor and composed of a plurality of pixels to be exposed at the same time. Further, the emission line is a line included in the captured image shown by, for example, FIG.
For example, in the imaging step, each of the plurality of exposure lines included in the image sensor is sequentially exposed at different timings.
As a result, since the emission line generated by the rolling shutter type imaging is included in the position corresponding to each exposure line in the image, a lot of information can be acquired from the subject.
Here, in the information acquisition step, the data specified by the pattern in the direction perpendicular to the exposure line among the patterns of the emission lines may be demodulated.
As a result, information corresponding to the change in luminance can be appropriately acquired.
Further, in the exposure time setting step, the exposure time may be set shorter than 10 milliseconds.
This makes it possible to more reliably generate emission lines in the image.
Here, in the imaging step, the subject whose brightness changes at a frequency of 200 Hz or higher may be imaged.
As a result, as shown in FIGS. 305A and 305B, a large amount of information can be acquired from the subject without the person feeling flicker.
Further, in the imaging step, the image including the emission line parallel to the exposure line may be acquired.
As a result, information corresponding to the change in luminance can be appropriately acquired.
Here, in the information acquisition step, 0 or 1 specified by whether or not there is a bright line in the region is specified for each region corresponding to each exposure line included in the image sensor in the acquired image. The data shown may be demodulated.
This makes it possible to acquire a lot of PPM-modulated information from the subject. For example, as shown in Fig. 2, when the number of images per second (frame rate) is f and the number of exposure lines constituting one image is l, it depends on whether each exposure line receives a certain amount of light or more. Once the information is acquired, the information can be acquired at a speed of up to fl bits per second.
Further, in the information acquisition step, it may be specified whether or not there is a bright line in the region according to whether or not the luminance value in the region is equal to or greater than the threshold value for each region.
This makes it possible to appropriately acquire information from the subject.
Here, in the imaging step, the subject whose brightness changes at a constant frequency corresponding to the predetermined period is imaged at predetermined periods, and in the information acquisition step, the predetermined period is specified. The data specified by the pattern of emission lines generated in response to a change in luminance at a constant frequency corresponding to a period may be demodulated.
This makes it possible to acquire a lot of FM-modulated information from the subject. For example, as shown in FIG. 111, appropriate information can be acquired by the pattern of the emission line corresponding to the frequency f1 and the pattern of the emission line corresponding to the frequency f2.
Further, in the imaging step, the subject whose luminance changes so as to transmit a signal is imaged by adjusting the time between the change of one of the rising and falling of the brightness and the next change of the one. In the information acquisition step, the data, which is a code associated with the time specified by the pattern of the luminance line, may be demodulated.
Thereby, as shown in FIG. 248, for example, the brightness of the subject (for example, a lighting device) perceived by a person can be adjusted by PWM control without changing the content of the information transmitted from the subject.
Here, in the imaging step, the subject whose luminance changes so that each luminance value when moving averaged with a width of 5 milliseconds or more with respect to the luminance change may fall within a predetermined range may be imaged. ..
As a result, a lot of information can be acquired from the subject without the person feeling flicker. For example, as shown in FIG. 8, when the modulated signal is 0, it is non-emission, when it is 1, it is emission, and if there is no bias in the transmission signal, each luminance value at the time of moving average is the luminance value at the time of emission. Can be about 75% of. As a result, it is possible to prevent a person from feeling flicker.
Further, the pattern of the emission line differs depending on the exposure time of the image sensor, and in the information acquisition step, the data specified by the pattern according to the set exposure time may be demodulated.
As a result, as shown in FIG. 14, for example, different information can be acquired from the subject depending on the exposure time.
Here, the information communication method further detects the state of the image pickup apparatus including the image sensor, and in the information acquisition step, acquires the information indicating the position of the subject, and further, with the acquired information. , The position of the image pickup apparatus may be calculated based on the detected state.
Thereby, for example, as shown in FIG. 108, the position of the image pickup apparatus can be specified more accurately than when GPS or the like cannot be used or when GPS or the like is used.
Further, in the imaging step, the subject may be imaged, which includes a plurality of regions arranged along the exposure line and whose brightness changes for each region.
This makes it possible to acquire a lot of information from the subject, for example, as shown in FIG. 258.
Here, in the imaging step, the subject that emits a plurality of types of metallic light at different timings may be imaged.
As a result, as shown in FIG. 272, for example, a large amount of information can be acquired from the subject without the person feeling flicker.
Further, the information communication method further estimates the location where the image pickup device equipped with the image sensor exists, and in the information acquisition step, the identification information of the subject is acquired as the information, and the location and the identification information are acquired. The related information associated with may be obtained from the server.
As a result, as shown in FIGS. 282 and 283, even when the same identification information is transmitted from a plurality of lighting devices by changing the brightness, the place (building) where the image pickup device exists, that is, the lighting device. Appropriate related information can be obtained according to the location (building) where the is located.
Here, the information communication method according to one aspect of the present invention is an information communication method for transmitting a signal by changing the luminance, and a determination step of determining a pattern of the change in luminance by modulating the signal to be transmitted. The first transmission step in which the light emitter transmits the signal to be transmitted by changing the luminance according to the determined pattern, and within 33 milliseconds after the light emitter transmits the signal to be transmitted. Including a second transmission step of transmitting the same signal as the signal to be transmitted by changing the luminance according to the same pattern as the determined pattern, and in the determination step, with respect to the luminance change. The pattern is determined so that each luminance value when moving and averaging with a width of 5 milliseconds or more falls within a predetermined range.
As a result, the pattern of the brightness change is determined so that each brightness value when moving averaged with a width of 5 milliseconds or more with respect to the brightness change falls within a predetermined range, so that the person feels flickering. If it is prevented, a signal can be transmitted by changing the brightness. Further, as shown in FIG. 301B, for example, the same signal is transmitted within 33 milliseconds, so that even if the receiver receiving the signal has blanking, the signal can be reliably transmitted to the receiver. ..
Further, in the determination step, the signal represented by the two bits is modulated into a signal represented by four bits consisting of three bits indicating the same value and one bit indicating another value. The signal to be transmitted may be modulated.
As a result, for example, as shown in FIG. 8, when the modulated signal is 0, it is non-emission, when it is 1, it is emission, and if there is no bias in the transmission signal, each luminance value at the time of moving average is emitted. It can be about 75% of the brightness value of. As a result, it is possible to more reliably prevent a person from feeling flicker.
Here, in the determination step, the luminance change is performed by adjusting the time between the change of one of the rise and fall of the luminance and the change of the next one according to the signal to be transmitted. The pattern of may be determined.
Thereby, for example, as shown in FIG. 248, the brightness of the light emitting body (for example, a lighting device) perceived by a person can be adjusted by PWM control without changing the content of the signal to be transmitted.
Further, the first transmission scan in the step and said second transmission step, different signals depending on the exposure time of the image sensor for imaging the light emitting element which changes brightness is obtained in the image pickup apparatus comprising the image sensor The brightness may be changed as follows.
As a result, as shown in FIG. 14, for example, different signals can be transmitted to the image pickup apparatus depending on the exposure time.
Here, in the first transmission step and the second transmission step, the plurality of light emitters synchronize with each other to change the brightness, so that the plurality of light emitters transmit common information, and the common information is transmitted. After being transmitted, different information may be transmitted for each light emitter by changing the brightness for each light emitter.
As a result, as shown in FIG. 21, for example, by transmitting common information at the same time by a plurality of light emitters, the plurality of light emitters can be regarded as one large light emitter, and the image pickup in which the common information is received. Since the device can capture a large image of the illuminant, information can be transmitted faster and from a greater distance. Further, as shown in FIG. 109A, for example, by transmitting common information by a plurality of light emitters, it is possible to suppress the amount of information of individual information transmitted for each light emitter.
Further, the information communication method further includes an instruction receiving step for receiving an instruction as to whether or not to perform the modulation of the signal to be transmitted, and when the instruction indicating to perform the modulation is received, the determination is made. If an instruction is received indicating that the step, the first transmission step and the second transmission step are performed and the modulation is not performed, the determination step, the first transmission step and the second transmission step are received. The light emitter may be turned on or off without performing the transmission step of.
As a result, for example, as shown in FIG. 109A, it is possible to switch whether or not to perform modulation, so that it is possible to suppress the influence of noise on changes in the luminance of other light emitters.
Here, the illuminant includes a plurality of regions arranged along an exposure line of an image sensor that images the illuminant, and in the first transmission step and the second transmission step, the illuminant. The brightness may change for each region of.
This allows a lot of information to be transmitted, for example as shown in FIG. 258.
Further, in the first transmission step and the second transmission step, the light emitter may change the brightness by emitting a plurality of metallic types of light at different timings from each other.
As a result, as shown in FIG. 272, for example, a large amount of information can be transmitted without the person feeling flicker.
Here, in the first transmission step and the second transmission step, the identification information of the light emitter may be transmitted as the signal to be transmitted or the same signal.
As a result, as shown in FIG. 282, for example, the identification information of the illuminant is transmitted, so that the image pickup device that receives the identification information can send more information associated with the identification information to a communication line such as the Internet. It can be obtained from a server or the like via.
Further, the information communication method according to one aspect of the present invention is an information communication method in which a signal is transmitted by changing the brightness, and a determination step of determining a plurality of frequencies by modulating a signal to be transmitted and a light emitting body are used. , The transmission step of transmitting a signal by changing the brightness according to a certain frequency of the plurality of determined frequencies, and the frequency used for the change of the brightness among the determined frequencies. Including a change step of sequentially changing to another frequency in a cycle of 33 milliseconds or more, in the transmission step, each brightness of the light emitter is moved and averaged with a width of 5 milliseconds or more with respect to the change in brightness. The brightness may be changed so that the value falls within a predetermined range.
As a result, the pattern of the brightness change is determined so that each brightness value when moving averaged with a width of 5 milliseconds or more with respect to the brightness change falls within a predetermined range, so that the person feels flickering. If it is prevented, a signal can be transmitted by changing the brightness. In addition, many FM-modulated signals can be transmitted. For example, as shown in FIG. 111, appropriate information can be transmitted by changing the frequency of luminance change (f1, f2, etc.) in a cycle of 33 milliseconds or more.
(Embodiment 14) In the present embodiment, each application example using a receiver such as a smartphone in the above embodiments 1 to 13 and a transmitter that transmits information as a blinking pattern such as an LED or an organic EL is provided. explain.
FIG. 335 is a diagram showing an example of each mode of the receiver in the present embodiment.
In the normal shooting mode, the receiver 8000 acquires a normally shot image by shooting at a shutter speed of, for example, 1/100 second, and displays the normally shot image on the display. In this case, the subject such as a street light or a signage configured as a signboard of a store and its surroundings are clearly projected on the normally photographed image.
Further, in the visible light communication mode, the receiver 8000 acquires a visible light communication image by taking a picture at a shutter speed of, for example, 1/10000 seconds. For example, when the above-mentioned street light or signage transmits a signal by a change in brightness as the transmitter shown in the above-described first to thirteenth embodiment, in this visible light communication image, the place where the signal is transmitted is the place where the signal is transmitted. One or more emission lines (hereinafter referred to as emission line patterns) are projected, and nothing is projected other than that part. That is, in this visible light communication image, only the emission line pattern is projected, and the portion where the brightness of the subject is not changed and the periphery of the subject are not projected.
Further, in the intermediate mode, the receiver 8000 acquires an intermediate image by shooting at a shutter speed of, for example, 1/3000 second. In this intermediate image, a bright line pattern is projected, and the above-mentioned portion where the brightness of the subject does not change and the periphery of the subject are also projected. Therefore, by displaying the intermediate image on the display by the receiver 8000, the user can know where or from which position the signal is transmitted. The emission line pattern, the subject and its surroundings projected by this intermediate image are not as clear as the emission line pattern of the visible light communication image and the subject and its surroundings of the normal photographed image, respectively, but have the sharpness recognized by the user. ..
In the following description, shooting in the normal shooting mode or the normal shooting mode is referred to as normal shooting, and shooting in the visible light communication mode or the visible light communication mode is referred to as visible light shooting (visible light communication). Further, instead of normal shooting and visible light shooting, shooting in the intermediate mode may be used, or an intermediate image may be used instead of the composite image described later.
FIG. 336 is a diagram showing an example of the photographing operation of the receiver in the present embodiment.
The receiver 8000 switches the shooting mode to normal shooting, visible light communication, normal shooting, and so on. Then, the receiver 8000 generates a composite image in which the emission line pattern and the subject and its surroundings are clearly projected by synthesizing the normally captured image and the visible light communication image, and displays the composite image on the display. .. This composite image is an image generated by superimposing a bright line pattern of a visible light communication image on a portion of a normally captured image in which a signal is transmitted. Further, the emission line pattern projected by this composite image, the subject and its surroundings are each clear, and have a sharpness sufficiently recognized by the user. By displaying such a composite image, the user can know more clearly from where or from what position the signal is transmitted.
FIG. 337 is a diagram showing another example of the photographing operation of the receiver in the present embodiment.
The receiver 8000 includes a camera Ca1 and a camera Ca2. In such a receiver 8000, the camera Ca1 takes a normal picture and the camera Ca2 takes a visible light picture. As a result, the camera Ca1 acquires the normal captured image as described above, and the camera Ca2 acquires the visible light communication image as described above. Then, the receiver 8000 generates the above-mentioned composite image and displays it on the display by synthesizing the normally captured image and the visible light communication image.
FIG. 338A is a diagram showing another example of the photographing operation of the receiver in the present embodiment.
In the receiver 8000 having two cameras, the camera Ca1 switches the shooting mode between normal shooting, visible light communication, normal shooting, and so on. On the other hand, the camera Ca2 continues normal shooting. Then, when normal shooting is being performed by the cameras Ca1 and Ca2 at the same time, the receiver 8000 receives from the normal shot images acquired by those cameras by using stereo vision (principle of triangulation). Estimate the distance from the machine 8000 to the subject (hereinafter referred to as the subject distance). By using the subject distance estimated in this way, the receiver 8000 can superimpose the emission line pattern of the visible light communication image on an appropriate position of the normally captured image. That is, an appropriate composite image can be generated.
FIG. 338B is a diagram showing another example of the photographing operation of the receiver in the present embodiment.
The receiver 8000 includes, for example, three cameras (camera Ca1, camera Ca2 and camera Ca3). In such a receiver 8000, two cameras (camera Ca2 and camera Ca3) continuously perform normal shooting, and the remaining one camera (camera Ca1) continuously performs visible light communication. This makes it possible to estimate the subject distance based on the normally captured images obtained by the two normally captured images at any timing.
FIG. 338C is a diagram showing another example of the photographing operation of the receiver in the present embodiment.
The receiver 8000 includes, for example, three cameras (camera Ca1, camera Ca2 and camera Ca3). In such a receiver 8000, each camera switches the shooting mode to normal shooting, visible light communication, normal shooting, and so on. Here, in one period, the shooting mode of each camera is switched for each period so that any two of these cameras perform normal shooting and the remaining one camera performs visible light communication. Be done. That is, the combination of cameras that normally shoots changes periodically. This makes it possible to estimate the subject distance based on the normally captured images obtained by the two normally captured images during any period.
FIG. 339A is a diagram showing an example of the camera arrangement of the receiver in the present embodiment.
If the receiver 8000 includes two cameras Ca1 and Ca2, the two cameras Ca1 and Ca2 are located at distance from each other, as shown in FIG. 339A. This makes it possible to accurately estimate the subject distance. In other words, the longer the distance between the two cameras, the more accurate the subject distance can be estimated.
FIG. 339B is a diagram showing another example of the camera arrangement of the receiver in the present embodiment.
When the receiver 8000 includes three cameras Ca1, a camera Ca2 and a camera Ca3, the two cameras Ca1 and Ca2 for normal shooting are arranged apart from each other as shown in FIG. 339B. Further, the camera Ca3 for visible light communication is arranged between the camera Ca1 and the camera Ca2, for example. This makes it possible to accurately estimate the subject distance. In other words, by using the two farthest cameras for normal shooting, the subject distance can be estimated accurately.
FIG. 340 is a diagram showing an example of the display operation of the receiver in the present embodiment.
As described above, the receiver 8000 switches the shooting mode as visible light communication, normal shooting, visible light communication, and so on. Here, the receiver 8000 starts the application program when the visible light communication is first performed. Then, the receiver 8000 estimates its position based on the signal received by the visible light communication, as shown in the above embodiments 1 to 13. Next, when the receiver 8000 performs normal shooting, AR (Augmented Reality) information is displayed on the normal shooting image acquired by the normal shooting. This AR information is acquired based on the position estimated as described above. In addition, the receiver 8000 estimates the movement and change of direction of the receiver 8000 based on the detection result by the 9-axis sensor and the motion detection of the normally captured image, and matches the estimated movement and change of direction. And move the display position of AR information. As a result, the AR information can be made to follow the subject image of the normally captured image.
Further, when the receiver 8000 switches the shooting mode from normal shooting to visible light communication, the receiver 8000 superimposes AR information on the latest normal shooting image acquired at the time of the immediately preceding normal shooting during the visible light communication. Then, the receiver 8000 displays a normally captured image on which AR information is superimposed. In addition, the receiver 8000 estimates the movement and change of direction of the receiver 8000 based on the detection result by the 9-axis sensor, as in the case of normal shooting, and AR according to the estimated movement and change of direction. Move information and normally captured images. As a result, even during visible light communication, AR information can be made to follow the subject image of the normal shooting image in accordance with the movement of the receiver 8000 or the like, as in the case of normal shooting. In addition, the normal image can be enlarged or reduced according to the movement of the receiver 8000 or the like.
FIG. 341 is a diagram showing an example of the display operation of the receiver in the present embodiment.
For example, the receiver 8000 may display the composite image in which the emission line pattern is projected, as shown in FIG. 341 (a). Further, as shown in FIG. 341 (b), the receiver 8000 normally captures a signal explicit object which is an image having a predetermined color for notifying that a signal is being transmitted, instead of the emission line pattern. A composite image may be generated by superimposing on the image, and the composite image may be displayed.
Further, in the receiver 8000, as shown in (c) of FIG. 341, the place where the signal is transmitted is usually indicated by a dotted frame and an identifier (for example, ID: 101, ID: 102, etc.). The captured image may be displayed as a composite image. Further, as shown in FIG. 341 (d), the receiver 8000 is a signal identification image having a predetermined color for notifying that a specific type of signal is being transmitted instead of the emission line pattern. A composite image may be generated by superimposing an object on a normally captured image, and the composite image may be displayed. In this case, the color of the signal identification object depends on the type of signal output from the transmitter. For example, if the signal output from the transmitter is position information, the red signal identification object is superimposed, and if the signal output from the transmitter is a coupon, the green signal identification object is superimposed. It is superimposed.
FIG. 342 is a diagram showing an example of the operation of the receiver in the present embodiment.
For example, when the receiver 8000 receives a signal by visible light communication, the receiver 8000 may display a normally captured image and output a sound for notifying the user that the transmitter has been found. In this case, the receiver 8000 may differ in the type of sound output, the number of outputs, or the output time depending on the number of transmitters found, the type of signal received, or the type of information specified by the signal. You may let me.
FIG. 343 is a diagram showing another example of the operation of the receiver in the present embodiment.
For example, when the user touches the emission line pattern projected on the composite image, the receiver 8000 generates an information notification image based on the signal transmitted from the subject corresponding to the touched emission line pattern, and notifies the information. Display the image. This information notification image shows, for example, a coupon or a place of a store. The emission line pattern may be a signal explicit object, a signal identification object, a dotted line frame, or the like shown in FIG. 341. The same applies to the emission line pattern described below.
FIG. 344 is a diagram showing another example of the operation of the receiver in the present embodiment.
For example, when the user touches the emission line pattern projected on the composite image, the receiver 8000 generates an information notification image based on the signal transmitted from the subject corresponding to the touched emission line pattern, and notifies the information. Display the image. This information notification image shows, for example, the current location of the receiver 8000 by a map or the like.
FIG. 345 is a diagram showing another example of the operation of the receiver in the present embodiment.
For example, receiver 8000 receives signals from two streetlights that are subjects configured as transmitters. Then, the receiver 8000 estimates its own current location based on these signals, as in the above-described first to thirteenth embodiments. Then, the receiver 8000 displays the normally captured image and superimposes and displays the information notification image (image showing the latitude, longitude, etc.) indicating the estimation result on the normally captured image. The receiver 8000 may display the auxiliary information notification image superimposed on the normally captured image. This auxiliary information notification image recommends to the user, for example, an operation for calibrating a 9-axis sensor (particularly, a geomagnetic sensor), that is, an operation for canceling drift. By performing such an operation, the above-mentioned current location is estimated with high accuracy.
Further, when the displayed information notification image is touched by the user, the receiver 8000 may display a map showing the estimated position instead of the normally captured image.
FIG. 346 is a diagram showing another example of the operation of the receiver in the present embodiment.
For example, when the user swipes on the receiver 8000 displaying the composite image, the receiver 8000 has a dotted frame and an identifier similar to the normally captured image shown in FIG. 341 (c). Display an image and a list of information to follow the swipe operation. This list shows the information identified by the signal transmitted from the location (transmitter) indicated by each identifier. Further, the swipe may be, for example, an operation of moving a finger from the outside to the inside of the right side of the display in the receiver 8000. The swipe may be an operation of moving the finger from the upper side, the lower side, or the left side of the display.
Further, when the information included in the list is tapped by the user, the receiver 8000 may display an information notification image (for example, an image showing a coupon) showing the information in more detail.
FIG. 347 is a diagram showing another example of the operation of the receiver in the present embodiment.
For example, when the user swipes the receiver 8000 on which the composite image is displayed, the receiver 8000 superimposes and displays the information notification image on the composite image so as to follow the swipe operation. This information notification image shows the subject distance together with an arrow in an easy-to-understand manner to the user. Further, the swipe may be, for example, an operation of moving a finger from the outside to the inside of the lower side of the display in the receiver 8000. The swipe may be an operation of moving the finger from the left side of the display, from the upper side, or from the right side to the inside.
FIG. 348 is a diagram showing another example of the operation of the receiver in the present embodiment.
For example, the receiver 8000 shoots a transmitter, which is a signage indicating a plurality of stores, as a subject, and displays a normally shot image acquired by the shooting. Here, when the user taps the image of the signage of one store included in the subject projected in the normally shot image, the receiver 8000 generates an information notification image based on the signal transmitted from the signage of that store. Then, the information notification image 8001 is displayed. The information notification image 8001 is an image showing, for example, the availability of seats in a store.
FIG. 349 is a diagram showing an example of the operation of the receiver, the transmitter, and the server in the present embodiment.
First, the transmitter 8012 configured as a television transmits a signal to the receiver 8011 by changing the luminance. This signal contains, for example, information to encourage the user to purchase content related to the program being watched. When the receiver 8011 receives the signal by visible light communication, the receiver 8011 displays an information notification image prompting the user to purchase the content based on the signal. When the user performs an operation to purchase the content, the receiver 8011 is inserted into the receiver 8011 as a SIM (Subscriber Identity). Module) Sends at least one of the information contained on the card, user ID, terminal ID, credit card information, billing information, password, and transmitter ID to server 8013. The server 8013 manages the user ID and payment information in association with each user. Then, the server 8013 identifies the user ID based on the information transmitted from the receiver 8011, and confirms the payment information associated with the user ID. Based on this confirmation, the server 8013 determines whether or not to allow the user to purchase the content. Then, when the server 8013 determines that it is permitted, it transmits the permission information to the receiver 8011. When the receiver 8011 receives the permission information, the receiver 8011 transmits the permission information to the transmitter 8012. Upon receiving the permission information, the transmitter 8012 acquires and plays the content via, for example, a network.
Further, the transmitter 8012 may transmit information including the ID of the transmitter 8012 to the receiver 8011 by changing the brightness. In this case, the receiver 8011 transmits the information to the server 8013. When the server 8013 acquires the information, it can determine that, for example, a television program is being watched by the transmitter 8012, and can conduct an audience rating survey of the television program.
Further, the receiver 8011 includes the content operated by the user (voting, etc.) in the above information and transmits the content to the server 8013, so that the server 8013 can reflect the content in the television program. That is, it is possible to realize a viewer participation type program. Further, when the receiver 8011 accepts the writing by the user, the contents of the writing are included in the above information and transmitted to the server 8013, so that the server 8013 sends the writing to the TV program or the bulletin board on the network. It can be reflected in such as.
Further, when the transmitter 8012 transmits the above-mentioned information, the server 8013 can charge for viewing a television program by pay broadcasting or an on-demand program. In addition, the server 8013 may display an advertisement on the receiver 8011, display the detailed information of the TV program displayed on the transmitter 8012, and display the URL of the site showing the detailed information. Can be done. Further, the server 8013 obtains the number of times the advertisement is displayed by the receiver 8011 or the amount of the product purchased by the advertisement, and charges the advertiser according to the number of times or the amount. be able to. Billing based on such an amount can be performed without the user who sees the advertisement purchasing the product immediately. Further, when the server 8013 acquires information indicating the manufacturer of the transmitter 8012 from the transmitter 8012 via the receiver 8011, the server 8013 provides a service to the manufacturer indicated by the information (for example, a reward for selling the above-mentioned product). Payment) can be made.
FIG. 350 is a diagram showing another example of the operation of the receiver in the present embodiment.
For example, the user points the camera of receiver 8021 at a plurality of transmitters 8020a-8020d configured as lighting. At this time, the receiver 8021 is moved so that each of the transmitters 8020a to 8020d is photographed as a subject in order. Then, the receiver 8021 receives a signal from each of the transmitters 8020a to 8020d by performing visible light communication while being operated. These signals contain information indicating the location of the transmitter. The receiver 8021 is based on the position indicated by the signal from the received transmitters 8020a to 8020d, the detection result of the 9-axis sensor provided in the receiver 8021, and the motion of the image obtained by shooting. , Estimate the position of receiver 8021 using the principle of triangulation. In this case, by moving the receiver 8021, the drift of the 9-axis sensor (particularly, the geomagnetic sensor) is eliminated, so that the position can be estimated with higher accuracy.
FIG. 351 is a diagram showing another example of the operation of the receiver in the present embodiment.
The receiver 8030 is configured, for example, as a head-mounted display with a camera. When the start button is pressed, the receiver 8030 starts shooting in the visible light communication mode, that is, visible light communication. Then, when the signal is received by the visible light communication, the receiver 8030 notifies the user of the information corresponding to the received signal. This notification is performed, for example, by outputting sound from a speaker provided in the receiver 8030, or by displaying an image. In visible light communication, the receiver 8030 receives a voice input instructing the start or a signal instructing the start by wireless communication other than when the start button is pressed. It may be started as soon as it is done. In addition, visible light communication is started when the range of change in the value obtained by the 9-axis sensor provided in the receiver 8030 exceeds a predetermined range, or when a bright line pattern appears even a little in the normally captured image. You may.
FIG. 352 is a diagram showing an example of initial setting of the receiver in the present embodiment.
The receiver 8030 displays the alignment image 8031 at the time of initial setting. The alignment image 8031 is for aligning the position pointed to by the user in the image obtained by the image taken by the camera of the receiver 8030 with the image displayed by the receiver 8030. When the user aligns the fingertip with the position of the circle indicated by the alignment image 8031, the receiver 8030 associates the position of the fingertip with the position of the circle and performs alignment. That is, the position pointed to by the user is calibrated.
FIG. 353 is a diagram showing another example of the operation of the receiver in the present embodiment.
When the receiver 8030 identifies a location where a signal is transmitted by visible light communication, the receiver 8030 displays a composite image 8034 in which a bright line pattern is projected at that location. The user performs an operation such as tapping or double tapping on the bright line pattern. Upon receiving this operation, the receiver 8030 identifies the emission line pattern targeted for the operation, and displays the information notification image 8032 based on the signal transmitted from the location corresponding to the emission line pattern.
FIG. 354 is a diagram showing another example of the operation of the receiver in the present embodiment.
The receiver 8030 displays the composite image 8034 as described above. Here, the user performs an operation of moving the fingertip so as to surround the bright line pattern in the composite image 8034. Upon receiving this operation, the receiver 8030 identifies the emission line pattern targeted for the operation, and displays the information notification image 8032 based on the signal transmitted from the location corresponding to the emission line pattern.
FIG. 355 is a diagram showing another example of the operation of the receiver in the present embodiment.
The receiver 8030 displays the composite image 8034 as described above. Here, the user performs an operation of applying the fingertip to the bright line pattern in the composite image 8034 for a predetermined time or longer. Upon receiving this operation, the receiver 8030 identifies the emission line pattern targeted for the operation, and displays the information notification image 8032 based on the signal transmitted from the location corresponding to the emission line pattern.
FIG. 356 is a diagram showing another example of the operation of the receiver in the present embodiment.
The receiver 8030 displays the composite image 8034 as described above. Here, the user swipes to move the fingertip toward the bright line pattern of the composite image 8034. Upon receiving this operation, the receiver 8030 identifies the emission line pattern targeted for the operation, and displays the information notification image 8032 based on the signal transmitted from the location corresponding to the emission line pattern.
FIG. 357 is a diagram showing another example of the operation of the receiver in the present embodiment.
The receiver 8030 displays the composite image 8034 as described above. Here, the user performs an operation of continuing the state in which the line of sight is directed to the bright line pattern of the composite image 8034 for a predetermined time or longer. Alternatively, the user performs an operation of blinking a predetermined number of times with the line of sight directed to the bright line pattern. Upon receiving this operation, the receiver 8030 identifies the emission line pattern targeted for the operation, and displays the information notification image 8032 based on the signal transmitted from the location corresponding to the emission line pattern.
FIG. 358 is a diagram showing another example of the operation of the receiver in the present embodiment.
The receiver 8030 displays the composite image 8034 and the arrows associated with each of the emission line patterns in the composite image 8034, as described above. These arrows point in different directions for each emission line pattern. Here, the user performs an operation of moving the head along any of the arrows. When the receiver 8030 accepts this operation based on the detection result of the 9-axis sensor, the receiver 8030 identifies the bright line pattern associated with the arrow corresponding to the operation, that is, the arrow pointing in the direction in which the head is moved. Then, the receiver 8030 displays the information notification image 8032 based on the signal transmitted from the portion corresponding to the emission line pattern.
FIG. 359A is a diagram showing a pen used for operating the receiver in the present embodiment.
The pen 8033 includes a transmitter 8033a, a button 8033b, and a button 8033c that transmit a signal by changing the brightness. When the button 8033b is pressed, the transmitter 8033a transmits a predetermined first signal, and when the button 8033c is pressed, the transmitter 8033a is a predetermined first signal different from the first signal. Send the signal of 2.
FIG. 359B is a diagram showing the operation of the receiver using the pen in the present embodiment.
The pen 8033 is used as a substitute for the user's fingertips described above and is used like a stylus pen. Further, by properly using the button 8033b and the button 8033c, the pen 8033 can be used as a normal pen or like an eraser.
FIG. 360 is a diagram showing an example of the appearance of the receiver according to the present embodiment.
The receiver 8030 includes a first touch sensor 8030a and a second touch sensor 8030b. These touch sensors are mounted on the frame of the receiver 8030. For example, when the user touches the first touch sensor 8030a with a fingertip and moves it, the receiver 8030 moves the pointer according to the movement of the fingertip on the image displayed to the user. Further, when the user touches the second touch sensor 8030b, the receiver 8030 performs a process of selecting an object to which the pointer is applied on the image to be displayed to the user.
FIG. 361 is a diagram showing another example of the appearance of the receiver in the present embodiment.
The receiver 8030 is equipped with a touch sensor 8030c. The touch sensor 8030c is attached to the frame of the receiver 8030. For example, when the user touches the touch sensor 8030c with a fingertip and moves it, the receiver 8030 moves the pointer according to the movement of the fingertip on the image displayed to the user. Further, when the user presses the touch sensor 8030c, the receiver 8030 performs a process of selecting an object to which the pointer is applied on the image to be displayed to the user. That is, the touch sensor 8030c is configured as a so-called clickable touch sensor.
FIG. 362 is a diagram showing another example of the operation of the receiver in the present embodiment.
The receiver 8030 displays the composite image 8034 and displays the pointer 8035 in the composite image 8034 in the same manner as described above. When the receiver 8030 is equipped with the first touch sensor 8030a and the second touch sensor 8030b, the user moves the pointer by moving the first touch sensor 8030a with a fingertip, and the emission line pattern is displayed. Point the pointer at an object that is. Then, the user touches the second touch sensor 8030b to cause the receiver 8030 to select the emission line pattern. When the receiver 8030 selects the emission line pattern, the receiver 8030 displays the information notification image 8032 based on the signal transmitted from the part of the emission line pattern.
When the receiver 8030 is equipped with the touch sensor 8030c, the user moves the pointer by touching the touch sensor 8030c with a fingertip and moves the pointer, and the pointer is placed on the object having a bright line pattern. Then, the user presses the touch sensor 8030c to cause the receiver 8030 to select the emission line pattern. When the receiver 8030 selects the emission line pattern, the receiver 8030 displays the information notification image 8032 based on the signal transmitted from the part of the emission line pattern.
FIG. 363A is a diagram showing another example of the operation of the receiver in this embodiment.
The receiver 8030 displays the gesture confirmation image 8036 based on the signal obtained by performing the visible light communication. This gesture confirmation image 8036 urges the user to make a predetermined gesture, for example, in order to provide the service to the user.
FIG. 363B is a diagram showing an application example using a receiver in this embodiment.
User 8038 wears receiver 8030 and is in a store, for example. Here, the receiver 8030 displays the above-mentioned gesture confirmation image 8036 to the user 8038. The user 8038 makes a predetermined gesture according to the gesture confirmation image 8036. Here, the clerk 8039 in the store is wearing the receiver 8037. The receiver 8037 is configured as a head-mounted display equipped with a camera, and may have the same configuration as the receiver 8030. The receiver 8037 also displays the gesture confirmation image 8036 based on the signal obtained by performing visible light communication. The clerk 8039 determines whether or not the predetermined gesture indicated by the displayed gesture confirmation image 8036 matches the gesture performed by the user 8038. When the clerk 8039 determines that they match, the clerk 8039 provides the user 8038 with a service associated with the gesture confirmation image 8036.
FIG. 364A is a diagram showing another example of the operation of the receiver in this embodiment.
The receiver 8030 displays the gesture confirmation image 8040 based on the signal obtained by performing the visible light communication. This gesture confirmation image 8040 prompts the user to make a predetermined gesture, for example, in order to allow wireless communication.
FIG. 364B is a diagram showing an application example using the receiver in this embodiment.
User 8038 is equipped with receiver 8030. Here, the receiver 8030 displays the above-mentioned gesture confirmation image 8040 to the user 8038. The user 8038 makes a predetermined gesture according to the gesture confirmation image 8040. Here, the person 8041 around the user 8038 is wearing the receiver 8037. The receiver 8037 is configured as a head-mounted display equipped with a camera, and may have the same configuration as the receiver 8030. The receiver 8037 acquires authentication information such as a password included in the gesture by photographing a predetermined gesture performed by the user 8038. Then, when the receiver 8037 determines that the authentication information matches the predetermined information, the receiver 8037 establishes a wireless connection with the receiver 8030. After this establishment, the receiver 8030 and the receiver 8037 can communicate wirelessly with each other.
FIG. 365A is a diagram showing an example of the operation of the transmitter in the present embodiment.
The transmitter alternately transmits the signal 1 and the signal 2 at a predetermined cycle, for example. The transmission of the signal 1 and the transmission of the signal 2 are performed by changing the luminance such as blinking of visible light, respectively. Further, the pattern of the luminance change for transmitting the signal 1 and the pattern of the luminance change for transmitting the signal 2 are different from each other.
FIG. 365B is a diagram showing another example of the operation of the transmitter in the present embodiment.
The transmitter may intermittently transmit the signal 1 and the signal 2 with a buffering time without transmitting the signal 1 and the signal 2 continuously as described above. Here, the transmitter does not change the luminance during the buffering time. Alternatively, the transmitter may transmit a signal indicating that the buffer time is a buffer time by a luminance change, or may perform a luminance change different from the luminance change for transmitting each of the signal 1 and the signal 2. .. As a result, the receiver can properly receive the signal 1 and the signal 2 without interfering with each other.
FIG. 366 is a diagram showing another example of the operation of the transmitter in the present embodiment.
The transmitter repeatedly transmits a signal sequence of a structural unit consisting of a preamble, a block 1, a block 2, a block 3, and a check signal by changing the luminance. Here, block 1 has a preamble, an address 1, a data 1, and a check signal. Block 2 and block 3 are configured in the same way as block 1. In addition, specific information can be obtained by using the data contained in each of block 1, block 2, and block 3.
That is, in the signal sequence as described above, one data or information is stored in three blocks. Therefore, as shown in the above embodiments 1 to 13, the receiver that requires a blanking period for shooting cannot receive all the data of block 1, block 2, and block 3 from one signal string. However, the rest of the data can be received from other signal sequences. As a result, even a receiver that requires a blanking period can appropriately acquire specific information from at least one signal string.
Further, in the signal sequence as described above, a preamble and a check signal are arranged for a set of three blocks. Therefore, a receiver capable of receiving light without requiring a blanking period, for example, a receiver equipped with an illuminance sensor, uses a preamble and a check signal arranged for the set to form one signal sequence. It can be received all at once, and specific information can be obtained in a short period of time.
FIG. 367 is a diagram showing another example of the operation of the transmitter in the present embodiment.
As described above, when the transmitter repeatedly transmits a signal sequence of a structural unit including block 1, block 2, and block 3, the arrangement of the blocks included in the signal sequence is changed for each signal sequence. May be good. For example, each block is arranged in the order of block 1, block 2, and block 3 in the first signal string, and each block is arranged in the order of block 3, block 1, and block 2 in the next signal string. This makes it possible to avoid acquiring only the same block by a receiver that requires a periodic blanking period.
FIG. 368 is a diagram showing an example of a communication mode between a plurality of transmitters and a receiver in the present embodiment.
The receiver 8050 may receive a signal (visible light) transmitted from the transmitter 8051a and the transmitter 8051b configured as illumination and reflected by the reflecting surface. As a result, signals from many transmitters can be collectively received. Further, in this case, the transmitter 8051a and the transmitter 8051b transmit signals having different frequencies or protocols from each other. This allows the receiver 8050 to receive signals from those transmitters without interference.
FIG. 369 is a diagram showing an example of the operation of a plurality of transmitters in the present embodiment.
One of the transmitter 8051a and the transmitter 8051b may transmit a signal so as to monitor the transmission status of the signal from the other and prevent interference with the other signal. For example, one transmitter receives a signal transmitted from the other transmitter and transmits a signal having a protocol different from that signal. Alternatively, one transmitter detects a period during which no signal is transmitted from the other transmitter, and transmits a signal during that period.
FIG. 370 is a diagram showing another example of the communication mode between the plurality of transmitters and the receiver in the present embodiment.
Transmitter 8051a and transmitter 8051b may transmit signals of the same frequency or protocol, respectively. In this case, the receiver 8050 identifies the intensity of the signals transmitted from those transmitters, i.e., the edge intensity of the emission lines contained in the image obtained by imaging. This intensity becomes weaker as the distance between the receiver 8050 and the transmitter increases. If the distances between the receiver 8050 and each of the transmitter 8051a and the transmitter 8051b are different, such a difference in distance can be used. That is, the receiver 8050 can appropriately separate and receive the signals transmitted from each of the transmitter 8051a and the transmitter 8051b depending on the specified intensity.
FIG. 371 is a diagram showing another example of the operation of the receiver in the present embodiment.
The receiver 8050 receives the signal transmitted from the transmitter 8051a and reflected by the reflecting surface. At this time, the receiver 8050 may estimate the position of the transmitter 8051a based on the intensity distribution of the luminance (difference in luminance at a plurality of positions) in the image obtained by photographing.
FIG. 372 is a diagram showing an application example of the receiver in this embodiment.
For example, the receiver 7510a configured as a smartphone captures the light source 7510b with the back camera (out-camera) 7510c, receives the signal transmitted from the light source 7510b, and acquires the position and orientation of the light source 7510b from the received signal. The receiver 7510a estimates the position and orientation of the receiver 7510a itself from the appearance of the light source 7510b in the captured image and the sensor value of the 9-axis sensor provided in the receiver 7510a. The receiver 7510a uses the front camera (face camera, in-camera) 7510f to image the user 7510e, and estimates the position and orientation of the head of the 7510e and the direction of the line of sight (position and orientation of the eyeball) by image processing. .. The receiver 7510a sends the estimation result to the server. The receiver 7510a changes its behavior (display contents and playback sound) according to the line-of-sight direction of the user 7510e. The image taken by the back camera 7510c and the image taken by the front camera 7510f may be performed simultaneously or alternately.
FIG. 373 is a diagram showing an application example of the receiver in this embodiment.
For example, the receivers 7511d and 7511i configured as a smartphone receive signals from the light sources 7511b and 7511g, estimate their own position and orientation, and estimate the line-of-sight direction of the users 7511e and 7511j in the same manner as described above. In addition, the receivers 7511d and 7511i acquire information on surrounding objects 7511a to 7511c and 7511f to 7511h from the server based on the received data. The receivers 7511d and 7511i change the display on the display as if the user sees the object on the other side through the receivers 7511d and 7511i. The receivers 7511d and 7511i display an AR (Augmented Reality) object such as the 7511k, depending on what is shown on the display. When the line of sight of the user 7511j exceeds the imaging range of the image taken by the camera, the receiver 7511i indicates that the line of sight is out of the range, such as 7511l. Alternatively, display AR objects and other information in areas outside the range. Alternatively, the images obtained by capturing the out-of-range area in the past are stitched together and displayed.
FIG. 374 is a diagram showing an application example of the receiver in this embodiment.
For example, the receiver 7512c configured as a smartphone receives the signal from the light source 7512a, estimates its position and direction, and estimates the line-of-sight direction of the user 7512d, as described above. The receiver 7512c performs processing on the object 7512b in the line-of-sight direction of the user 7512d. For example, display information about the object 7512b on the screen. When the line-of-sight direction of the user 7512h moves from the object 7512f to the receiver 7512g, the receiver 7512g determines that the user 7512h is interested in the object 7512f and continues the processing related to the object 7512f. For example, keep the information of the object 7512f displayed on the screen.
FIG. 375 is a diagram showing an application example of the transmitter in the present embodiment.
For example, the transmitter 7513a configured as lighting exceeds the upper limit brightness when the image is taken by the receiver, whether the brightness is high and the brightness is high (high) or low (low) as a transmission signal. , 7513b does not show a bright line. Therefore, as shown in 7513c, the receiver can image the emission line like the 7513e by providing a portion 7513d that diffuses or weakens the light such as a diffuser plate or a prism to reduce the brightness. ..
FIG. 376 is a diagram showing an application example of the transmitter in the present embodiment.
For example, in the transmitter 7514a configured as lighting, since the light source is not uniform, the captured image has uneven brightness like the 7514b, which induces a reception error. Therefore, as shown in 7514c, reception errors can be suppressed by providing a portion 7514d that diffuses light, such as a diffuser plate or a prism, so that the brightness becomes uniform.
FIG. 377 is a diagram showing an application example of the receiving method in the present embodiment.
In the transmitters 7515a and 7515b, the brightness of the central portion is high and the emission line does not appear in the image captured by the receiver, and the emission line appears in the peripheral portion. The receiver cannot receive the signal from the 7515d part because the emission line is interrupted, but it can receive the signal from the 7515c part. The receiver can receive signals from more emission lines than the 7515c portion by reading the emission lines along the 7515e path.
FIG. 378 is a diagram showing an application example of the transmitter in the present embodiment.
For example, the transmitters 7516a, 7516b, 7516c, and 7516d configured as illumination have high brightness like the 7513a, and are less likely to generate emission lines when imaged by a receiver. Therefore, by providing a diffuser plate / prism 7516e, a reflector 7516f, a reflector / half mirror 7516g, a reflector 7516h, and a reflector / prism 7516j, light is diffused and the part where the emission line is generated is widened. Can be done. With these transmitters, the captured image is captured in the form of a bright line around it, such as 7515a. Since the receiver estimates the distance between the receiver and the transmitter using the size of the transmitter on the captured image, the part where the light is diffused is set as the size of the light source, and the server etc. is associated with the transmission ID. By storing it in the receiver, the receiver can accurately estimate the distance to the transmitter.
FIG. 379 is a diagram showing an application example of the transmitter in the present embodiment.
For example, the transmitter 7517a configured as illumination has high brightness like the 7513a, and it is unlikely that a bright line is generated when an image is taken by a receiver. Therefore, by providing the reflector 7517b, it is possible to diffuse the light and widen the portion where the emission line is generated.
FIG. 380 is a diagram showing an application example of the transmitter according to the present embodiment.
The transmitter 7518a reflects the light from the light source at the 7518c, so that the receiver can image the emission line in a wide range. The transmitter 7518d directs the light source to the diffuser plate or prism 7518e, so that the receiver can image the emission line in a wide range.
FIG. 381 is a diagram showing another example of the operation of the receiver in the present embodiment.
The receiver displays the emission line pattern by a composite image or an intermediate image as described above. At this time, the receiver may not be able to receive the signal from the transmitter corresponding to this emission line pattern. Here, when the emission line pattern is selected by the user performing an operation (for example, tapping) on the emission line pattern, the receiver performs an optical zoom to enlarge the portion of the emission line pattern or an intermediate image. Display the image. By performing such optical zooming, the receiver can appropriately receive the signal from the transmitter corresponding to the emission line pattern. That is, even if the image obtained by imaging is too small to acquire a signal, the signal can be appropriately received by performing optical zooming. Further, even when an image having a size capable of acquiring a signal is displayed, fast reception can be performed by performing optical zooming.
(Summary of the present embodiment) The information communication method in the present embodiment is an information communication method for acquiring information from a subject, and is included in the image sensor in an image obtained by photographing the subject by an image sensor. The first exposure time setting step for setting the exposure time of the image sensor and the image sensor setting the subject whose brightness changes so that the emission line corresponding to the exposure line is generated according to the change in the brightness of the subject. By taking a picture with the said exposure time, the bright line image acquisition step of acquiring the bright line image which is an image including the bright line and the spatial position of the portion where the bright line appears are identified based on the bright line image. In the obtained embodiment, by demodulating the data specified by the image display step of displaying the display image in which the subject and the surroundings of the subject are projected and the pattern of the emission line included in the acquired emission line image. Includes an information acquisition step to acquire transmission information.
For example, a composite image or an intermediate image as shown in FIGS. 335 to 337 and 341 is displayed as a display image. Further, in the display image in which the subject and the surroundings of the subject are projected, the spatial position of the portion where the emission line appears is identified by the emission line pattern, the signal explicit object, the signal identification object, the dotted line frame, or the like. Therefore, the user can easily find the subject transmitting the signal by the change in luminance by looking at such a display image.
Further, the information communication method further includes a second exposure time setting step of setting an exposure time longer than the exposure time, and the image sensor captures the subject and the periphery of the subject with the long exposure time. By doing so, the normal image acquisition step for acquiring a normal shot image and the portion where the bright line appears in the normal shot image are specified based on the bright line image, and the signal object which is an image pointing to the portion is the normal. In the image display step, the composite image may be displayed as the display image, including a composite step of generating a composite image by superimposing the composite image on the captured image.
For example, the signal object is a bright line pattern, a signal explicit object, a signal identification object, a dotted line frame, or the like, and a composite image is displayed as a display image as shown in FIGS. 336, 337, and 341. This allows the user to more easily find the subject transmitting the signal due to the change in luminance.
Further, in the first exposure time setting step, the exposure time is set to 1/3000 second, in the emission line image acquisition step, the emission line image in which the surroundings of the subject are projected is acquired, and in the image display step, the emission line image is acquired. , The emission line image may be displayed as the display image.
For example, as shown in FIG. 335, the emission line image is acquired and displayed as an intermediate image. Therefore, it is not necessary to perform processing such as acquiring and synthesizing a normally captured image and a visible light communication image, and the processing can be simplified.
Further, the image sensor includes a first image sensor and a second image sensor, and in the normal image acquisition step, the first image sensor captures the image to acquire the normal captured image, and the emission line is captured. In the image acquisition step, the emission line image may be acquired by the second image sensor taking an image at the same time as the image taken by the first image sensor.
For example, as shown in FIG. 337, a normal captured image and a visible light communication image which is a emission line image are acquired by each camera. Therefore, as compared with the case of acquiring the normally captured image and the visible light communication image with one camera, those images can be acquired faster and the processing can be speeded up.
Further, when the portion where the emission line appears in the display image is designated by the operation by the user, the information communication method further uses the transmission information acquired from the pattern of the emission line in the designated portion. It may include an information presentation step that presents the based presentation information. For example, the operation by the user includes tapping, swiping, an operation of continuously touching the portion with a fingertip for a predetermined time or longer, an operation of continuously directing the line of sight to the portion for a predetermined time or longer, and an operation associated with the portion. The pointer displayed on the display image is applied to the part by moving a part of the user's body to the arrow, touching the part with a pen tip whose brightness changes, or touching the touch sensor. It is an operation.
For example, as shown in FIGS. 343 to 348 and 353 to 362, the presented information is displayed as an information notification image. This makes it possible to present desired information to the user.
Further, the image sensor is provided in the head-mounted display, and in the image display step, the projector mounted on the head-mounted display may display the display image.
Thereby, for example, as shown in FIGS. 351 to 358, information can be easily presented to the user.
Further, it is an information communication method for acquiring information from a subject, and a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in the brightness of the subject. As described above, the first exposure time setting step for setting the exposure time of the image sensor, and the image sensor taking an image of the subject whose brightness changes at the set exposure time, thereby including the bright line. The bright line image includes a bright line image acquisition step for acquiring a bright line image, and an information acquisition step for acquiring information by demodulating the data specified by the bright line pattern included in the acquired bright line image. In the acquisition step, by photographing a plurality of the subjects while the image sensor is being moved, the emission line image including a plurality of portions where the emission lines appear is acquired, and in the information acquisition step, for each of the portions. In addition, by demodulating the data specified by the pattern of the emission line of the portion, the positions of the plurality of the subjects are acquired, and the information communication method further obtains each of the acquired plurality of the subjects. A position estimation step for estimating the position of the image sensor based on the position and the moving state of the image sensor may be included.
Thereby, for example, as shown in FIG. 350, the position of the receiver including the image sensor can be accurately estimated by the brightness change due to the subject such as a plurality of lights.
Further, it is an information communication method for acquiring information from a subject, and a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in the brightness of the subject. As described above, the first exposure time setting step for setting the exposure time of the image sensor, and the image sensor taking an image of the subject whose brightness changes at the set exposure time, thereby including the bright line. The emission line image acquisition step for acquiring the emission line image, the information acquisition step for acquiring information by demodulating the data specified by the emission line pattern included in the acquired emission line image, and the acquired information. In the information presentation step, an image prompting a predetermined gesture may be presented as the information to the user of the image sensor.
Thereby, for example, as shown in FIGS. 363A to 364B, the user can be authenticated depending on whether or not the user performs the gesture as prompted, and the convenience can be enhanced.
Further, it is an information communication method for acquiring information from a subject, and a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in the brightness of the subject. As described above, the exposure time setting step for setting the exposure time of the image sensor and the image sensor taking a picture of the subject whose brightness changes at the set exposure time obtains a bright line image including the bright line. The image acquisition step includes an information acquisition step of acquiring information by demodulating the data specified by the emission line pattern included in the acquired emission line image, and the image acquisition step reflects the image on the reflective surface. The bright line image is acquired by photographing a plurality of the subjects, and in the information acquisition step, the bright line is converted into a bright line corresponding to each of the plurality of subjects according to the intensity of the bright line included in the bright line image. Information may be acquired by separating and demodulating the data specified by the pattern of the emission line corresponding to the subject for each subject.
Thereby, for example, as shown in FIG. 370, even when the brightness of each subject such as a plurality of lights changes, appropriate information can be acquired from each of the subjects.
Further, it is an information communication method for acquiring information from a subject, and a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in the brightness of the subject. As described above, the exposure time setting step for setting the exposure time of the image sensor and the image sensor taking a picture of the subject whose brightness changes at the set exposure time obtains a bright line image including the bright line. The image acquisition step includes an information acquisition step of acquiring information by demodulating the data specified by the emission line pattern included in the acquired emission line image, and the image acquisition step reflects the image on the reflective surface. The emission line image may be acquired by photographing the subject, and the information communication method may further include a position estimation step of estimating the position of the subject based on the brightness distribution in the emission line image.
Thereby, for example, as shown in FIG. 371, an appropriate subject position can be estimated based on the luminance distribution.
Further, it is an information communication method for transmitting a signal by a change in luminance, that is, a first determination step of determining a first pattern of a change in luminance by modulating a first signal of a transmission target, and a first determination step of the transmission target. The second determination step of determining the second pattern of luminance change by modulating the signal of 2, the luminance change according to the determined first pattern, and the determined first. It may include a transmission step of transmitting the first and second signals by alternately performing the luminance change according to the pattern 2.
Thereby, for example, as shown in FIG. 365A, the first signal and the second signal can be transmitted without delay.
Further, in the transmission step, when the luminance change is switched between the luminance change according to the first pattern and the luminance change according to the second pattern, the luminance change may be switched after a buffering time.
Thereby, for example, as shown in FIG. 365B, interference between the first signal and the second signal can be suppressed.
Further, it is an information communication method for transmitting a signal by a change in brightness, in which a determination step of determining a pattern of change in brightness by modulating a signal to be transmitted and a light emitting body change the brightness according to the determined pattern. The signal comprises a transmission step of transmitting the signal to be transmitted, the signal is composed of a plurality of large blocks, and each of the plurality of large blocks is a preamble for the first data and the first data. And a check signal for the first data, the first data is composed of a plurality of small blocks, the small blocks are a second data, a preamble for the second data, and the first. It may include a check signal for the data of 2.
Thereby, for example, as shown in FIG. 366, the data can be appropriately acquired by either the receiver that requires the blanking period or the receiver that does not require the blanking period.
Further, it is an information communication method in which a signal is transmitted by a change in brightness, and a determination step of determining a pattern of change in brightness by modulating a signal to be transmitted by each of a plurality of transmitters, and a determination step for each transmitter. A light emitter provided in a transmitter includes a transmission step of transmitting a signal to be transmitted by changing the brightness according to the determined pattern, and the transmission step transmits signals having different frequencies or protocols from each other. You may.
Thereby, for example, as shown in FIG. 368, interference of signals from a plurality of transmitters can be suppressed.
Further, it is an information communication method for transmitting a signal by changing the brightness, and a determination step of determining a pattern of the change in brightness by modulating a signal to be transmitted by each of a plurality of transmitters, and a determination step for each transmitter. A light emitting body provided in a transmitter includes a transmission step of transmitting a signal to be transmitted by changing the brightness according to the determined pattern, and in the transmission step, one of the plurality of transmitters is used. One transmitter may receive a signal transmitted from the other transmitter and transmit another signal in a manner that does not interfere with the received signal.
Thereby, for example, as shown in FIG. 369, interference of signals from a plurality of transmitters can be suppressed.
(Embodiment 15) In the present embodiment, each application example using a receiver such as a smartphone in the above embodiments 1 to 14 and a transmitter that transmits information as a blinking pattern such as an LED or an organic EL is provided. explain.
FIG. 382 is a flowchart showing an example of the operation of the receiver in the fifteenth embodiment.
First, the receiver receives the signal with the illuminance sensor (8101). Next, the receiver acquires information such as location information from the server based on the received signal (8102). Next, the receiver activates an image sensor capable of capturing the light receiving direction of the illuminance sensor (8103). The receiver then receives some or all of the signal on the image sensor and checks if some or all of it is the same signal received by the illuminance sensor (8104). Next, the receiver estimates the position of the receiver from the position of the transmitter in the captured image (captured image), the information from the 9-axis sensor provided in the receiver, and the position information of the transmitter. (8105). In this way, the receiver activates the illuminance sensor with low power consumption, and activates the image sensor when the signal is received by the illuminance sensor. Then, the receiver performs position estimation using the image pickup by the image sensor. This makes it possible to accurately estimate the position of the receiver while suppressing power consumption.
FIG. 383 is a flowchart showing another example of the operation of the receiver in the fifteenth embodiment.
The receiver recognizes the periodic change in luminance from the sensor value of the illuminance sensor (8111). Next, the receiver activates an image sensor capable of capturing the light receiving direction of the illuminance sensor and receives the signal (8112). That is, as described above, the receiver activates the illuminance sensor with low power consumption, and activates the image sensor when the periodic change in luminance is received by the illuminance sensor. Then, the receiver receives an accurate signal by the image pickup by the image sensor. As a result, it is possible to receive an accurate signal while suppressing power consumption.
FIG. 384A is a block diagram showing an example of the transmitter according to the fifteenth embodiment.
The transmitter 8115 includes a power supply unit 8115a, a signal control unit 8115b, a light emitting unit 8115c, and a light emitting unit 8115d. The power supply unit 8115a supplies power to the signal control unit 8115b. The signal control unit 8115b distributes the electric power supplied from the power supply unit 8115a to the light emitting unit 8115c and the light emitting unit 8115d, and controls the change in the brightness of the light emitting unit 8115c and the light emitting unit 8115d.
FIG. 384B is a block diagram showing another example of the transmitter according to the fifteenth embodiment.
The transmitter 8116 includes a power supply unit 8116a, a signal control unit 8116b, a light emitting unit 8116c, and a light emitting unit 8116d. The power supply unit 8116a supplies electric power to the light emitting unit 8116c and the light emitting unit 8116d. Here, the signal control unit 8116b controls the brightness change of the light emitting unit 8116c and the light emitting unit 8116d by controlling the electric power supplied from the power supply unit 8116a. In this way, the power supply unit 8116a that supplies electric power to each of the light emitting unit 8116c and the light emitting unit 8116d is controlled by the signal control unit 8116b, so that the power usage efficiency can be improved.
FIG. 385 is a diagram showing a configuration example of a system including a plurality of transmitters according to the fifteenth embodiment.
The system includes a centralized control unit 8118, a transmitter 8117 and a transmitter 8120. The centralized control unit 8118 controls the transmission of signals due to the respective luminance changes of the transmitter 8117 and the transmitter 8120. For example, the centralized control unit 8118 causes the transmitter 8117 and the transmitter 8120 to transmit the same signal at the same timing, or causes only one of the transmitters to transmit a signal unique to the transmitter.
The transmitter 8120 includes two transmission units 8121 and 8122, a signal change unit 8123, a signal storage unit 8124, a synchronization signal input unit 8125, a synchronization control unit 8126, and a light receiving unit 8127.
The two transmitter units 8121 and 8122 have the same configuration as the transmitter 8115 shown in FIG. 384A, respectively, and transmit a signal by changing the luminance. Specifically, the transmission unit 8121 includes a power supply unit 8121a, a signal control unit 8121b, a light emitting unit 8121c, and a light emitting unit 8121d. The transmission unit 8122 includes a power supply unit 8122a, a signal control unit 8122b, a light emitting unit 8122c, and a light emitting unit 8122d.
The signal changing unit 8123 modulates the signal to be transmitted into a signal showing a pattern of luminance change. The signal storage unit 8124 stores a signal indicating the pattern of the luminance change. The signal control unit 8121b of the transmission unit 121 reads out the signal stored in the signal storage unit 8124, and changes the brightness of the light emitting unit 8121c and the light emitting unit 8121d according to the signal.
The synchronization signal input unit 8125 acquires a synchronization signal in response to control by the centralized control unit 8118. When the synchronization signal is acquired, the synchronization control unit 8126 synchronizes the luminance change between the transmission unit 8121 and the transmission unit 8122. That is, the synchronization control unit 8126 synchronizes the luminance change between the transmission unit 8121 and the transmission unit 8122 by controlling the signal control unit 8121b and the signal control unit 8122b. Here, the light receiving unit 8127 detects light emission from the transmission unit 8121 and the transmission unit 8122. The synchronization control unit 8126 performs feedback control to the signal control unit 8121b and the signal control unit 8122b according to the light detected by the light receiving unit 8127.
FIG. 386 is a block diagram showing another example of the transmitter according to the fifteenth embodiment.
The transmitter 8130 includes a transmission unit 8131 that transmits a signal by changing the luminance, and a non-transmission unit 8132 that emits light without transmitting the signal.
The transmitter unit 8131 has the same configuration as the transmitter 8115 shown in FIG. 384A, and includes a power supply unit 8131a, a signal control unit 8131b, and a light emitting unit 8131c to 8131f. Further, the non-transmission unit 8132 includes a power supply unit 8132a and a light emitting unit 8132c to 8132f, and does not include a signal control unit. That is, when there are a plurality of units including a power supply and synchronous control of luminance change cannot be performed between those units, only one of the units is provided with a signal control unit as shown in FIG. 386. Change the brightness of only one unit.
Here, in such a transmitter 8130, the light emitting units 8131c to 8131f of the transmitter unit 8131 are continuously arranged in a row. That is, any of the light emitting units 8132c to 8132f of the non-transmission unit 8132 is not mixed with the set of the light emitting units 8131c to 8131f. As a result, the size of the light emitter whose luminance changes becomes large, so that the receiver can easily receive the signal transmitted by the luminance change.
FIG. 387A is a diagram showing an example of the transmitter according to the fifteenth embodiment.
The transmitter 8134 is configured as, for example, a signage, and includes three light emitting units (light emitting regions) 8134a to 8134c. The light from these light emitting units 8134a to 8134c does not interfere with each other. Here, when it is possible to transmit a signal by changing the brightness of only one of the light emitting units 8134a to 8134c, as shown in FIG. 387A (a), the light emitting unit 8134b in the center is used. It is desirable to change the brightness. If two of the light emitting units 8134a to 8134c can be changed in brightness, as shown in FIG. 387A (b), the light emitting unit 8134b in the center and the light emitting unit 8134a or the light emitting unit at the edges are emitted. It is desirable to change the brightness of part 8134c. By changing the brightness of the light emitting unit at such a position, the receiver can appropriately receive the signal transmitted by the change in brightness.
FIG. 387B is a diagram showing an example of the transmitter according to the fifteenth embodiment.
The transmitter 8135 is configured as, for example, a signage, and includes three light emitting units 8135a to 8135c. It should be noted that the light from the light emitting parts adjacent to each other among these light emitting parts 8135a to 8135c interfere with each other. Here, when it is possible to transmit a signal by changing the luminance of only one of the light emitting units 8135a to 8135c, as shown in FIG. 387B (a), the light emitting is arranged at the end. It is desirable to change the brightness of the unit 8135a or the light emitting unit 8135c. As a result, it is possible to prevent the change in luminance for transmitting the signal from being interfered with the light from another light emitting unit. If two of the light emitting units 8135a to 8135c can be changed in brightness, as shown in FIG. 387B (b), the light emitting unit 8135b in the center and the light emitting unit 8135a or the light emitting unit at the edges are emitted. It is desirable to change the brightness of part 8135c. By changing the luminance of the light emitting unit at such a position, the area of the luminance change becomes large, so that the receiver can appropriately receive the signal transmitted by the luminance change.
FIG. 387C is a diagram showing an example of the transmitter according to the fifteenth embodiment.
When the transmitter 8134 can change the brightness of two of the three light emitting units 8134a to 8134c, the transmitter 8134 may change the brightness of the light emitting parts 8134a and the light emitting part 8134c at both ends as shown in FIG. 378C. good. In this case, it is possible to widen the imaging range in which the portion whose brightness changes is reflected in the imaging by the receiver.
FIG. 388A is a diagram showing an example of the transmitter according to the fifteenth embodiment.
The transmitter 8137 is configured as, for example, a signage, and transmits a signal by changing the brightness of the character portion A Shop and the light emitting unit 8137a. The light emitting unit 8137a is formed, for example, in a horizontally long rectangular shape and uniformly changes in brightness. By uniformly changing the brightness of the light emitting unit 8137a, the receiver can appropriately receive the signal transmitted by the change in brightness.
FIG. 388B is a diagram showing an example of the transmitter according to the fifteenth embodiment.
The transmitter 8138 is configured as, for example, a signage, and transmits a signal by changing the brightness of the character portion A Shop and the light emitting unit 8138a. The light emitting portion 8138a is formed in a frame shape along the edge of the signage, for example, and the brightness changes uniformly. That is, the light emitting portion 8138a is formed so that the length of the continuous projection portion is maximized when the light emitting portion is projected onto an arbitrary straight line. By uniformly changing the brightness of the light emitting unit 8138a, the receiver can more appropriately receive the signal transmitted by the change in brightness.
FIG. 389 is a diagram showing an example of processing operations of the receiver, the transmitter, and the server in the fifteenth embodiment.
For example, the receiver 8142 configured as a smartphone acquires the position information indicating its own position and transmits the position information to the server 8141. The receiver 8142 acquires the position information when, for example, GPS is used or another signal is received. Server 8141 sends a list of IDs associated with the location indicated by its location information to receiver 8142. The ID list contains the ID and the information associated with the ID for each ID such as "abcd".
The receiver 8142 receives a signal from the transmitter 8143 configured as, for example, a lighting device. At this time, the receiver 8142 may receive only a part of the ID (for example, b) as the above signal. In this case, the receiver 8142 searches the ID list for an ID including a part of the ID. If no unique ID is found, receiver 8142 further receives a signal from transmitter 8143 containing other parts of that ID. This causes receiver 8142 to acquire more of its ID (eg, "bc"). Then, the receiver 8142 searches the ID list again for an ID including a part of the ID (for example, "bc"). By performing such a search, the receiver 8142 can specify all of the IDs even if only a part of the IDs can be acquired. When the receiver 8142 receives a signal from the transmitter 8143, the receiver 8142 receives not only a part of the ID but also a check part such as CRC (Cyclic Redundancy Check).
FIG. 390 is a diagram showing an example of processing operations of the receiver, the transmitter, and the server in the fifteenth embodiment.
For example, the receiver 8152 configured as a smartphone acquires position information indicating its own position. The receiver 8152 acquires the position information when, for example, GPS is used or another signal is received. Further, the receiver 8152 receives a signal from the transmitter 8153 configured as, for example, a lighting device. At this time, the signal contains only a part of the ID (for example, "b") among the IDs. Here, the receiver 8152 transmits the position information and a part of the ID to the server 8151.
The server 8151 searches for an ID including a part of the ID from the ID list associated with the position indicated by the position information. If no unique ID is found, server 8151 notifies receiver 8152 that the ID identification has failed.
The receiver 8152 then receives a signal from the transmitter 8153 that includes other parts of its ID. This causes the receiver 8152 to acquire more of its ID (eg, "be"). Then, the receiver 8152 transmits a part of the ID (for example, "be") and the location information to the server 8151.
The server 8151 searches for an ID including a part of the ID from the ID list associated with the position indicated by the position information. When the unique ID is found, the server 8151 notifies the receiver 8152 that the ID (for example, "abef") has been identified, and sends the information associated with the ID to the receiver 8152.
FIG. 391 is a diagram showing an example of processing operations of the receiver, the transmitter, and the server in the fifteenth embodiment.
The receiver 8152 may transmit not only a part of the IDs but all of them to the server 8151 together with the location information. At this time, if the ID in the complete state (for example, "wxyz") is not included in the ID list, the server 8151 notifies the receiver 8152 of the error.
FIG. 392A is an explanatory diagram for explaining synchronization of a plurality of transmitters in the fifteenth embodiment.
The transmitter 8155a and the transmitter 8155b transmit a signal by changing the luminance. Here, the transmitter 8155a changes the brightness in synchronization with the transmitter 8155b by transmitting the synchronization signal to the transmitter 8155b. Further, the transmitter 8155a and the transmitter 8155b each acquire a signal from the source and change the luminance according to the signal. Here, the time required for signal transmission from the source to the transmitter 8155a (first delay time) and the time required for signal transmission from the source to the transmitter 8155b (second delay time) may differ. be. Therefore, the round trip time of the signal between those transmitters 8155a, 8155b and the source is measured, and 1/2 of those round trip times is specified as the first or second delay time described above. The transmitter 8155a performs a luminance change synchronized with the transmitter 8155b by transmitting a synchronization signal so that the difference between the first and second delay times is cancelled.
FIG. 392B is an explanatory diagram for explaining synchronization of a plurality of transmitters in the fifteenth embodiment.
The light receiving sensor 8156 detects the light from the transmitter 8155a and the transmitter 8155b, and outputs the result as a detection signal to the transmitter 8155a and the transmitter 8155b. When the transmitter 8155a and the transmitter 8155b receive the detection signal from the light receiving sensor 8156, the transmitter 8155a and the transmitter 8155b perform a luminance change synchronized with each other or adjust the signal strength based on the detection signal.
FIG. 393 is a diagram showing an example of the operation of the transmitter and the receiver in the fifteenth embodiment.
For example, the transmitter 8165 configured as a television acquires an image and an ID (ID 1000) associated with the image from the control unit 8166. Then, the transmitter 8165 displays the image and transmits the ID (ID 1000) to the receiver 8167 by changing the brightness. The receiver 8167 receives the ID (ID 1000) and displays the information associated with the ID (ID 1000) by taking an image. Here, the control unit 8166 changes the image output to the transmitter 8165 to another image. At this time, the control unit 8166 also changes the ID output to the transmitter 8165. That is, the control unit 8166 outputs another ID (ID 1001) associated with the other image to the transmitter 8165 together with the other image. As a result, the transmitter 8165 displays other images and transmits another ID (ID 1001) to the receiver 8167 by changing the brightness. The receiver 8167 receives the other ID (ID 1001) and displays the information associated with the other ID (ID 1001) by taking an image.
FIG. 394 is a diagram showing an example of the operation of the transmitter and the receiver in the fifteenth embodiment.
The transmitter 8170 is configured as, for example, a signage, and switches and displays an image. Then, when displaying the image, the transmitter 8170 transmits the ID time information indicating the ID corresponding to the displayed image and the time when the image is displayed to the receiver 8171 by changing the luminance. For example, the transmitter 8170 displays an image showing a round figure at time t1, and sets the ID (ID: 1000) corresponding to the image and the time (TIME: t1) at which the image is displayed. Send the indicated ID time information.
Here, the transmitter 8170 transmits not only the ID time information corresponding to the currently displayed image but also at least one ID time information corresponding to the image displayed in the past. For example, the transmitter 8170 displays an image showing a square figure at time t2, and sets the ID (ID: 1001) corresponding to the image and the time (TIME: t2) at which the image is displayed. Sends the indicated ID time information. Further, at this time, the transmitter 8170 transmits ID time information indicating the ID (ID: 1000) corresponding to the image showing the round figure and the time (TIME: t1) when the image was displayed. Similarly, at time t3, the transmitter 8170 displays an image showing a triangular figure, and the ID (ID: 1002) corresponding to the image and the time (TIME: t3) at which the image is displayed. ID Time information indicating the time is sent. Further, at this time, the transmitter 8170 transmits ID time information indicating the ID (ID: 1001) corresponding to the image showing the square figure and the time (TIME: t2) when the image was displayed. That is, the transmitter 8170 transmits a plurality of ID time information at the same timing.
For example, in order to obtain information related to an image showing a square figure, the user holds the image sensor of the receiver 8171 over the transmitter 8170 at the time t2 when the image showing the square figure is displayed, and the receiver. Start imaging with 8171.
Here, even if the receiver 8171 starts imaging at time t2, the ID time information corresponding to the image cannot be acquired while the image showing the square figure is displayed on the transmitter 8170. In some cases. Even in such a case, as described above, since the ID time information corresponding to the image displayed in the past is also transmitted from the transmitter 8170, the receiver 8171 has a triangular shape at time t3. Not only the ID time information (ID: 1002, TIME: t3) corresponding to the image showing the figure but also the ID time information (ID: 1001, TIME: t2) corresponding to the image showing the square figure can be acquired. Then, the receiver 8171 selects the ID time information (ID: 1001, TIME: t2) indicating the time (t2) held over the transmitter 8170 from the ID time information, and the ID time information is used. Identify the indicated ID (ID: 1001). As a result, the receiver 8171 can obtain information about the image showing the square figure from, for example, a server, based on the specified ID (ID: 1001) at time t3.
The above time is not limited to the absolute time, but is the time between the time when the receiver 8171 is held over the transmitter 8170 and the time when the receiver 8171 acquires the ID time information (so-called relative time). It may be. In addition, the transmitter 8170 has transmitted the ID time information corresponding to the image displayed in the past together with the ID time information corresponding to the image currently displayed, but corresponds to the image scheduled to be displayed in the future. ID time information may be sent. Further, the transmitter 8170 may increase the number of past or future ID time information to be transmitted when the reception by the receiver 8171 is difficult.
Further, when the transmitter 8170 is configured as a television instead of a signage, the transmitter 8170 may transmit information indicating a channel corresponding to the displayed image instead of the ID time information. That is, when the image of the TV program being broadcast is displayed on the transmitter 8170 in real time, the display time of the image displayed on the transmitter 8170 can be uniquely specified for each channel. Therefore, the receiver 8171 determines the time when the receiver 8171 is held over the transmitter 8170, that is, the time when the receiver 8171 starts imaging, based on the image obtained by the imaging and its channel. Can be done. Then, the receiver 8171 can obtain information about the image obtained by imaging from, for example, a server, based on the channel and its time. The transmitter 8170 may transmit information indicating the display time of the displayed image instead of the ID time information. In this case, the receiver 8171 searches for a TV program including the image obtained by imaging from all the TV programs being broadcast at that time, and sets the channel of the TV program and its display time. Based on this, information related to the image can be obtained from a server or the like.
FIG. 395 is a diagram showing an example of the operation of the transmitter, receiver, and server in the fifteenth embodiment.
As shown in (a) of FIG. 395, the receiver 8176 acquires an image including a emission line by imaging the transmitter 8175, and identifies (acquires) the ID of the transmitter 8175 from the image. Further, the receiver 8176 sends the ID to the server 8177 and acquires the information associated with the ID from the server 8177.
On the other hand, as shown in FIG. 395 (b), the receiver 8176 may acquire an image including a emission line by imaging the transmitter 8175 and transmit the image as imaging data to the server 8177. Further, the receiver 8176 may perform preprocessing on the image including the emission line so as to reduce the amount of information in the image, and transmit the preprocessed image to the server 8177 as imaging data. This pre-processing is, for example, image binarization processing. When the server 8177 acquires the image pickup data, the server 8177 identifies (acquires) the ID of the transmitter 8175 from the image indicated by the image pickup data. In addition, server 8177 sends information associated with that ID to receiver 8176.
FIG. 396 is a diagram showing an example of the operation of the transmitter and the receiver in the fifteenth embodiment.
When the user is at position A, receiver 8183 locates receiver 8183 by acquiring a signal transmitted from transmitter 8181 whose luminance changes. As a result, receiver 8183 displays a point 8183b indicating its identified position with an error range 8183a at that position.
Next, when the user moves from the position A and arrives at the position B, the receiver 8183 cannot acquire the signal from the transmitter 8181. At this time, the receiver 8183 estimates its position by using a 9-axis sensor or the like provided in the receiver 8183. Then, the receiver 8183 displays the point 8183b indicating the estimated position together with the error range 8183a of the position. At this time, since the position is estimated by the 9-axis sensor, the error range 8183a is widely displayed.
Next, when the user moves from the position B and arrives at the position C, the receiver 8183 identifies the position of the receiver 8183 by acquiring the signal transmitted from the other transmitter 8182 whose brightness changes. As a result, receiver 8183 displays a point 8183b indicating its identified position with an error range 8183a at that position. Here, the receiver 8183 smoothly switches the point 8183b and the error range 8183a indicating the position estimated by using the 9-axis sensor to the position and the error range specified as described above without immediately switching and displaying them. Move them to switch. At this time, the error range 8183a becomes smaller.
FIG. 397 is a diagram showing an example of the appearance of the receiver according to the fifteenth embodiment.
The receiver 8183 is configured as, for example, a smartphone (high-performance mobile phone), and as shown in FIG. 397 (a), an image sensor 8183c, an illuminance sensor 8183d, and a display 8183e are arranged in front of the receiver 8183. ing. The image sensor 8183c acquires an image including bright lines by capturing an image of a subject whose brightness changes as described above. The illuminance sensor 8183d detects the above-mentioned change in the brightness of the subject. Therefore, the illuminance sensor 8183d can be used as a substitute for the image sensor 8183c depending on the condition or situation of the subject. The display 8183e displays an image or the like. Here, the receiver 8183 may have a function as a subject whose brightness changes. In this case, the receiver 8183 transmits the signal by changing the brightness of the display 8183e.
Further, as shown in FIG. 397 (b), an image sensor 8183f, an illuminance sensor 8183g, and a flash light emitting unit 8183h are arranged on the back surface of the receiver 8183. The image sensor 8183f is the same as the above-mentioned image sensor 8183c, and acquires an image including a bright line by capturing an image of a subject whose brightness changes as described above. The illuminance sensor 8183g is the same as the illuminance sensor 8183d described above, and detects a change in the brightness of the subject. Therefore, the illuminance sensor 8183g can be used as a substitute for the image sensor 8183f depending on the condition or situation of the subject. The flash light emitting unit 8183h emits a flash for imaging. Here, the receiver 8183 may have a function as a subject whose brightness changes, and in this case, the signal is transmitted by changing the brightness of the flash light emitting unit 8183h.
FIG. 398 is a diagram showing an example of the operation of the transmitter, receiver, and server in the fifteenth embodiment.
For example, the transmitter 8185 configured as a smartphone displays information indicating, for example, a "coupon 100 yen discount" by changing the brightness of the portion of the display 8185a excluding the barcode portion 8185b, that is, by visible light communication. Send. Further, the transmitter 8185 causes the barcode portion 8185b to display the barcode without changing the brightness of the barcode portion 8185b. This barcode indicates the same information as the information transmitted by the visible light communication described above. Further, the transmitter 8185 displays characters or pictures indicating information transmitted by visible light communication, for example, the characters "coupon 100 yen discount" on the portion of the display 8185a excluding the barcode portion 8185b. By displaying such characters or pictures, the user of the transmitter 8185 can easily grasp what kind of information is being transmitted.
By taking an image, the receiver 8186 acquires the information transmitted by the visible light communication and the information indicated by the barcode, and transmits the information to the server 8187. The server 8187 determines whether or not the information is matched or related, and when it is determined that the information is matched or related, the server 8187 executes the process according to the information. Alternatively, the server 8187 transmits the determination result to the receiver 8186, and causes the receiver 8186 to execute the process according to the information.
The transmitter 8185 may transmit a part of the information indicated by the barcode by visible light communication. Further, the barcode may indicate the URL of the server 8187. Further, the transmitter 8185 may acquire the ID as a receiver and transmit the ID to the server 8187 to acquire the information associated with the ID. The information associated with this ID is the same as the information transmitted by the above-mentioned visible light communication or the information indicated by the barcode. Further, the server 8187 may transmit an ID associated with the information (visible light communication information or barcode information) transmitted from the transmitter 8185 via the receiver 8186 to the transmitter 8185.
FIG. 399 is a diagram showing an example of the operation of the transmitter and the receiver in the fifteenth embodiment.
For example, the transmitter 8185 configured as a smartphone transmits a signal by changing the brightness of the display 8185a. The receiver 8188 includes a cone-shaped container 8188b having a light-shielding property and an illuminance sensor 8188a. The illuminance sensor 8188a is housed inside the container 8188b and is arranged near the tip of the container 8188b. When a signal is transmitted from the transmitter 8185 by visible light communication, the opening (bottom) of the container 8188b in the receiver 8188 is directed toward the display 8185a. As a result, light other than the light from the display 8185a does not enter the container 8188b, so that the illuminance sensor 8188a of the receiver 8188 properly receives the light from the display 8185a without being affected by the light that becomes noise. It can receive light. As a result, the receiver 8188 can properly receive the signal from the transmitter 8185.
FIG. 400 is a diagram showing an example of the operation of the transmitter and the receiver in the fifteenth embodiment.
The transmitter 8190 is configured as, for example, a sign tower of a bus stop, and transmits operation information indicating a bus operation status or the like to the receiver 8183 by changing the brightness. For example, operation information indicating the destination of the bus, the time when the destination bus arrives at the bus stop, the current location of the bus, and the like is transmitted to the receiver 8183. When the receiver 8183 receives the operation information, the receiver 8183 displays the content indicated by the operation information on the display.
Here, for example, when buses of different destinations stop at the bus stop, the transmitter 8190 transmits operation information regarding the buses of those destinations. When the receiver 8183 receives the operation information, the receiver 8183 selects the operation information of the destination bus frequently used by the user from the operation information, and displays the content indicated by the operation information on the display. do. Specifically, the receiver 8183 identifies the destination of the bus used by the user by using, for example, GPS, and records the history of the destination. By referring to this history, the receiver 8183 selects the operation information of the frequently used destination bus by the user. Alternatively, the receiver 8183 may display the content indicated by the operation information selected by the user's operation from the operation information on the display. Alternatively, the receiver 8183 may preferentially display the operation information of the frequently-selected destination bus selected by the user's operation.
FIG. 401 is a diagram showing an example of the operation of the transmitter and the receiver in the fifteenth embodiment.
For example, the transmitter 8191 configured as a signage transmits information of a plurality of stores to the receiver 8183 by changing the brightness. This information is a collection of information about a plurality of stores, and is not information unique to each store. Therefore, when the receiver 8183 receives the information by imaging, it can display information about not only one store but also a plurality of stores. Here, the receiver 8183 selects information about a store (for example, "B shop") included in the imaging range from the information about the plurality of stores, and displays the selected information. When displaying the information, the receiver 8183 translates the language for expressing the information into a language registered in advance, and displays the information in the translated language. Further, the transmitter 8191 may display a message prompting the image sensor (camera) of the receiver 8183 to take an image by characters or the like. Specifically, a message (for example, "Get information with the camera") is displayed on the transmitter 8191 to inform that the information can be provided by starting the dedicated application program and taking an image with the camera. NS.
FIG. 402 is a diagram showing an example of the operation of the transmitter and the receiver in the fifteenth embodiment.
For example, receiver 8183 captures a subject including a plurality of people 8197 and a streetlight 8195. The streetlight 8195 is equipped with a transmitter 8195a that transmits information by changing the brightness. By this imaging, the receiver 8183 acquires an image in which the image of the transmitter 8195a appears as the above-mentioned emission line pattern. Further, the receiver 8183 acquires the AR object 8196a associated with the ID indicated by the emission line pattern from, for example, a server. Then, the receiver 8183 superimposes the AR object 8196a on the normal shooting image 8196 obtained by the normal shooting, and displays the normal shooting image 8196 on which the AR object 8196a is superposed.
FIG. 403A is a diagram showing an example of the configuration of information transmitted by the transmitter in the fifteenth embodiment.
For example, the information transmitted by the transmitter includes a preamble unit, a fixed-length data unit, and a check unit. The receiver checks the data unit using the check unit, and normally receives the information consisting of each of these parts. Here, the receiver receives the preamble unit and the data unit, and when the check unit cannot be received, the check using the check unit is omitted. Even if such a check is omitted, the receiver can normally receive the information consisting of each part thereof.
FIG. 403B is a diagram showing another example of the configuration of the information transmitted by the transmitter in the fifteenth embodiment.
For example, the information transmitted by the transmitter includes a preamble unit, a check unit, and a variable length data unit. The next information transmitted by the transmitter also consists of a preamble section, a check section, and a variable length data section. Here, when the receiver receives the preamble section during reception and further receives the next preamble section, the receiver has one meaning of the information from the previous preamble section to immediately before the next preamble section. Recognize as information. Further, the receiver may specify the end of the data unit to be received next to the check unit by using the check unit. In this case, the receiver properly receives one meaningful piece of information transmitted immediately before, even if the receiver cannot receive the next preamble section (all or part of the preamble section) described above. be able to.
FIG. 404 is a diagram showing an example of a four-value PPM modulation method using a transmitter in the fifteenth embodiment.
The transmitter modulates the signal to be transmitted into a pattern of luminance change by a 4-value PPM modulation method. At this time, the transmitter can keep the brightness of the light whose brightness changes constant regardless of the signal to be transmitted.
For example, if the brightness is kept at 75%, the transmitter will play each of the signals "00", "01", "10" and "11" to be transmitted in any of four consecutive slots. One indicates the luminance L (Low), and the remaining three modulate the luminance pattern indicating the luminance H (High). Specifically, the transmitter has a luminance change pattern (L, H, H,) in which the signal "00" to be transmitted indicates the luminance L in the first slot and the luminance H in the second to fourth slots. Modulate to H). That is, in this change in brightness, there is a rise in brightness between the first slot and the second slot. Similarly, in the transmitter, the signal "01" to be transmitted has a luminance change pattern (H,) in which the second slot indicates the luminance L and the first, third and fourth slots indicate the luminance H. Modulate to L, H, H). That is, in this change in brightness, there is a rise in brightness between the second slot and the third slot.
Also, if the brightness is kept at 50%, the transmitter will use any two of the four slots for each of the signals "00", "01", "10" and "11" to be transmitted. The brightness L (Low) is shown, and the remaining two are modulated into a luminance change pattern showing the brightness H (High). Specifically, the transmitter indicates a luminance change pattern (L,) in which the signal "00" to be transmitted indicates the luminance L in the first and fourth slots and the luminance H in the second and third slots. Modulate to H, H, L). That is, in this change in brightness, there is a rise in brightness between the first slot and the second slot. Similarly, in the transmitter, the signal "01" to be transmitted has a luminance change pattern (L,) in which the first and second slots indicate the luminance L and the third and fourth slots indicate the luminance H. Modulate to L, H, H). Alternatively, the transmitter has a luminance change pattern (H, L, H) in which the signal "01" to be transmitted indicates the luminance L in the second and fourth slots and the luminance H in the first and third slots. , L). That is, in these luminance changes, there is a luminance rise between the second slot and the third slot.
Also, if the brightness is kept at 25%, the transmitter will use any three of the four slots for each of the signals "00", "01", "10" and "11" to be transmitted. The luminance L (Low) is indicated, and the remaining one is modulated into a luminance change pattern indicating the luminance H (High). Specifically, the transmitter displays the signal "00" to be transmitted, the first, third and fourth slots indicate the luminance L, and the second slot indicates the luminance H, which is a pattern of luminance change (L, Modulate to H, L, L). That is, in this change in brightness, there is a rise in brightness between the first slot and the second slot. Similarly, in the transmitter, the signal "01" to be transmitted has a luminance change pattern (L,) in which the first, second and fourth slots indicate the luminance L and the third slot indicates the luminance H. Modulate to L, H, L). That is, in this change in brightness, there is a rise in brightness between the second slot and the third slot.
The transmitter can suppress flicker by the 4-value PPM modulation method as described above, and can easily adjust the brightness step by step. Further, the receiver can appropriately demodulate the pattern of the luminance change by specifying the position of the rising edge of the luminance. The receiver ignores the presence or absence of a rise in luminance at the boundary between the slot group consisting of four slots and the next slot group without using it for demodulating the pattern of luminance change.
FIG. 405 is a diagram showing an example of the PPM modulation method by the transmitter in the fifteenth embodiment.
Similar to the 4-value PPM modulation method shown in FIG. 404, the transmitter modulates the signal to be transmitted in a pattern of luminance change, but performs PPM modulation without switching the luminance between L and H for each slot. May be good. That is, the transmitter performs PPM modulation by switching the rising position of the luminance in the time width (hereinafter referred to as a unit time width) of the four consecutive slots shown in FIG. 404 according to the signal to be transmitted. For example, as shown in FIG. 405, the transmitter modulates the signal "00" to be transmitted into a pattern of luminance change such that the luminance rises at a position of 25% of the unit time width. Similarly, as shown in FIG. 405, the transmitter modulates the signal "01" to be transmitted into a pattern of luminance change such that the luminance rises at a position of 50% of the unit time width.
In addition, when the brightness is kept at 75%, the transmitter shows the luminance L at the position of 0 to 25% in the above-mentioned unit time width of the signal "00" to be transmitted, and the position of 25 to 100%. Modulates to a luminance change pattern indicating luminance H. Here, when the brightness is kept at 99%, the transmitter indicates the brightness L of the signal "00" to be transmitted at the position of 24 to 25% in the above-mentioned unit time width, and is 0 to 24%. Modulates to a pattern of luminance change indicating luminance H at position and 25-100% position. Similarly, when the brightness is kept at 1%, the transmitter sets the brightness L of the signal "00" to be transmitted at the 0 to 25% position and the 26 to 100% position in the above-mentioned unit time width. It is shown and modulated into a pattern of luminance change indicating luminance H at a position of 25 to 26%.
In this way, the brightness can be continuously adjusted by switching the brightness between L and H at an arbitrary position in the unit time width without switching the brightness between L and H for each slot.
FIG. 406 is a diagram showing an example of the PPM modulation method in the transmitter according to the fifteenth embodiment.
The transmitter performs modulation in the same manner as the PPM modulation method shown in FIG. 405, but regardless of the signal to be transmitted, the signal always shows the luminance H at the beginning of the unit time width. Moreover, at the end of the unit time width, it is always modulated to a luminance change pattern indicating the luminance L. As a result, a rise in luminance occurs at the boundary between the unit time width and the next unit time width, so that the receiver can appropriately specify the boundary. Therefore, the receiver and the transmitter can correct the clock deviation.
FIG. 407A is a diagram showing an example of a luminance change pattern corresponding to a header (preamble portion) in the fifteenth embodiment.
For example, when the transmitter transmits the header (preamble portion) shown in FIGS. 403A and 403B, the luminance changes according to the pattern shown in FIG. 407A. That is, when the header is composed of 7 slots, the transmitter changes its luminance according to the pattern indicated by L, H, L, H, L, H, H. Further, when the header is composed of 8 slots, the transmitter changes the brightness according to the pattern indicated by H, L, H, L, H, L, H, H. Since these patterns can be distinguished from the pattern of luminance change shown in FIG. 404, it is possible to clearly inform the receiver that the signal indicated by these patterns is a header.
FIG. 407B is a diagram showing an example of the luminance change pattern in the fifteenth embodiment.
As shown in FIG. 404, in the 4-value PPM modulation method, when the signal "01" to be transmitted contained in the data section is modulated while the brightness is kept at 50%, the transmitter transmits the signal. Is modulated into one of the two patterns. That is, it is modulated to the first pattern represented by L, L, H, H or the second pattern represented by H, L, H, L.
Here, when the pattern of the luminance change corresponding to the header is the pattern shown in FIG. 407A, the transmitter indicates the above-mentioned signal "01" to be transmitted by L, L, H, and H. It is desirable to modulate to the pattern of. For example, the transmission target signal "11,01,11" included in the above-mentioned data unit is "H, H, L, L, L, L, H, H" when the first pattern is used. , H, H, L, L "pattern. On the other hand, when the second pattern is used, the transmission target signal "11,01,11" included in the above data unit is "H, H, L, L, H, L, H, L". , H, H, L, L "pattern. In this case, the pattern "H, H, L, L, H, L, H, L, H, H, L, L" has the same pattern as the header pattern composed of 7 slots shown in FIG. 407A. Is included. Therefore, in order to clarify the distinction between the header and the data unit, it is desirable to modulate the above-mentioned signal "01" to be transmitted to the first pattern.
FIG. 408A is a diagram showing an example of a pattern of luminance change in the fifteenth embodiment.
As shown in FIG. 404, in the 4-value PPM modulation method, when the signal "11" to be transmitted is modulated, the transmitter transmits the signal "H, H, H" so that the luminance does not rise. , L pattern, H, H, L, L pattern, or H, L, L, L pattern. However, as shown in FIG. 408A, the transmitter sets the signal "11" to be transmitted to the "H, H, H, H" pattern or "L, L, L, L" in order to adjust the brightness. May be modulated into the pattern of.
FIG. 408B is a diagram showing an example of the luminance change pattern in the fifteenth embodiment.
As shown in FIG. 404, in the 4-value PPM modulation method, when the brightness is maintained at 75% and the signal "11,00" to be transmitted is modulated, the transmitter transmits the signal "H, H". , H, L, L, H, H, H "pattern. However, when the luminance L is to be continuously generated, the luminance other than the last luminance L of the continuous luminance L may be changed to H so that the luminance L is not continuous. In this case, the transmitter modulates the signal "11,00" into a pattern of "H, H, H, H, L, H, H, H".
As a result, since the luminance L is not continuous, the load on the transmitter can be suppressed. Further, the capacity of the capacitor provided in the transmitter can be reduced, and the volume of the control circuit can be reduced. Further, since the load of the light source of the transmitter is small, it is possible to easily make the light source. In addition, the power efficiency of the transmitter can be increased. Further, since it is guaranteed that the luminance L is not continuous, the receiver can easily demodulate the pattern of the luminance change.
(Summary of the present embodiment) The information communication method in the present embodiment is an information communication method that transmits a signal by a change in luminance, and determines a pattern of change in luminance by modulating the signal to be transmitted. A step and a transmission step in which the light emitter transmits the signal of the transmission target by changing the brightness according to the determined pattern, and the pattern of the brightness change is any one in a predetermined time width. A pattern in which one of two different luminance values appears at a position, and in the determination step, for each of the different signals to be transmitted, the rising position or the falling position of the luminance in the time width. The pattern of the luminance change is determined so that the luminance change positions are different from each other and the integrated value of the luminance of the light emitter in the time width is the same value according to the preset luminance. ..
For example, as shown in FIG. 405, the rising position (luminance change position) of the luminance is different from each other for each of the signals "00", "01", "10", and "11" which are different from each other to be transmitted, and , The brightness change so that the integral value of the brightness of the illuminant in the predetermined time width (unit time width) becomes the same value according to the predetermined brightness (for example, 99% or 1%). Pattern is determined. As a result, the brightness of the light emitter can be kept constant for each of the signals to be transmitted, flicker can be suppressed, and the receiver that images the light emitter is based on the brightness change position. , The pattern of the brightness change can be appropriately demodulated. Further, since the luminance change pattern is a pattern in which one of two different luminance values (luminance H (High) or luminance L (Low)) appears at any position in the unit time width, light emission is emitted. The brightness of the body can be changed continuously.
Further, the information communication method further includes an image display step of sequentially switching and displaying each of the plurality of images, and in the determination step, each time an image is displayed in the image display step, the displayed image is displayed. By modulating the identification information corresponding to the above as the signal to be transmitted, the pattern of the change in brightness with respect to the identification information is determined, and in the transmission step, the image is displayed every time the image is displayed in the image display step. The identification information may be transmitted by changing the brightness of the light emitter according to the pattern of the brightness change determined for the identification information corresponding to the image.
As a result, as shown in FIG. 393, for example, each time an image is displayed, the identification information corresponding to the displayed image is transmitted, so that the user receives the information to the receiver based on the displayed image. The identification information to be used can be easily selected.
Further, in the transmission step, each time an image is displayed in the image display step, the light emitter becomes brighter according to a pattern of luminance change determined for identification information corresponding to an image displayed in the past. The identification information may be transmitted by changing.
As a result, as shown in FIG. 394, for example, even if the receiver cannot receive the identification signal transmitted before the switching because the displayed image is switched, the identification information corresponding to the currently displayed image is obtained. At the same time, since the identification information corresponding to the image displayed in the past is also transmitted, the identification information transmitted before the switching can be appropriately received again by the receiver.
Also, in the determining step, each time the image is displayed by the image display step, the identification information corresponding to the image displayed, before modulation and a time at which the SL image is displayed as a signal of the transmission object By doing so, the identification information and the pattern of the brightness change with respect to the time are determined, and in the transmission step, each time the image is displayed in the image display step, the identification information and the time corresponding to the displayed image are set. The identification information and the time are transmitted by changing the brightness of the light emitter according to the determined pattern of the brightness change, and further determined for the identification information and the time corresponding to the image displayed in the past. The identification information and the time may be transmitted by changing the brightness of the light emitter according to the pattern of the change in brightness.
As a result, as shown in FIG. 394, for example, each time an image is displayed, a plurality of ID time information (information consisting of identification information and time) is transmitted, so that the receiver receives a plurality of ID times. From the information, the identification signal that was transmitted in the past and could not be received can be easily selected based on the time included in each of the ID time information.
Further, each of the light emitters has a plurality of regions that emit light, and the light in the regions adjacent to each other among the plurality of regions interferes with each other, and only one of the plurality of regions is determined. When the luminance changes according to the luminance change pattern, in the transmission step, only the region arranged at the end of the plurality of regions may change the luminance according to the determined luminance change pattern.
As a result, for example, as shown in (a) of FIG. 387B, since only the region (light emitting portion) arranged at the end changes the brightness, the brightness changes only in the region arranged other than the end. It is possible to suppress the influence of the light from the region on the change in brightness. As a result, the receiver can appropriately capture the pattern of the brightness change by shooting.
Further, when only two of the plurality of regions change the luminance according to the determined pattern of the luminance change, in the transmission step, the region arranged at the end of the plurality of regions and the region described above. The area adjacent to the area arranged at the end may change the luminance according to the determined pattern of the luminance change.
As a result, for example, as shown in (b) of FIG. 387, the area arranged at the end (light emitting part) and the area adjacent to the area arranged at the end (light emitting part) change in brightness, so that each other. Compared with the case where the distant region changes the brightness, the area of the range where the brightness changes continuously in space can be kept wider. As a result, the receiver can appropriately capture the pattern of the brightness change by shooting.
The information communication method in the present embodiment is an information communication method for acquiring information from a subject, and includes a position information transmission step for transmitting position information indicating the position of an image sensor used for photographing the subject, and the position information. Corresponds to the exposure line included in the image sensor in the list receiving step of receiving the ID list including a plurality of identification information associated with the position indicated by the image sensor and the image obtained by shooting the subject by the image sensor. The exposure time setting step for setting the exposure time of the image sensor and the image sensor taking a picture of the subject whose brightness changes at the set exposure time so that the emission line is generated according to the change in the brightness of the subject. By doing so, an image acquisition step of acquiring an emission line image including the emission line, an information acquisition step of acquiring information by demodulating the data specified by the emission line pattern included in the acquired emission line image, and the like. It includes a search step for searching the ID list for the acquired identification information including the information.
As a result, for example, as shown in FIG. 389, since the ID list has been received in advance, even if the acquired information "bc" is only a part of the identification information, the appropriate identification information "" is based on the ID list. abcd "can be identified.
If the acquired identification information including the information is not uniquely specified in the search step, new information is acquired by repeating the image acquisition step and the information acquisition step, and the information communication is performed. The method may further include a re-search step of searching the ID list for identification information including the acquired information and the new information.
As a result, as shown in FIG. 389, for example, even if the acquired information "b" is only a part of the identification information and the identification information is not uniquely identified by the information alone, the new information "b" is obtained. Since "c" is acquired, appropriate identification information "abcd" can be identified based on the new information and the ID list.
The information communication method in the present embodiment is an information communication method for acquiring information from a subject, and the image obtained by photographing the subject by an image sensor has a bright line corresponding to an exposure line included in the image sensor. An exposure time setting step of setting the exposure time of the image sensor so as to occur in response to a change in the brightness of the subject, and the image sensor taking a picture of the subject whose brightness changes at the set exposure time. An image acquisition step for acquiring an emission line image including the emission line, an information acquisition step for acquiring identification information by demodulating the data specified by the emission line pattern included in the acquired emission line image, and acquisition. The identification acquired in an ID list including a plurality of identification information associated with the transmission step of transmitting the identification information and the position information indicating the position of the image sensor and the position indicated by the position information. If there is no information, it includes an error receiving step to receive error notification information for notifying an error.
As a result, as shown in FIG. 391, for example, when the acquired identification information is not in the ID list, the error notification information is received, so that the user of the receiver who has received the error notification information has acquired the error notification information. It is easy to understand that the information associated with the identification information cannot be obtained.
(Embodiment 16) In the present embodiment, each application example using a receiver such as a smartphone in the above embodiments 1 to 15 and a transmitter that transmits information as a blinking pattern such as an LED or an organic EL is provided. explain.
FIG. 409 is a diagram showing an example of the operation of the transmitter configured as a television in the present embodiment.
The transmitter configured as a television alternately displays an image for the left eye (left eye image) and an image for the right eye (right eye image) as display images. Here, the user of the television (transmitter) sees the left eye image only with the left eye and the right eye image only with the right eye by wearing 3D glasses. As a result, the user can visually recognize a three-dimensional (3D) image having a depth corresponding to the parallax between the left eye image and the right eye image.
Further, the backlight of this television is arranged on the back side of the panel on which the displayed image is projected, and normally lights up and signals are transmitted alternately. Normal lighting is to constantly illuminate the above-mentioned display image brightly. The signal transmission is to illuminate the display image with the light whose brightness is changing while transmitting a signal corresponding to the pattern of the brightness change to the outside of the television through the above-mentioned panel by changing the brightness. ..
Here, one left-eye image and one right-eye image are displayed during the period of normal lighting, and the left-eye image is displayed during the period of signal transmission.
The 3D glasses open and close the field of view of the user's left and right eyes, respectively. Specifically, when the 3D glass is set to 3D display mode, it closes the field of view of the left eye and the right eye, respectively, during the period when the backlight is signaling. Furthermore, the 3D glass opens the field of view of the left eye and closes the field of view of the right eye when the image of the left eye is displayed while the backlight is normally lit, and conversely, the image of the right eye is displayed. When it is, the left eye's field of vision is closed and the right eye's field of vision is opened. As a result, the user wearing the 3D glass set to the 3D display mode can see the left eye image only with the left eye and the right eye image only with the right eye while the backlight is normally lit. For viewing, a three-dimensional (3D) image can be visually recognized. Further, since the field of view of both eyes of the user is closed during the period when the signal is transmitted, it is possible to suppress the user from feeling the flicker due to the change in luminance.
Also, when the 3D glass is set to a flicker-resistant pattern in the 2D display mode, it closes the field of view of the left eye and the right eye during the period when the backlight is transmitting a signal. In addition, the 3D glass opens the left and right eye fields of view when the left eye image is displayed and left when the right eye image is displayed during the period when the backlight is normally lit. Close the visual field of the eye and the right eye respectively. As a result, the field of view of both eyes of the user is closed during the period when the signal is transmitted, and only the left eye image is reflected in both eyes of the user during the period when the normal lighting is performed. It is possible to visually recognize a two-dimensional (2D) image in which flicker due to a change in brightness is suppressed.
Also, the 3D glasses, when set to a bright pattern in 2D display mode, open the field of view of the left and right eyes while the backlight is signaling. As a result, a left-eye image whose brightness changes is displayed in both eyes of the user. In addition, the 3D glass opens the left and right eye fields of view when the left eye image is displayed and left when the right eye image is displayed during the period when the backlight is normally lit. Close the visual field of the eye and the right eye respectively. As a result, even during the period when the signal is transmitted, the field of view of both eyes is open, so that the user can visually recognize a bright two-dimensional (2D) image as compared with the case of the above-mentioned flicker-resistant pattern. .. On the contrary, since the 3D glasses keep the view of both eyes open for a long period of time, the backlight can be dimmed and the power consumption of the television can be suppressed.
FIG. 410 is a diagram showing an example of the operation of the transmitter and the receiver in the present embodiment.
For example, a transmitter configured as a television repeats normal lighting and signal transmission as described above. In addition, normal lighting is continued for 1/30 seconds or more.
The receiver takes an image of the transmitter during the period of normal lighting. At this time, the receiver performs the above-mentioned visible light photographing. As a result, the receiver obtains a lighting image 8201a showing a range (normal lighting range) 8201b in which the television is normally lit. Next, the receiver takes an image of the transmitter during the period when the signal transmission is being performed. Also at this time, the receiver performs the above-mentioned visible light photographing. As a result, the receiver obtains the emission line image 8202a on which the above-mentioned emission line pattern is projected.
Here, the receiver is located at the same position as the above-mentioned normal lighting range 8201b in the lighting image 8201a, and a range having the same size and shape as the normal lighting range 8201b is specified from the emission line image 8202a as a signal transmission range 8202b. Then, the receiver determines that the signal transmission range 8202b in the emission line image 8202a has an emission line pattern, and demodulates the data specified by the emission line pattern in the signal transmission range 8202b. As a result, since the signal transmission range 8202b is specified from the emission line image 8202a based on the normal lighting range 8201b, the receiver can accurately specify the range in which the signal is transmitted. For example, even if there is a dark part at the end of the signal transmission range 8202b, the receiver does not mistakenly think that the signal is not transmitted from that part, and the signal indicating that there is no emission line from that part. Can be properly recognized that is being transmitted.
In addition, the receiver can measure the size of its normal lighting range 8201b and can calculate the exact distance to the television based on that size.
FIG. 411 is a diagram showing an example of the operation of the transmitter, receiver, and server in the present embodiment.
First, the transmitter 8203 configured as a television acquires broadcast data broadcast from a broadcasting station. This broadcast data includes additional information as well as data such as broadcast programs. For example, the additional information is an ID for identifying the broadcast program or a scene in the broadcast program, or content related to the broadcast program or scene. When the transmitter 8203 acquires the broadcast data, it outputs data such as a broadcast program included in the broadcast data as images and audio, and transmits an ID or content included in the broadcast data due to a change in brightness.
The receiver 8204 receives the ID or content by imaging the transmitter 8203 (taking visible light). Then, the receiver 8204 transmits the ID to the server 8205.
When the server 8205 receives the ID from the receiver 8204, the server 8205 sends the information associated with the ID to the receiver 8204. When the receiver 8204 receives the related information from the server 8205, the receiver 8204 displays the content indicated by the related information. When the receiver 8204 receives the content from the transmitter 8203, the receiver 8204 may display the content without transmitting the ID to the server 8205. The transmitter 8203 and the server 8205 may be integrated.
FIG. 412 is a diagram showing an example of the operation of the transmitter and the receiver in the present embodiment.
The transmitter 8207 configured as a television displays an image of broadcast content (for example, a broadcast program) and transmits information related to the image by changing the brightness. The transmitted information includes the channel of the broadcast content being displayed and the time when the image of the broadcast content is displayed. For example, at time t1, an image showing a round figure included in the broadcast content is displayed, and information including the channel "CH: 1" of the broadcast content and the time t1 in which the image showing the round figure is displayed. Is sent. Further, at time t2, an image showing a quadrangle included in the broadcast content is displayed, and information including the channel "CH: 1" of the broadcast content and the time t2 in which the image showing the quadrangle is displayed is transmitted. Will be done. Similarly, at time t3, an image showing a triangle included in the broadcast content is displayed, and information including the channel "CH: 1" of the broadcast content and the time t3 in which the image showing the triangle is displayed is displayed. Will be sent.
The receiver 8208 acquires the above information by imaging the transmitter 8207 and transmits the information to the server 8209. Hereinafter, the time included in this information is referred to as a reference time. When the server 8209 receives the information, the server 8209 determines the time around the reference time included in the information as the peripheral time. For example, the server 8209 determines the time before or after a predetermined time from the reference time as the peripheral time. In the example shown in FIG. 412, when the reference time is time t3, the server 8209 determines the peripheral times t1, t2, and t4.
Then, the server 8209 selects the related information associated with each of the channels included in the received information and the reference time and the peripheral time included in the information, and transmits the related information to the receiver 8208. .. For example, server 8209 may have relevant information associated with channel "CH: 1" and reference time t3, associated information associated with channel "CH: 1" and peripheral time t1, and channel "CH: 1". And the related information associated with the peripheral time t2 and the related information associated with the channel "CH: 1" and the peripheral time t4 are transmitted to the receiver 8208.
When the receiver 8208 receives a plurality of related information from the server 8209, the receiver 8208 selects and displays one of the related information from the related information. For example, the receiver 8208 selects the related information corresponding to the reference time from the related information. Further, when any related information is specified by the user, the receiver 8208 selects and displays the pointed out related information. Alternatively, receiver 8208 may display each of the plurality of related information received.
In this way, not only the related information associated with the information (channel and reference time) transmitted and received from the transmitter 8207, but also the related information associated with the peripheral time is transmitted from the server 8209. Therefore, even if the receiver 8208 cannot receive the information desired by the user from the transmitter 8207 due to the timing of imaging the transmitter 8207 being shifted, the receiver 8208 can display the related information associated with the information. It can be received from server 8209.
Instead of transmitting the time included in the received information to the server 8209 as the reference time, the receiver 8208 sets the time when the image sensor of the receiver 8208 is held over the transmitter 8207 to receive the information. It may be sent to the server 8209 as a reference time. Here, the holding time is the time when the holding state of the image sensor is started when the holding state of the image sensor continues to be held by the transmitter 8027.
For example, the receiver 8208 specifies the time N seconds before (for example, 5 seconds before) the time included in the received information as the time when the image sensor is held over. Then, the receiver 8208 treats the specified time as a reference time, and transmits the reference time and the channel included in the received information to the server 8209. Alternatively, the receiver 8208 transmits the latest time output from the built-in 9-axis sensor, which indicates the magnitude of movement of the receiver 8208, to the server 8209 as the above-mentioned reference time. You may.
FIG. 413 is a diagram showing an example of the operation of the transmitter in the present embodiment.
When displaying an image on a display, the transmitter 8210 configured as a television transmits a signal by changing the brightness of the display. Here, the display on which the image is displayed has a bright part and a dark part. Therefore, the transmitter 8210 changes the brightness of only the portion (bright portion) that emits light with a brightness higher than a predetermined brightness in order to display an image in the display, thereby that portion. Send a signal only from. As a result, since the brightness of the dark portion does not change, the darkness can be stabilized and a darker gradation can be expressed.
FIG. 414 is a diagram showing an example of the operation of the transmitter in the present embodiment.
The transmitter 8211 configured as a television is equipped with a motion sensor 8211a. The motion sensor 8211a detects a person within the viewing angle of the transmitter 8211 (display) or a person around the viewing angle. The periphery of the viewing angle is, for example, a range deviating from the viewing angle by X (X is a predetermined real number larger than 0)% of the viewing angle.
Here, when there is a person in front of the display, that person is within the viewing angle of the display, and the person is detected by the motion sensor 8211a. In this case, the transmitter 8211 transmits a signal by changing the brightness of the display by a normal amount of change.
On the other hand, when there is a person diagonally forward of the display, the person is detected by the motion sensor 8211a because the person is around the viewing angle of the display. In this case, the transmitter 8211 transmits the signal by changing the luminance of the display by a change amount larger than the normal change amount. For example, a user of transmitter 8211 can see the image displayed on the display brightly and clearly when he is within the viewing angle. However, when the user is around the viewing angle, it becomes difficult to see the image brightly and clearly. That is, it becomes difficult for the light from the display to reach outside the viewing angle. Therefore, as described above, when a person in the vicinity of the viewing angle is detected, the brightness of the display changes with a change amount larger than the normal change amount, so that the receiver 8212 carried by that person can be used. Even outside the viewing angle, the signal can be properly received by imaging the display whose brightness changes.
If the motion sensor 8211a does not detect a person, the transmitter 8211 stops transmitting the signal without changing the brightness of the display. As a result, the power consumption of the transmitter 8211 can be suppressed.
The viewing angle may be a predetermined angle according to the specifications of the display, or may be an arbitrarily determined angle. For example, the viewing angle may be in a range in which the intensity of light from the display is equal to or higher than a predetermined intensity, and the reception sensitivity of the receiver 8212 (sensitivity to receive a signal transmitted due to a change in brightness) is predetermined. It may be in a range having a sensitivity or higher.
FIG. 415 is a diagram showing an example of the operation of the transmitter in the present embodiment.
A transmitter configured as a television repeatedly darkens, for example, for a period of 1000 μsec, and then brightens for a period of 3000 μsec, in order to perform black insertion. At this time, the transmitter transmits (superimposes) a signal by changing the luminance during the period of 3000 μsec when it becomes bright. Here, the change in luminance is expressed by outputting high-luminance (High) light or low-luminance (Low) light in each period of, for example, 104 μsec. However, if 3000 μs is divided by 104 μs, there will be a remainder of 88 μs, and there will be a period of 3000 μs in which high or low brightness light cannot be continuously output for 104 μsec. Therefore, the transmitter in the present embodiment is characterized in that the signal is transmitted even in a period shorter than 104 μsec (88 μsec). For example, the transmitter outputs high or low luminance light to represent a signal, even in that short period of time. Alternatively, the transmitter may not transmit the signal for that short period of time. As a result, the original bright light is emitted because the signal is not superimposed on the original light in that short period, that is, the luminance is not changed. Therefore, the image on the television can be brightened.
(Supplement) When the scanning direction on the imaging side is the vertical direction (vertical direction) of the mobile terminal, if the exposure time is shortened and the image is taken, the ON / OFF of the entire LED lighting device is shown in (a) of Fig. 416. ), It is possible to image a bright line, which is a white / black pattern, in the same direction as the scanning direction. In FIG. 416 (a), since the long side direction of the vertically long LED lighting device is imaged so as to be perpendicular to the scanning direction on the imaging side (horizontal direction of the mobile terminal), the same direction as the scanning direction. In addition, it is possible to image a large number of bright lines with white and black patterns. That is, the amount of information that can be transmitted and received can be increased. On the other hand, as shown in FIG. 416 (b), when a vertically long LED lighting device is imaged so as to be parallel to the scanning direction on the image pickup side (vertical direction of the mobile terminal), a white / black pattern that can be imaged is obtained. There are fewer bright lines. That is, the amount of information that can be transmitted becomes small.
In this way, when a large number of white and black pattern emission lines can be imaged depending on the orientation of the LED lighting device with respect to the scanning direction on the imaging side (the long side direction of the vertically long LED lighting device is perpendicular to the scanning direction on the imaging side). When the image is taken so that it becomes Occurs.
In the present embodiment, a control method of a lighting device capable of capturing a large number of bright lines even when only a small number of bright lines of white / black patterns can be captured will be described.
FIG. 417 shows an example of a lighting device in which a plurality of LEDs are arranged in the vertical direction and a drive signal thereof. FIG. 417 (a) is a lighting device in which a plurality of LEDs are arranged in the vertical direction. It is assumed that each LED element corresponds to the minimum unit of the coded horizontal stripes of the visible light communication signal, and corresponds to the encoded ON / OFF signal. In this way, by generating white and black patterns, turning on / off each LED element, and illuminating, it is assumed that the scanning direction on the image pickup side and the long side direction of the vertically long LED lighting device are parallel. , It is possible to shoot white and black patterns for each LED element.
FIGS. 417 (c) and 417 (d) show an example in which a white / black pattern is generated and each LED element is turned ON / OFF for illumination. When illuminating as a white / black pattern in a lighting device, unevenness may occur in the light even for a short time. Therefore, (c) and (d) of FIG. 417 show an example of generating anti-phase patterns and illuminating them alternately. The element that was turned on in FIG. 417 (c) is turned off in FIG. 417 (d), and the element that was turned off in FIG. 417 (c) is shown in FIG. 417 (d). In d), it is ON. In this way, by illuminating the white and black patterns alternately with the positive phase pattern and the opposite phase pattern in sequence, the scanning direction on the imaging side and the illumination can be performed without causing unevenness in the light. It is possible to send and receive a lot of information in visible light communication without being affected by the relationship with the orientation of the device. Further, it is not limited to the case where two types of patterns, a positive phase pattern and an anti-phase pattern, are alternately generated and illuminated, but it is also conceivable to generate and illuminate three or more types of patterns. FIG. 418 shows an example of sequentially illuminating four types of patterns.
Normally, a configuration is also conceivable in which the entire LED illumination blinks ((b) in FIG. 417), a white / black pattern is generated for a predetermined time, and the LED element is illuminated. For example, it is conceivable that the transmission / reception time of a predetermined data unit blinks in the entire LED lighting, and then the white / black pattern is illuminated in a short time in the LED element unit. Here, the predetermined data unit means, for example, a data unit from the first header to the next second header. At this time, when the image is taken in the direction of (a) of FIG. 416, the signal is received from the emission line which captured the blinking of the entire LED illumination, and when it is received in the direction of (b) of FIG. Receive a signal from the light emission pattern.
The present embodiment is not limited to the LED lighting device, and may be any element as long as it can control ON / OFF in small element units like the LED element. Further, the device is not limited to the lighting device, and may be a device such as a television, a projector, or a signage.
Further, in the present embodiment, an example of illuminating with a white / black pattern has been described, but a color may be used instead of the white / black pattern. For example, of RGB, RG may be constantly lit and only B may be used to blink. Using only B rather than R or G is less likely to be recognized by humans, and it is possible to suppress flicker. As another example, instead of the white / black pattern, use a complementary color in additive mixing (red and cyan pattern, green and magenta pattern, yellow and blue pattern, etc.) to display ON / OFF. May be good. By using a complementary color in additive color mixing, it is possible to suppress flicker.
Further, in the present embodiment, the example in which the LED elements are arranged in one dimension has been described, but instead of arranging the LED elements in one dimension, they are arranged in two dimensions and displayed like a two-dimensional bar code. May be done.
(Summary of this embodiment) The information communication method in the present embodiment is an information communication method for acquiring information from a subject, and the image obtained by photographing the subject by an image sensor has a bright line corresponding to an exposure line included in the image sensor. An exposure time setting step of setting the exposure time of the image sensor so as to occur in response to a change in the brightness of the subject, and the image sensor taking a picture of the subject whose brightness changes at the set exposure time. A bright line image acquisition step for acquiring a bright line image including the bright line, a first information acquisition step for acquiring information by demodulating the data specified by the bright line pattern included in the acquired bright line image, and the like. The information acquisition step includes a second information acquisition step of transmitting the information to the server and acquiring the related information associated with the information from the server, and the emission line image acquisition step displays the broadcasted and received content. In the first information acquisition step, the channel used for broadcasting the content displayed by the subject and the reference time which is the time when the content is displayed are included. Information is acquired, and in the second information acquisition step, a plurality of related information associated with the channel and each of the reference time and the peripheral time which is at least one time around the reference time is described. Get from the server.
As a result, as shown in FIG. 412, for example, not only the related information transmitted from the transmitter which is the subject whose brightness changes and associated with the information (channel and reference time) but also the related information associated with the peripheral time is acquired. NS. Therefore, even if the information desired by the user cannot be acquired from the transmitter in the first information acquisition step due to the timing of imaging the transmitter being shifted, the information is associated with the information. Related information can be obtained from the server.
Further, the information communication method further captures a lighting image showing the lighting range in which the subject is lit by photographing the subject which is lit without changing the brightness for transmitting a signal. A range specifying step for specifying a range from the emission line image, which is at the same position as the lighting range in the lighting image and has the same size and shape as the lighting range, as a signal transmission range. In the first information acquisition step, the data specified by the pattern of the emission line included in the specified signal transmission range may be demolished.
As a result, as shown in FIG. 410, for example, the signal transmission range is specified from the emission line image based on the lighting range (normal lighting range), so that the range in which the signal is transmitted is accurately specified from the emission line image. can do. For example, even if there is a dark part at the end within the signal transmission range, a signal indicating that there is no emission line is transmitted from that part without misunderstanding that no signal is transmitted from that part. Can be properly recognized.
The information communication method in the present embodiment is an information communication method in which a signal is transmitted by a change in luminance, and a determination step of determining a pattern of change in luminance by modulating the signal to be transmitted and a display display an image. The transmission step includes a transmission step of transmitting a signal to be transmitted by changing the brightness according to the determined pattern while displaying, and the transmission step is predetermined in order to display the image of the display. Only the portion that emits light with a brightness equal to or higher than the specified brightness changes its brightness according to the pattern.
As a result, as shown in FIG. 413, for example, in the display, the portion of the display set to a brightness lower than the predetermined brightness, that is, the dark portion does not change the brightness, so that the darkness is reduced. It can be stabilized and a darker gradation can be expressed.
Further, the display is provided with a backlight, and the information communication method further displays an image for the left eye and an image for the right eye, which are illuminated by the backlight with a predetermined brightness, in order. 3 The transmission step and the three-dimensional image display step may be alternately repeated, including a three-dimensional image display step.
As a result, as shown in FIG. 409, for example, when the brightness is changed in the transmission step, the view of both eyes of the user is closed by the 3D glass, and the image for the left eye and the image for the right eye are displayed in the 3D image display step. When and are displayed in order, if only the user's eye view corresponding to those images is opened by the 3D glass, the user can visually recognize the 3D image in which the flicker due to the change in brightness is suppressed. can.
Further, the information communication method further includes a detection step in which a sensor detects a person within the viewing angle of the display or a person in the vicinity of the viewing angle, and in the transmission step, the peripheral is included. When a person within the viewing angle is detected, the signal to be transmitted is transmitted by changing the brightness by a larger amount of change than when a person within the viewing angle is detected, and the viewing angle is within the range. If neither the person inside nor the person in the vicinity is detected, the transmission of the signal due to the change in brightness may be stopped.
As a result, as shown in FIG. 414, for example, when a person in the vicinity of the viewing angle is detected, the display changes the normal amount of change (the brightness change when a person within the viewing angle is detected). The brightness changes with a change amount larger than the change amount). Therefore, the receiver carried by the person can appropriately receive the signal by imaging the display whose brightness changes even if it is outside the viewing angle.
(Embodiment 17) In the present embodiment, each application example using a receiver such as a smartphone in the above-described first to 17th embodiments and a transmitter for transmitting information as an LED blinking pattern will be described.
FIG. 419 is a diagram showing an example of the transmission signal of the 17th embodiment.
The transmission signal D is divided into data pieces Dx (for example, Dx = D1, D2, D3) of a predetermined size, and the frame check sequence FCS and header Hdr for error detection / correction calculated from each data piece are divided into each data piece. To add to. Furthermore, a frame check sequence FCS2 and header Hdr2 for error detection / correction calculated from the original data are added. The data consisting of Hdr, Dx, and FCS is configured to be received by the image sensor. Since the image sensor is suitable for receiving continuous data in a short time, Hdr, Dx, and FCS transmit continuously. Hdr2, Dx, The data consisting of FCS2 is configured to be received by the illuminance sensor. It is desirable that the Hdr and FCS received by the image sensor are short, but the Hdr2 and FCS2 received by the illuminance sensor can be a longer signal sequence. By using a long signal sequence for Hdr2, the header detection accuracy can be improved. By lengthening FCS2, it is possible to adopt a code that can detect and correct many bit errors, and it is possible to improve the performance of error detection and correction. It should be noted that the HDR2 and the FCS2 may not be transmitted, but the HDR and the FCS may be received by the illuminance sensor instead. The illuminance sensor may receive both Hdr and Hdr2, or both FCS and FCS2.
FIG. 420 is a diagram showing an example of the transmission signal of the 17th embodiment.
FCS2 has a long signal length, and if it is inserted frequently, the reception efficiency of the image sensor deteriorates. Therefore, the frequency of inserting FCS2 is reduced, and instead, the signal PoFCS2 indicating the location of FCS2 is inserted. As an example, when 4PPM with 2 bits of information per unit time is used for signal representation, 16 units of transmission time is required when CRC32 is used for FCS2, but PoFCS2 with a range of 0 to 3 requires 1 unit time. You can send it with. Since the transmission time is shorter than when only FCS2 is inserted, the efficiency of image sensor reception can be improved. The illuminance sensor receives PoFCS2 following the transmission signal D, identifies the transmission time of FCS2 from PoFCS2, and receives FCS2. Further, PoFCS2 following FCS2 is received, the transmission time of the next FCS2 is specified, and the next FCS2 is received. If the FCS2 received earlier and the FCS2 received later are the same, it is presumed that the receiver is receiving the same signal.
FIGS. 421A to 421C are diagrams showing an example of a captured image (bright line image) of the receiver in the 17th embodiment.
In the captured image shown in FIG. 421A, the number of emission lines is small because the image is small. Therefore, only a small amount of data can be received at one time from this captured image. The captured image shown in FIG. 421B is an image captured by using a zoom, and has a large number of bright lines because the image is large. Therefore, if an image is taken using a zoom, a large amount of data can be received at one time. It can also receive data from a distance and can receive signals from small transmitters. Optical zoom or Ex zoom is used as the zoom method. Optical zoom is a zoom by increasing the focal length of the lens. Ex-zoom is a zoom method for enlarging a part of an image taken by taking an image using only a part of the image sensor, not all of the image sensor, when the image is taken at a resolution lower than the capacity of the image sensor. The captured image shown in FIG. 421C is an image captured by using an electronic zoom (enlargement of the image). Although the image is large, the bright lines become thicker when magnified by electronic zoom, and the number of bright lines does not change from before zooming, so the reception characteristics are the same as before zooming.
FIGS. 422A and 422B are diagrams showing an example of a captured image (bright line image) of the receiver according to the seventeenth embodiment.
The captured image shown in FIG. 422A is an image captured by focusing on the subject, and the captured image shown in FIG. 422B is an image captured out of focus. In the captured image shown in FIG. 422B, since the image is blurred, the emission lines can be observed up to the periphery of the actual transmitter, and more emission lines can be observed. Therefore, if the image is taken out of focus, a large amount of data can be received at one time, and data can be received from a greater distance. By taking an image using the macro mode, it is possible to take an image similar to the image taken as shown in FIG. 422B.
FIGS. 423A to 423C are diagrams showing an example of a captured image (bright line image) of the receiver in the 17th embodiment.
By setting the exposure time longer than the visible light communication mode and shorter than the normal imaging mode for imaging, an image as shown in FIG. 423A can be obtained. The imaging mode in which such an image is obtained is called a emission line detection mode (intermediate mode). In the image shown in FIG. 423A, the emission line of the transmitter can be observed in the center left, and a dark normal captured image appears in the other parts. By displaying this image on the receiver, it is possible to facilitate the user to take an image by pointing the receiver at the target transmitter. In the emission line detection mode, the image is darker than in the normal imaging mode. Therefore, by imaging in the high-sensitivity mode, it is possible to capture an image having a brightness close to that in the normal imaging mode and easily visible to humans. If the sensitivity is set too high, the dark part of the bright line will become bright, so set the sensitivity to the extent that the bright line can be observed. The receiver shifts to the visible light communication mode and receives the transmission signal of the transmitter imaged in the portion designated by the means such as touching the image by the user. When the emission line (transmission signal) is found in the captured image, the receiver may automatically shift to the visible light communication mode and receive the signal.
The receiver finds the transmission signal from the emission line in the captured image and highlights that portion as shown in FIG. 423B, so that the portion where the signal is transmitted can be presented to the user in an easy-to-understand manner. The emission line may be observed not only by the transmitted signal but also by the pattern of the subject. Therefore, instead of determining the presence or absence of the transmission signal from the emission line of one image, it may be determined that there is a transmission signal when the position of the emission line changes in a plurality of images.
Since the image captured in the emission line detection mode is darker and has poor visibility than the image captured in the normal imaging mode, an image whose visibility has been improved by image processing may be displayed. The image shown in FIG. 423C is an example of an image in which edges are extracted to emphasize the boundaries of the image to be imaged.
FIG. 424 is a diagram showing an example of a captured image (bright line image) of the receiver according to the 17th embodiment. Specifically, FIG. 424 is a diagram in which a transmitter with a signal transmission cycle of 1/9600 seconds is imaged at the ratio of the exposure time shown at the bottom of the figure. With an exposure time shorter than the transmission cycle of 1/9600 seconds, the captured images are almost the same, and clear emission lines can be captured. When the exposure time becomes long, the outline of the emission line becomes blurred, but in this signal expression example, the emission line pattern can be observed and the signal can be received if the exposure time is up to about 1.5 times the transmission cycle. Further, in the example of this signal expression, the emission line can be observed if the exposure time is up to about 20 times the transmission cycle, and the exposure time in this range can be used as the exposure time in the emission line detection mode.
How long the signal can be received depends on the method of signal expression. If a signal expression rule with few emission lines and a long interval between emission lines is used, the transmission efficiency is lowered, but the signal can be received even with a longer exposure time, and the emission lines can be observed even with a longer exposure time.
FIG. 425 is a diagram showing an example of a transmission signal according to the 17th embodiment.
Since the receiver integrates multiple received data pieces and receives a series of signals, if the transmission signal is suddenly changed, the data pieces before and after the change will be mixed and the signals cannot be integrated correctly. .. Therefore, as shown in FIG. 425 (a), when the transmission signal is changed, the transmitter normally lights up as a buffer zone for a predetermined time and does not transmit the signal. When the receiver cannot receive the signal during the predetermined time T2 shorter than the predetermined time T1, the receiver discards the data pieces received so far to avoid mixing the data pieces before and after the change. can do. Alternatively, as shown in FIG. 425 (b), when the transmission signal is changed, the transmitter repeatedly transmits the signal X notifying the transmission signal. By repeatedly transmitting, the reception omission of the transmission signal change notification X is prevented. Alternatively, as shown in FIG. 425 (c), the transmitter repeatedly transmits the preamble when the transmission signal is changed. When the receiver receives the preamble in a cycle shorter than the cycle in which the preamble appears in a normal signal, the receiver discards the data pieces received so far.
FIG. 426 is a diagram showing an example of the operation of the receiver in the 17th embodiment.
The image shown in (a) of FIG. 426 is an image obtained by capturing the transmitter with just focus. The receiver can take an image as shown in FIG. 426 (b) by taking an image out of focus. When the focus is further removed, the captured image looks like the image shown in FIG. 426 (c). In (c) of FIG. 426, the emission lines of a plurality of transmitters overlap, and the receiver cannot receive the signal. Therefore, the receiver adjusts the focus so that the emission lines of the plurality of transmitters do not overlap, and takes an image. If there is only one transmitter in the imaging range, the receiver adjusts the focus so that the size of the transmitter is maximized in the captured image.
The receiver may compress the captured image in a direction parallel to the emission line, but does not compress the image in a direction perpendicular to the emission line. Alternatively, the receiver reduces the degree of compression in the vertical direction. As a result, it is possible to prevent the bright line from being blurred due to compression and the reception error from occurring.
427 and 428 are diagrams showing an example of an instruction to the user to be displayed on the screen of the receiver in the 17th embodiment.
By imaging a plurality of transmitters, the receiver can estimate the position of the receiver from the position information of each transmitter and the position, size, and angle of each transmitter in the captured image in the manner of triangulation. Therefore, when only one transmitter is imaged in a receivable form, the user changes the direction of the receiver or moves backward to image a plurality of transmitters, so that the receiver can be imaged. Instructs the imaging direction and the moving direction by displaying an image including an arrow or the like. FIG. 427 (a) is a display example of an instruction to point the receiver to the right and image the transmitter on the right side, and FIG. 427 (b) is an instruction to move backward to image the transmitter in front. This is a display example. FIG. 428 is a display example of an instruction to image another transmitter by shaking the receiver because the position of the other transmitter is unknown to the receiver. It is desirable that a plurality of transmitters are captured in one captured image, but the positional relationship of the transmitters in the plurality of images may be estimated by using image processing or the sensor value of the 9-axis sensor. The receiver may use the ID received from one transmitter, inquire the server about the position information of the transmitters in the vicinity, and instruct the user to take an image of the transmitter that is the easiest to take an image.
The receiver detects that the user is moving the receiver from the sensor value of the 9-axis sensor, and displays a screen based on the last received signal after the movement is completed and a predetermined time elapses. As a result, when the receiver is pointed at the transmitter intended by the user, the signal of another transmitter is received while the receiver is moving, and processing based on the transmission signal of the unintended transmitter is performed. You can prevent that.
The receiver may continue the reception processing while being moved, and may perform processing based on the received signal, for example, information acquisition from the server using the received signal as a key. In this case, the reception process is continued even after the process, and the process based on the last received signal is the final process.
The receiver may process the signal received a predetermined number of times or notify the user. The receiver may process the signal that was received the most times while being moved.
The receiver may be provided with a notification means for notifying the user when the signal is successfully received or when the presence of the signal is detected in the captured image. The notification means notifies by sound, vibration, display update (pop-up display, etc.), or the like. This allows the user to know the existence of the transmitter.
FIG. 429 is a diagram showing an example of the signal transmission method according to the seventeenth embodiment.
For example, a plurality of transmitters configured as displays are arranged adjacent to each other. When transmitting the same signal, the plurality of transmitters synchronize the timing of signal transmission and transmit the signal from the entire surface as shown in FIG. 429 (a). With this configuration, the receiver observes a plurality of displays as one large transmitter, so that the receiver can receive signals at higher speed and from a longer distance. When a plurality of transmitters transmit different signals, as shown in FIG. 429 (b), the plurality of transmitters transmit signals by providing a buffer zone (non-transmission area) in which the signals are not transmitted. With this configuration, the receiver can recognize that the plurality of transmitters are different transmitters across the buffer zone and receive different signals.
FIG. 430 is a diagram showing an example of the signal transmission method according to the seventeenth embodiment.
As shown in FIG. 430 (a), the liquid crystal display is provided with a backlight extinguishing period, and the state of the liquid crystal is changed while the backlight is extinguished to make the image during the state change invisible and dynamically resolve. You can enhance the feeling. As shown in FIG. 430 (b), a signal is superimposed on the liquid crystal display that performs such backlight control according to the lighting cycle of the backlight. Receiving efficiency can be improved by continuously transmitting a set of data (Hdr, Data, FCS). Further, at the beginning and the end of the lighting period of the backlight, the light emitting portion is in a bright state (Hi). This is because when the light emitting unit is in a dark state (Lo) continuously with the backlight extinguishing period, the receiver cannot determine whether Lo is transmitted as a signal or whether the light emitting unit is in a dark state due to the backlight extinguishing period.
During the backlight extinguishing period, a signal having a low average brightness may be superimposed.
By superimposing the signal, the average brightness changes as compared with the case where the signal is not superposed. Therefore, the backlight lighting period is increased or decreased, or the brightness when the backlight is lit is increased or decreased to adjust the average brightness to be equal.
FIG. 431 is a diagram showing an example of the signal transmission method according to the seventeenth embodiment.
The liquid crystal display can reduce the change in the brightness of the entire screen by controlling the backlight at different timings for each position. This is called a backlight scan. The backlight scan is usually performed so that the backlight is turned on in order from the end as shown in (a) of FIG. 431. At this time, the captured image 8802a is obtained. However, in the captured image 8802a, the portion where the emission line exists is divided, and it may not be possible to estimate that the entire screen of the display is one transmitter. Therefore, as shown in (b) of Fig. 431, all the parts that emit light (overlapping signals) when the vertical axis is displayed as the space axis in the division direction of the backlight scan and the horizontal axis is displayed as the time axis. By setting the order of the backlight scans so that they are connected, the captured image 8802b can be obtained. In the captured image 8802b, all the emission lines are connected, and it can be easily estimated that the signal is transmitted from one transmitter. In addition, since the number of emission lines that can be continuously received increases, signals can be received quickly and from a distance. Further, since the size of the transmitter can be easily estimated, the position of the receiver can be accurately estimated from the position, size, and angle of the transmitter in the captured image.
FIG. 432 is a diagram showing an example of the signal transmission method according to the seventeenth embodiment.
In a time-division backlight scan, the lighting period of the backlight is short, and light is emitted (signals are superimposed) on a graph whose vertical axis is the space axis in the division direction of the backlight scan and the horizontal axis is the time axis. ) When the parts cannot be connected, a signal is superimposed on each light emitting part according to the light emitting timing of the backlight as in the case of FIG. 430. At this time, by controlling the backlight so that the distance from the other backlight lighting portions on the graph is maximized, it is possible to prevent the emission lines of the adjacent portions from being mixed.
FIG. 433 is a diagram for explaining the use case in the seventeenth embodiment. The system in the present embodiment includes a lighting fixture 100 that performs visible light communication, a wearable device 101 having a visible light communication function, a smartphone 102, and a server 103.
An object of the present embodiment is to save a series of troubles when a user makes a purchase at a store by using visible light communication, and to shorten the time required for shopping. Conventionally, when purchasing a product at a store, it has been necessary to search the site of the store in advance to obtain coupon information. In addition, there is a problem that it takes time to search for the product that is the target of the coupon in the store.
As shown in FIG. 433, the lighting fixture 100 periodically transmits the lighting ID information of its own unit in front of a store (assuming a home appliance mass retailer as an example) by using visible light communication. Upon receiving this lighting ID information, the user's wearable device 101 transmits the lighting ID information to the smartphone 102 using short-range wireless communication. The smartphone 102 transmits user information and lighting ID information to the server 103 using a mobile line or the like. The smartphone 102 receives point information, coupon information, and the like of the store in front of the user from the server 103. The user browses the information received from the server 103 on the wearable device 101 or the smartphone 102. The user can purchase the displayed product information of the store on the spot or receive guidance to the exhibition place in the store. Hereinafter, it will be described in detail with reference to figures.
FIG. 434 is a diagram showing an information table transmitted by the smartphone 102 to the server 103. In addition to the member number, store ID information, transmission time, and location information of the store held in the smartphone 102, the user's taste information, biometric information, search history, and behavior history information stored in the smartphone 102 are stored in the smartphone 102. Will send.
FIG. 435 is a block diagram of the server 103. The transmission / reception unit 201 receives the information transmitted from the smartphone 102. The control unit 202 controls the entire system. The member information DB 203 stores the member number, the name and date of birth of the user of the member number, point information, purchase history, and the like. The store DB 204 stores information in the store, such as a store ID, product information sold at the store, store display information, and store map information. The notification information generation unit 205 generates coupon information and recommended product information according to the user's taste.
FIG. 436 is a flowchart showing the entire processing of the system. The wearable device 102 receives the illumination ID from the illumination 100 (S301). Next, the wearable device 101 transmits the illumination ID to the smartphone 102 using proximity wireless communication such as Bluetooth (registered trademark) (S302). Next, the smartphone 102 transmits the user's history information, the membership number, and the lighting ID shown in FIG. 434 to the server 103 (S303). When the server 103 receives the data, the data is first transmitted to the control unit 202 (S304). Next, the control unit 202 inquires the member number to the member DB 203 and acquires the member information (S305). Next, the control unit 202 queries the store DB 204 for the lighting ID and acquires store information (S306). The store information includes product information in stock in the store, information on products for which sales are to be promoted as a store, coupon information, in-store map information, and the like. The control unit 202 transmits the member information and the store information to the notification information generation unit (S307). The notification information generation unit 205 generates advertisement information suitable for the user from the member information and the store information, and transmits the advertisement information to the control unit 202 (S308). The control unit 202 transmits the member information and the advertisement information to the transmission / reception unit 201 (S309). Member information includes user point information, expiration date information, and the like. The transmission / reception unit 201 transmits member information and advertisement information to the smartphone 102 (S310). The smartphone 102 displays the received information on the display screen (S311).
In addition, the smartphone 102 transfers the information received from the server 103 to the wearable device 101 (S312). If the notification setting of the wearable device 101 is ON, the wearable device 101 displays information (S314). When displaying information on a wearable device, it is desirable to alert the user by vibration or the like. This is because the user does not necessarily enter the store, and even if the coupon information is sent, the user may not notice it.
FIG. 437 is a diagram showing an information table transmitted by the server 103 to the smartphone 102. The store map DB is information on the inside of the store, such as which product is displayed at which position in the store. The product information of a store is information on products in stock at the store, product price information, and the like. User member information is user point information, membership card expiration date information, and the like.
FIG. 438 is a diagram showing a screen flow displayed on the wearable device 101 from the time when the user receives information from the server 103 in front of the store to the time when the user actually purchases the product. In front of the store, points given when the user visits the store and coupon information are displayed. When the user taps the coupon information, the information sent from the server 103 according to the user's taste is displayed. For example, when a user taps a TV, information on recommended TVs is displayed. If you press the purchase button here, a screen for selecting the receiving method will be displayed, and you can select whether to deliver to your home or receive in the store. In this embodiment, since the user knows which store the user is in, there is an advantage that the user can receive it at the store. If you select to direct to the sales floor in flow 403, the wearable device 101 shifts to Guide Mode. This mode is a mode in which the user is guided to a specific place by using an arrow or the like, and the user can be guided to the place where the selected product is actually displayed. When guided to the front of the product shelf, the wearable device 101 transitions to a screen inquiring whether or not to purchase. By actually seeing the product, the user can try the size, color, usability, etc. before making a purchase decision.
The visible light communication in the present invention can accurately identify the user's position. Therefore, it is possible to give a warning when a user is likely to enter a dangerous area in a factory as shown in FIG. 439, for example. Furthermore, since it is possible for the wearable device to determine whether or not to issue a warning, it is possible to construct a warning system with a high degree of freedom, for example, to issue a warning to children under a specific age.
(Embodiment 18) FIG. 440 is a diagram showing a service providing system using the receiving method described in the above-described embodiment.
First, another company B or an individual ex8001 requests the company A ex8000 that manages the server ex8002 to deliver the information to the mobile terminal. For example, a mobile terminal that communicates with signage by visible light is requested to deliver detailed advertisement information, coupon information, map information, or the like. The company A ex8000 that manages the server manages the information distributed to the mobile terminal in correspondence with arbitrary ID information. The mobile terminal ex8003 acquires ID information from the subject ex8004 by visible light communication, and transmits the acquired ID information to the server ex8002. The server ex8002 transmits the information corresponding to the ID information to the mobile terminal and counts the number of times the information corresponding to the ID information is transmitted. The company A ex8000 that manages the server charges the requesting company B and the individual ex8001 according to the number of times of counting. For example, the larger the number of counts, the larger the amount to be charged.
FIG. 441 is a flowchart showing the flow of service provision.
In Step ex8000, the company A that manages the server receives a request for information distribution from another company B. Next, in Step ex8001, the information for which the distribution request has been received is associated with the specific ID information on the server managed by the company A. In Step ex8002, the mobile terminal receives specific ID information from the subject by visible light communication and sends it to the server managed by company A. The details of the visible light communication method will be omitted because they have already been described in the other embodiments. The server transmits information corresponding to the specific ID information transmitted from the mobile terminal to the mobile terminal. Step ex8003 counts the number of times information is distributed on the server. Finally, in Step ex8004, the fee according to the count number of information distribution is charged to the company B. In this way, by charging according to the number of counts, it is possible to charge the company B an appropriate charge according to the advertising effect of the information distribution.
FIG. 442 is a flowchart showing service provision in another example. The description of the steps overlapping with FIG. 441 will be omitted.
In Step ex8008, it is determined whether or not a predetermined time has elapsed from the start of information distribution. If it is determined that the time is within the specified time, the company B will not be charged in Step ex8011. On the other hand, if it is determined that the predetermined period has passed, the number of times the information is distributed is counted in Step ex8009. Then, in Step ex8010, the fee according to the count of information distribution is charged to the company B. In this way, since the information is distributed free of charge within the predetermined period, the company B can receive the billing service after confirming the advertising effect and the like.
FIG. 443 is a flowchart showing service provision in another example. The description of the steps overlapping with FIG. 442 will be omitted.
In Step ex8014, the number of times the information is distributed is counted. If it is determined in Step ex8015 that the predetermined period has not passed since the start of information distribution, no charge will be made in Step ex8016. On the other hand, if it is determined that the predetermined period has passed, it is determined in Step ex8017 whether or not the number of times the information has been distributed is equal to or greater than the predetermined value. If the number of times the information is delivered is less than the predetermined value, the count number is reset and the number of times the information is delivered is counted again. In this case, the company B is not charged for the predetermined period in which the number of times the information is distributed is less than the predetermined value. In Step ex8017, if the count number is equal to or more than the predetermined value, the count number is reset once in Step ex8018 and the count is restarted again. Step In ex8019, the charge according to the number of counts is charged to the company B. In this way, when the count number is small within the period of free distribution, the company B can receive the billing service at an appropriate timing by setting the period of free distribution again. In addition, company A can also analyze the information content when the count number is small, and propose to company B to change the information content, for example, when the information does not correspond to the season. It will be possible. If the free information distribution period is to be provided again, the period may be shorter than the initial predetermined period. By making it shorter than the initial predetermined period, the burden on the company A can be reduced. In addition, a free distribution period may be provided again after a certain period of time. For example, if the information is affected by the season, a free distribution period can be set again after a certain period of time until the season changes.
The billing fee may be changed according to the amount of data regardless of the number of times the information is distributed. Distribution of a certain amount of data may be free of charge, and charges may be made for more than a predetermined amount of data. Further, as the amount of data increases, the billing fee may also increase. Further, a management fee may be charged when managing the information in association with specific ID information. By charging as a management fee, it is possible to determine the fee at the time of requesting information distribution.
Although the information and communication methods according to one or more embodiments have been described above based on the embodiments, the present invention is not limited to the embodiments. As long as it does not deviate from the gist of the present invention, a form in which various modifications conceived by those skilled in the art are applied to the present embodiment or a form constructed by combining components in different embodiments is also within the scope of one or a plurality of embodiments. May be included within.
FIG. 444A is a flowchart of an information communication method according to one aspect of the present invention.
The information communication method according to one aspect of the present invention is an information communication method for acquiring information from a subject, and includes steps SK51 to SK56.
That is, this information communication method includes a position information transmission step SK51 for transmitting position information indicating the position of the image sensor used for photographing the subject, and a plurality of identification information associated with the position indicated by the position information. In the list receiving step SK52 for receiving the ID list including the image sensor and the image obtained by photographing the subject by the image sensor, a bright line corresponding to the exposure line included in the image sensor is generated according to the change in the brightness of the subject. In addition, an exposure time setting step SK53 for setting the exposure time of the image sensor and the image sensor taking a picture of the subject whose brightness changes at the set exposure time obtains a bright line image including the bright line. Image acquisition step SK54 to acquire information by demodulating the data specified by the emission line pattern included in the acquired emission line image, and information acquisition step SK55 to acquire information, and identification information including the acquired information. It includes a search step SK56 for searching from the ID list.
FIG. 444B is a block diagram of an information communication device according to an aspect of the present invention.
The information communication device K50 according to one aspect of the present invention is an information communication device that acquires information from a subject, and includes components K51 to K56.
That is, the information communication device K50 has a position information transmission unit K51 that transmits position information indicating the position of the image sensor used for photographing the subject, and a plurality of identifications associated with the position indicated by the position information. A bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by the list receiving unit K52 for receiving an ID list containing information and the image sensor according to a change in the brightness of the subject. As described above, the exposure time setting unit K53 for setting the exposure time of the image sensor and the image sensor for acquiring the emission line image including the emission line by photographing the subject whose brightness changes at the set exposure time. The image acquisition unit K54 having the Is provided with a search unit K56 that searches the ID list.
In the information communication method and the information communication device K50 shown by FIGS. 444A and 444B, the information transmitted by the change in the brightness of the subject is acquired by the exposure of the exposure line of the image sensor, and therefore, for example, wireless communication is performed. It is possible to enable communication between various devices without requiring a special communication device for the purpose. Further, as shown in FIG. 389, for example, since the ID list has been received in advance, even if the acquired information "bc" is only a part of the identification information, the appropriate identification information "" is based on the ID list. abcd "can be identified.
In each of the above embodiments, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. For example, the program causes the computer to execute the information communication method shown by the flowchart of FIG. 444A.
The present invention can be used for information communication devices and the like, and in particular, for information communication devices and the like used for communication methods between mobile terminals such as smartphones, tablets and mobile phones and home appliances such as air conditioners, lighting devices and rice cookers. It can be used.
1101 TV 1105 Smartphone 1106 Microphone 1107 Air purifier 1201 Smartphone 1301 Transmitter device 1309 Transmission speed determination unit 1401 Receiver device 1404 Image acquisition unit 1406 Blinking information acquisition unit 3001a Mobile terminal 3001b Doorphone Home appliance 3001c Doorphone Home delivery order 3001e Server 3001f Delivery person mobile terminal 4040a Microscope table 4040b Mobile phone table 4040c Mobile phone model table 4040d User voice characteristic table 4040e User keyword voice table 4040f User-owned device position table 4040h User position table K50 Information communication device K51 Location information transmitter K52 List receiver K53 Exposure time setting unit K54 Image acquisition unit K55 Information acquisition unit K56 Search unit
500 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23 Sheet 24 Sheet 25 Sheet 26 Sheet 27 Sheet 28 Sheet 29 Sheet 30 Sheet 31 Sheet 32 Sheet 33 Sheet 34 Sheet 35 Sheet 36 Sheet 37 Sheet 38 Sheet 39 Sheet 40 Sheet 41 Sheet 42 Sheet 43 Sheet 44 Sheet 45 Sheet 46 Sheet 47 Sheet 48 Sheet 49 Sheet 50 Sheet 51 Sheet 52 Sheet 53 Sheet 54 Sheet 55 Sheet 56 Sheet 57 Sheet 58 Sheet 59 Sheet 60 Sheet 61 Sheet 62 Sheet 63 Sheet 64 Sheet 65 Sheet 66 Sheet 67 Sheet 68 Sheet 69 Sheet 70 Sheet 71 Sheet 72 Sheet 73 Sheet 74 Sheet 75 Sheet 76 Sheet 77 Sheet 78 Sheet 79 Sheet 80 Sheet 81 Sheet 82 Sheet 83 Sheet 84 Sheet 85 Sheet 86 Sheet 87 Sheet 88 Sheet 89 Sheet 90 Sheet 91 Sheet 92 Sheet 93 Sheet 94 Sheet 95 Sheet 96 Sheet 97 Sheet 98 Sheet 99 Sheet 100 Sheet 101 Sheet 102 Sheet 103 Sheet 104 Sheet 105 Sheet 106 Sheet 107 Sheet 108 Sheet 109 Sheet 110 Sheet 111 Sheet 112 Sheet 113 Sheet 114 Sheet 115 Sheet 116 Sheet 117 Sheet 118 Sheet 119 Sheet 120 Sheet 121 Sheet 122 Sheet 123 Sheet 124 Sheet 125 Sheet 126 Sheet 127 Sheet 128 Sheet 129 Sheet 130 Sheet 131 Sheet 132 Sheet 133 Sheet 134 Sheet 135 Sheet 136 Sheet 137 Sheet 138 Sheet 139 Sheet 140 Sheet 141 Sheet 142 Sheet 143 Sheet 144 Sheet 145 Sheet 146 Sheet 147 Sheet 148 Sheet 149 Sheet 150 Sheet 151 Sheet 152 Sheet 153 Sheet 154 Sheet 155 Sheet 156 Sheet 157 Sheet 158 Sheet 159 Sheet 160 Sheet 161 Sheet 162 Sheet 163 Sheet 164 Sheet 165 Sheet 166 Sheet 167 Sheet 168 Sheet 169 Sheet 170 Sheet 171 Sheet 172 Sheet 173 Sheet 174 Sheet 175 Sheet 176 Sheet 177 Sheet 178 Sheet 179 Sheet 180 Sheet 181 Sheet 182 Sheet 183 Sheet 184 Sheet 185 Sheet 186 Sheet 187 Sheet 188 Sheet 189 Sheet 190 Sheet 191 Sheet 192 Sheet 193 Sheet 194 Sheet 195 Sheet 196 Sheet 197 Sheet 198 Sheet 199 Sheet 200 Sheet 201 Sheet 202 Sheet 203 Sheet 204 Sheet 205 Sheet 206 Sheet 207 Sheet 208 Sheet 209 Sheet 210 Sheet 211 Sheet 212 Sheet 213 Sheet 214 Sheet 215 Sheet 216 Sheet 217 Sheet 218 Sheet 219 Sheet 220 Sheet 221 Sheet 222 Sheet 223 Sheet 224 Sheet 225 Sheet 226 Sheet 227 Sheet 228 Sheet 229 Sheet 230 Sheet 231 Sheet 232 Sheet 233 Sheet 234 Sheet 235 Sheet 236 Sheet 237 Sheet 238 Sheet 239 Sheet 240 Sheet 241 Sheet 242 Sheet 243 Sheet 244 Sheet 245 Sheet 246 Sheet 247 Sheet 248 Sheet 249 Sheet 250 Sheet 251 Sheet 252 Sheet 253 Sheet 254 Sheet 255 Sheet 256 Sheet 257 Sheet 258 Sheet 259 Sheet 260 Sheet 261 Sheet 262 Sheet 263 Sheet 264 Sheet 265 Sheet 266 Sheet 267 Sheet 268 Sheet 269 Sheet 270 Sheet 271 Sheet 272 Sheet 273 Sheet 274 Sheet 275 Sheet 276 Sheet 277 Sheet 278 Sheet 279 Sheet 280 Sheet 281 Sheet 282 Sheet 283 Sheet 284 Sheet 285 Sheet 286 Sheet 287 Sheet 288 Sheet 289 Sheet 290 Sheet 291 Sheet 292 Sheet 293 Sheet 294 Sheet 295 Sheet 296 Sheet 297 Sheet 298 Sheet 299 Sheet 300 Sheet 301 Sheet 302 Sheet 303 Sheet 304 Sheet 305 Sheet 306 Sheet 307 Sheet 308 Sheet 309 Sheet 310 Sheet 311 Sheet 312 Sheet 313 Sheet 314 Sheet 315 Sheet 316 Sheet 317 Sheet 318 Sheet 319 Sheet 320 Sheet 321 Sheet 322 Sheet 323 Sheet 324 Sheet 325 Sheet 326 Sheet 327 Sheet 328 Sheet 329 Sheet 330 Sheet 331 Sheet 332 Sheet 333 Sheet 334 Sheet 335 Sheet 336 Sheet 337 Sheet 338 Sheet 339 Sheet 340 Sheet 341 Sheet 342 Sheet 343 Sheet 344 Sheet 345 Sheet 346 Sheet 347 Sheet 348 Sheet 349 Sheet 350 Sheet 351 Sheet 352 Sheet 353 Sheet 354 Sheet 355 Sheet 356 Sheet 357 Sheet 358 Sheet 359 Sheet 360 Sheet 361 Sheet 362 Sheet 363 Sheet 364 Sheet 365 Sheet 366 Sheet 367 Sheet 368 Sheet 369 Sheet 370 Sheet 371 Sheet 372 Sheet 373 Sheet 374 Sheet 375 Sheet 376 Sheet 377 Sheet 378 Sheet 379 Sheet 380 Sheet 381 Sheet 382 Sheet 383 Sheet 384 Sheet 385 Sheet 386 Sheet 387 Sheet 388 Sheet 389 Sheet 390 Sheet 391 Sheet 392 Sheet 393 Sheet 394 Sheet 395 Sheet 396 Sheet 397 Sheet 398 Sheet 399 Sheet 400 Sheet 401 Sheet 402 Sheet 403 Sheet 404 Sheet 405 Sheet 406 Sheet 407 Sheet 408 Sheet 409 Sheet 410 Sheet 411 Sheet 412 Sheet 413 Sheet 414 Sheet 415 Sheet 416 Sheet 417 Sheet 418 Sheet 419 Sheet 420 Sheet 421 Sheet 422 Sheet 423 Sheet 424 Sheet 425 Sheet 426 Sheet 427 Sheet 428 Sheet 429 Sheet 430 Sheet 431 Sheet 432 Sheet 433 Sheet 434 Sheet 435 Sheet 436 Sheet 437 Sheet 438 Sheet 439 Sheet 440 Sheet 441 Sheet 442 Sheet 443 Sheet 444 Sheet 445 Sheet 446 Sheet 447 Sheet 448 Sheet 449 Sheet 450 Sheet 451 Sheet 452 Sheet 453 Sheet 454 Sheet 455 Sheet 456 Sheet 457 Sheet 458 Sheet 459 Sheet 460 Sheet 461 Sheet 462 Sheet 463 Sheet 464 Sheet 465 Sheet 466 Sheet 467 Sheet 468 Sheet 469 Sheet 470 Sheet 471 Sheet 472 Sheet 473 Sheet 474 Sheet 475 Sheet 476 Sheet 477 Sheet 478 Sheet 479 Sheet 480 Sheet 481 Sheet 482 Sheet 483 Sheet 484 Sheet 485 Sheet 486 Sheet 487 Sheet 488 Sheet 489 Sheet 490 Sheet 491 Sheet 492 Sheet 493 Sheet 494 Sheet 495 Sheet 496 Sheet 497 Sheet 498 Sheet 499 Sheet 500
Every citation, both ways
| Document | Relation | Office |
|---|---|---|
| JP2004326241A | Cites | Japan |
| JP2012010269A | Cites | Japan |
| JP2012113655A | Cites | Japan |
| WO2012127439A1 | Cites | World Intellectual Property Organization (WIPO) |
| JP2012209622A | Cites | Japan |
| JP2011254317A | Cites | Japan |
| JP2007124404A | Cites | Japan |
| JP2006319545A | Cites | Japan |
| JP2005038181A | Cites | Japan |
| US20120292392A1 | Cites | United States of America |
| CN1914940A | Cites | China |
557 members in 20 offices
Priority claims84
| Document | Office | Kind | Date |
|---|---|---|---|
| 2012286339 | Japan | A | |
| 2012286339 | Japan | A | |
| 2012286339 | Japan | – | |
| 201261746315 | United States of America | P | |
| 201261746315 | United States of America | P | |
| 61746315 | United States of America | – | |
| 2013070740 | Japan | A | |
| 2013070740 | Japan | A | |
| 2013070740 | Japan | – | |
| 201361805978 | United States of America | P | |
| 201361805978 | United States of America | P | |
| 61805978 | United States of America | – | |
| 2013082546 | Japan | A | |
| 2013082546 | Japan | A | |
| 2013082546 | Japan | – | |
| 201361810291 | United States of America | P | |
| 201361810291 | United States of America | P | |
| 61810291 | United States of America | – | |
| 13902215 | United States of America | – | |
| 13902436 | United States of America | – | |
| 2013003318 | Japan | W | |
| 2013003318 | Japan | W | |
| 2013003319 | Japan | W | |
| 2013003319 | Japan | W | |
| 201313902215 | United States of America | A | |
| 201313902215 | United States of America | A | |
| 201313902436 | United States of America | A | |
| 201313902436 | United States of America | A | |
| PCTJP2013003318 | Japan | – | |
| PCTJP2013003319 | Japan | – | |
| 2013158359 | Japan | A | |
| 2013158359 | Japan | A | |
| 2013158359 | Japan | – | |
| 201361859902 | United States of America | P | |
| 201361859902 | United States of America | P | |
| 61859902 | United States of America | – | |
| 2013180729 | Japan | A | |
| 2013180729 | Japan | A | |
| 2013180729 | Japan | – | |
| 201361872028 | United States of America | P | |
| 201361872028 | United States of America | P | |
| 61872028 | United States of America | – | |
| 2013210623 | Japan | A | |
| 2013210623 | Japan | A | |
| 2013210623 | Japan | – | |
| 201361887541 | United States of America | P | |
| 201361887541 | United States of America | P | |
| 61887541 | United States of America | – | |
| 2018156280 | Japan | A | |
| 2018156280 | Japan | A | |
| 13902215 | – | – | – |
| 13902436 | – | – | – |
| 2012286339 | – | – | – |
| 2013070740 | – | – | – |
| 2013082546 | – | – | – |
| 2013158359 | – | – | – |
| 2013180729 | – | – | – |
| 2013210623 | – | – | – |
| 2018156280 | – | – | – |
| 61746315 | – | – | – |
| 61805978 | – | – | – |
| 61810291 | – | – | – |
| 61859902 | – | – | – |
| 61872028 | – | – | – |
| 61887541 | – | – | – |
| JP20120286339 | – | – | – |
| JP20130070740 | – | – | – |
| JP20130082546 | – | – | – |
| JP20130158359 | – | – | – |
| JP20130180729 | – | – | – |
| JP20130210623 | – | – | – |
| JP20180156280 | – | – | – |
| PCTJP2013003318 | – | – | – |
| PCTJP2013003319 | – | – | – |
| US201261746315P | – | – | – |
| US201313902215 | – | – | – |
| US201313902436 | – | – | – |
| US201361805978P | – | – | – |
| US201361810291P | – | – | – |
| US201361859902P | – | – | – |
| US201361872028P | – | – | – |
| US201361887541P | – | – | – |
| WO2013JP03318 | – | – | – |
| WO2013JP03319 | – | – | – |
Members557
| Document | Office | Kind | |
|---|---|---|---|
| WO2013175803A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2013175804A1 | World Intellectual Property Organization (WIPO) | A1 | |
| US2013330088A1 | United States of America | A1 | |
| US2013335592A1 | United States of America | A1 | |
| US2013337787A1 | United States of America | A1 | |
| JP5393917B1 | Japan | B1 | |
| JP5395293B1 | Japan | B1 | |
| JP5405695B1 | Japan | B1 | |
| US2014037296A1 | United States of America | A1 | |
| CN103650383A | China | A | |
| CN103650384A | China | A | |
| JP5521125B2 | Japan | B2 | |
| JP5521128B1 | Japan | B1 | |
| JP5525661B1 | Japan | B1 | |
| JP5525662B1 | Japan | B1 | |
| JP5525663B1 | Japan | B1 | |
| JP5525664B1 | Japan | B1 | |
| JP5530578B1 | Japan | B1 | |
| SG11201400469SA | Singapore | A | |
| JP5536293B1 | Japan | B1 | |
| JP5536294B1 | Japan | B1 | |
| US2014184914A1 | United States of America | A1 | |
| US2014185860A1 | United States of America | A1 | |
| US2014186026A1 | United States of America | A1 | |
| US2014186047A1 | United States of America | A1 | |
| US2014186048A1 | United States of America | A1 | |
| US2014186049A1 | United States of America | A1 | |
| US2014186050A1 | United States of America | A1 | |
| US2014186052A1 | United States of America | A1 | |
| US2014186055A1 | United States of America | A1 | |
| WO2014103151A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2014103152A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2014103153A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2014103154A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2014103155A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2014103156A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2014103157A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2014103158A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2014103159A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2014103160A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2014103161A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2014103329A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2014103333A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2014103340A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2014103341A1 | World Intellectual Property Organization (WIPO) | A1 | |
| US2014192185A1 | United States of America | A1 | |
| US2014192226A1 | United States of America | A1 | |
| JP5557972B1 | Japan | B1 | |
| US2014204129A1 | United States of America | A1 | |
| US2014205136A1 | United States of America | A1 | |
| US2014207517A1 | United States of America | A1 | |
| JP5564630B1 | Japan | B1 | |
| JP5564631B1 | Japan | B1 | |
| JP5564632B1 | Japan | B1 | |
| JP5564636B1 | Japan | B1 | |
| US2014212145A1 | United States of America | A1 | |
| US2014212146A1 | United States of America | A1 | |
| US2014232896A1 | United States of America | A1 | |
| US2014232903A1 | United States of America | A1 | |
| SG11201400251XA | Singapore | A | |
| US8823852B2 | United States of America | B2 | |
| JP5589200B1 | Japan | B1 | |
| JP5590431B1 | Japan | B1 | |
| US2014286644A1 | United States of America | A1 | |
| US2014290138A1 | United States of America | A1 | |
| US2014294397A1 | United States of America | A1 | |
| US2014294398A1 | United States of America | A1 | |
| JP5602966B1 | Japan | B1 | |
| JP5603512B1 | Japan | B1 | |
| JP5603513B1 | Japan | B1 | |
| JP5603523B1 | Japan | B1 | |
| JP5606653B1 | Japan | B1 | |
| JP5606655B1 | Japan | B1 | |
| JP5607277B1 | Japan | B1 | |
| JP5608307B1 | Japan | B1 | |
| JP5608834B1 | Japan | B1 | |
| US2014307155A1 | United States of America | A1 | |
| US2014307156A1 | United States of America | A1 | |
| US2014307157A1 | United States of America | A1 | |
| SG11201400255RA | Singapore | A | |
| JP2014212503A | Japan | A | |
| JP2014212504A | Japan | A | |
| JP2014220783A | Japan | A | |
| JP2014220787A | Japan | A | |
| JP2014220788A | Japan | A | |
| JP2014220789A | Japan | A | |
| JP2014220790A | Japan | A | |
| JP2014220791A | Japan | A | |
| US8908074B2 | United States of America | B2 | |
| US8913144B2 | United States of America | B2 | |
| US2014376922A1 | United States of America | A1 | |
| US8922666B2 | United States of America | B2 | |
| US2015050027A1 | United States of America | A1 | |
| US8965216B2 | United States of America | B2 | |
| JP5683737B1 | Japan | B1 | |
| JP2015046860A | Japan | A | |
| JP2015046861A | Japan | A | |
| JP2015046862A | Japan | A | |
| JP2015046863A | Japan | A | |
| JP2015046864A | Japan | A |
10 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Written amendmentJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Written request for extension of timeJAPANESE INTERMEDIATE CODE: A601A601 | A601 | |
| Written request for extension of timeJAPANESE INTERMEDIATE CODE: A601A601 | A601 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Report on retrievalJAPANESE INTERMEDIATE CODE: A971007A977 | A977 | |
| Written request for application examinationJAPANESE INTERMEDIATE CODE: A621A621 | A621 |
Numbers
- Publication
- 6970146
- Publication, DOCDB
- 6970146
- Publication, EPODOC
- JP6970146B
- Application
- 142553
- Application, DOCDB
- 2019142553
- Application, EPODOC
- JP20190142553
Titles2
- Japanese
- プログラム、制御方法、および情報通信装置
- English
- Programs, control methods, and information and communication equipment
Classification
- IPC, 5
- H04B10 116
- H04B10 516
- H04B10 69
- H04Q9 00
- F21V23 00
