Signal recognizing device, signal recognizing method and signal recognizing program

TECHNICAL FIELD

0001The present invention relates to a signal recognizing technology of recognizing a display of a traffic signal.

BACKGROUND TECHNIQUE

0002Conventionally, there is proposed a technique of recognizing a traffic signal by using the time variation of the colors in the lamps of the traffic signal. For example, Patent Reference-1 discloses a technique which considers the traffic signal lamps, which are not determined to be lit, to be unlit, and which recognizes the lit state of the lamp by comparing the brightness detected at the unlit state to the present brightness. Patent Reference-2 discloses a technique of extracting a candidate area of the signal lamp on the basis of the color and the degree of the circularity. Furthermore, Patent Reference-2 also discloses a technique of determining whether or not it happens that a candidate area of the red lamp vanishes and that a new candidate area of the red lamp arises by comparing the previous frame to the present frame in order to detect a lit state of the green lamp.

0003Patent Reference-1: Japanese Patent Application Laid-open under No. 2000-353292

0004Patent Reference-2: Japanese Patent Application Laid-open under No. 2005-301519

DISCLOSURE OF INVENTION

Problem to be Solved by the Invention

0005As described above, by detecting the variation of the candidate area of the red lamp from vanishing to arising by comparing the previous frame to the present frame, it is possible to detect a lit state of the green lamp. In contrast, when a state of a simultaneous lighting (hereinafter referred to as “red-green simultaneous lighting”) of the red lamp and the green lamp is displayed in the image due to the exposure time of the camera, it occurs that the difference of the brightness of the candidate area of the green lamp between the previous frame and the present frame becomes small. Thus, in this case, the signal recognizing device cannot properly recognize the lit state of the green lamp. In a case where the traffic signal is temporarily obscured by a tall vehicle such as a truck, a similar problem as the above-mentioned red-green simultaneous lighting arises. In Patent Reference-1 and Patent Reference-2, the above problem is not disclosed at all.

0006The above is an example of the problem to be solved by the present invention. An object of the present invention is to provide a signal recognizing device which precisely recognizes the display of a traffic signal even in the case where the red-green simultaneous lighting or the obstruction occurs.

Means for Solving the Problem

0007The invention according to claim <b>1</b> is a signal recognizing device which is mounted on a moving body and which electromagnetically connects to a camera, comprising: a signal display candidate detecting unit which detects a green lamp candidate area from an image obtained from the camera; a reference calculating unit which calculates a time-series statistic of unit area values per unit area of the green lamp candidate area in a state where a red lamp is lit, and which stores the statistics on a storage unit; and a green lighting determination unit which determines whether or not a green lamp is lit based on the image including the green lamp candidate area in a state where the red lamp is unlit and the statistics, in case of determining that the red lamp varies from a lit state to an unlit state; wherein the reference calculating unit omits unit area values of unit areas of the green lamp candidate area from samples for calculating the statistics, provided that between the unit area values and either the statistics or unit area values of unit areas of a green lamp candidate area existing in a previously obtained image are larger than a predetermined threshold.

0008The invention according to claim <b>8</b> is a signal recognizing method executed by a signal recognizing device which is mounted on a moving body and which electromagnetically connects to a camera, comprising: a signal display candidate detecting process which detects a green lamp candidate area from an image obtained from the camera; a reference calculating process which calculates a time-series statistic of unit area values per unit area of the green lamp candidate area in a state where a red lamp is lit, and which stores the statistics on a storage unit; and a green lighting determination process which determines whether or not a green lamp is lit based on the image including the green lamp candidate area in a state where the red lamp is unlit and the statistics, in case of determining that the red lamp varies from a lit state to an unlit state; wherein the reference calculating process omits unit area values of unit areas of the green lamp candidate area from samples for calculating the statistics, provided that between the unit area values and either the statistics or unit area values of unit areas of a green lamp candidate area existing in a previously obtained image are larger than a predetermined threshold.

0009The invention according to claim <b>9</b> is a signal recognizing program executed by a signal recognizing device which is mounted on a moving body and which electromagnetically connects to a camera, making the signal recognizing device function as: a signal display candidate detecting unit which detects a green lamp candidate area from an image obtained from the camera; a reference calculating unit which calculates a time-series statistic of unit area values per unit area of the green lamp candidate area in a state where a red lamp is lit, and which stores the statistics on a storage unit; and a green lighting determination unit which determines whether or not a green lamp is lit based on the image including the green lamp candidate area in a state where the red lamp is unlit and the statistics, in case of determining that the red lamp varies from a lit state to an unlit state; wherein the reference calculating unit omits unit area values of unit areas of the green lamp candidate area from samples for calculating the statistics, provided that between the unit area values and either the statistics or unit area values of unit areas of a green lamp candidate area existing in a previously obtained image are larger than a predetermined threshold.

BRIEF DESCRIPTION OF THE DRAWINGS

0010<figref idref="DRAWINGS">FIG. 1</figref> is one example of a schematic configuration of the signal recognizing device.

0011<figref idref="DRAWINGS">FIG. 2</figref> is one example of the functional block of the signal recognizing device.

0012<figref idref="DRAWINGS">FIGS. 3A and 3B</figref> show images each including the traffic signal and enlarged green lamp candidate area Lb.

0013<figref idref="DRAWINGS">FIGS. 4A and 4B</figref> show the image captured in the case where the obstruction occurs and an outline of the process executed by the reference calculating unit <b>202</b>.

0014<figref idref="DRAWINGS">FIG. 5</figref> shows an outline of the process executed by the green lighting determination unit <b>203</b>.

0015<figref idref="DRAWINGS">FIG. 6</figref> is one example of a flowchart showing a procedure of the process according to the embodiment.

0016<figref idref="DRAWINGS">FIG. 7</figref> is an outline of the process executed by the green lighting determination unit <b>203</b> according to the comparison example.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

0017According to one aspect of the present invention, there is provided a signal recognizing device which is mounted on a moving body and which electromagnetically connects to a camera, comprising: a signal display candidate detecting unit which detects a green lamp candidate area from an image obtained from the camera; a reference calculating unit which calculates a time-series statistic of unit area values per unit area of the green lamp candidate area in a state where a red lamp is lit, and which stores the statistics on a storage unit; and a green lighting determination unit which determines whether or not a green lamp is lit based on the image including the green lamp candidate area in a state where the red lamp is unlit and the statistics, in case of determining that the red lamp varies from a lit state to an unlit state; wherein the reference calculating unit omits unit area values of unit areas of the green lamp candidate area from samples for calculating the statistics, provided that between the unit area values and either the statistics or unit area values of unit areas of a green lamp candidate area existing in a previously obtained image are larger than a predetermined threshold.

0018The above signal recognizing device is mounted on a moving body such as a vehicle, and electromagnetically connects to a camera by wire or wireless. The signal recognizing device includes a storage unit, a signal display candidate detecting unit, a reference calculating unit and a green lighting determination unit. The signal display candidate detecting unit detects a green lamp candidate area of the traffic signal from an image obtained from the camera. The term “green lamp candidate area” herein indicates an area used by the signal recognizing device as a display part of the green lamp in the image obtained from the camera. The reference calculating unit calculates time-series statistics of unit area values of unit areas to store the statistics in the storage unit. The term “unit area” herein indicates an area composed of a predetermined number of adjacent pixels and divided on the basis of a predetermined protocol. In other words, the unit area is an area which includes a predetermined number of adjacent pixels, and it may consist of one pixel or may be an area corresponding to the green lamp candidate area. The term “unit area value” herein indicates a value calculated based on statistical processing of the pixel values of each pixel existing in a unit area such as the average and the representative value. The term “pixel value” is an all-inclusive term of an index value which shows information each pixel has such as brightness, saturation and hue. The term “statistic” indicates a value obtained by statistical processing, and the statistical processing is an averaging processing for calculating the moving average, for example. At that time, the reference calculating unit calculates each difference between the unit area values in the green lamp candidate area and either the statistics stored in the storage unit or the unit area values of unit areas of the green lamp candidate area existing in a previously obtained image. Then, the reference calculating unit omits the unit area values from the samples for calculating the statistics when the above-mentioned each difference is larger than a predetermined threshold. In other words, as to the unit areas of the green lamp candidate area where the above-mentioned each difference is equal to or smaller than the threshold, the reference calculating unit updates the statistics by using the unit area values, and as to the unit areas in the green lamp candidate area where the above-mentioned each difference is larger than the threshold, it keeps using the statistics without updating them. Here, “the previously obtained image” may be the most recently obtained image or may be an image obtained before a predetermined duration. The green lighting determination unit determines whether or not a green lamp is lit based on the image including the red lamp candidate area at the time when the red lamp is unlit and the statistics stored in the storage unit, in case of determining that the red lamp varies from a lit state to an unlit state. Thereby, the signal recognizing device can precisely recognize the green lit state even when the obstruction or the red-green simultaneous lighting occurs.

0019In one mode of the signal recognizing device, the statistic is a moving average.

0020In another mode of the signal recognizing device, the reference calculating unit changes the threshold based on a state of the moving body and/or the image. The term “changes the threshold based on a state of the moving body” herein indicate dynamically changing the threshold based on the velocity and/or the acceleration of the moving body, for example. The term “changes the threshold based on the image” herein indicates dynamically setting the threshold based on the pixel values or the unit area values of the red lamp area or yellow lamp area in the obtained image. Thereby, the signal recognizing device can flexibly set the above-mentioned threshold based on the state of the moving body and others.

0021In another mode of the signal recognizing device, the signal display candidate detecting unit further detects a red lamp candidate area from the image; the unit area value is a value calculated based on brightness of each pixel existing in the unit area; and the threshold is set based on brightness of the red lamp candidate area. Generally, the brightness in the lit state of the red lamp almost coincides with the brightness in the lit state of the green lamp, and the brightness in the unlit state of the red lamp almost coincides with the brightness in the unlit state of the green lamp. Thus, by setting the threshold based on the brightness of the red lamp candidate area, the signal recognizing device can prevent the unit area value of the green lamp candidate area in the lit state from being included in the calculation of the statistic even when the red-green simultaneous lighting occurs.

0022In another mode of the signal recognizing device, the threshold is set to one half of the brightness of the red lamp candidate area in a state where the red lamp is lit. The term “the brightness of the red lamp candidate area” herein indicates an average or another representative value of the brightness of the pixels existing in the red lamp candidate area. In this mode, the signal recognizing device considers the brightness of the red lamp in the unlit state as 0, and sets the above-mentioned threshold to one half of the brightness of the red lamp candidate area in the lit state of the red lamp. In this way, by setting the above-mentioned threshold based on the brightness of the red lamp candidate area, the signal recognizing device can prevent the unit area value of the green lamp candidate area in the lit state from being included in the calculation of the statistic even when the red-green simultaneous lighting occurs.

0023In another mode of the signal recognizing device, the threshold is set to a larger value as a speed of the moving body is higher. In this mode, considering the fact that the time variation in the image obtained from the camera becomes great in a case where the velocity of the moving body is high, the signal recognizing device changes the threshold to a more permissive value in the case. In this way, the signal recognizing device can flexibly set the above-mentioned threshold according to the velocity of the moving body.

0024In another mode of the signal recognizing device, the reference calculating unit calculates the statistics by omitting a predetermined number of the images obtained just before it determines that the red lamp varies from a lit state to an unlit state, and the green lighting determination unit determines whether or not the green lamp is lit based on the statistics. In this mode, the signal recognizing device determines that the red-green simultaneous lighting is likely to be displayed in the predetermined number of the images obtained just before it determines that the red lamp varies from a lit state to an unlit state, and does not use them for calculating the statistics. Thereby, the signal recognizing device can properly calculate the statistics of the green lamp candidate area at the time of the unlit state of the green lamp even when the red-green simultaneous lighting is displayed in the image.

0025According to another aspect of the present invention, there is provided a signal recognizing method executed by a signal recognizing device which is mounted on a moving body and which electromagnetically connects to a camera, comprising: a signal display candidate detecting process which detects a green lamp candidate area from an image obtained from the camera; a reference calculating process which calculates a time-series statistic of unit area values per unit area of the green lamp candidate area in a state where a red lamp is lit, and which stores the statistics on a storage unit; and a green lighting determination process which determines whether or not a green lamp is lit based on the image including the green lamp candidate area in a state where the red lamp is unlit and the statistics, in case of determining that the red lamp varies from a lit state to an unlit state; wherein the reference calculating process omits unit area values of unit areas of the green lamp candidate area from samples for calculating the statistics, provided that between the unit area values and either the statistics or unit area values of unit areas of a green lamp candidate area existing in a previously obtained image are larger than a predetermined threshold. By executing the above-mentioned method, the signal recognizing device can precisely recognize the green lit state even when the obstruction or the red-green simultaneous lighting occurs.

0026According to still another aspect of the present invention, there is provided a signal recognizing program executed by a signal recognizing device which is mounted on a moving body and which electromagnetically connects to a camera, making the signal recognizing device function as: a signal display candidate detecting unit which detects a green lamp candidate area from an image obtained from the camera; a reference calculating unit which calculates a time-series statistic of unit area values per unit area of the green lamp candidate area in a state where a red lamp is lit, and which stores the statistics on a storage unit; and a green lighting determination unit which determines whether or not a green lamp is lit based on the image including the green lamp candidate area in a state where the red lamp is unlit and the statistics, in case of determining that the red lamp varies from a lit state to an unlit state; wherein the reference calculating unit omits unit area values of unit areas of the green lamp candidate area from samples for calculating the statistics, provided that between the unit area values and either the statistics or unit area values of unit areas of a green lamp candidate area existing in a previously obtained image are larger than a predetermined threshold. By the above program being installed and executed, the signal recognizing device can precisely recognize the green lit state even when the obstruction or the red-green simultaneous lighting occurs. In a preferred example, the above program is stored in a recording medium.

Embodiment

0027Preferred embodiments of the present invention will be explained hereinafter with reference to the drawings. The term “pixel value” herein indicates an all-inclusive term of index values which indicates information that each pixel has, such as brightness, saturation and hue. The term “red lamp part” herein indicates apart emitting red light in a traffic signal and the term “green light part” herein indicates apart emitting green light in a traffic signal.

0028[Schematic Configuration]

0029First, a description will be given of a configuration of the signal recognizing device <b>100</b> according to the embodiment. <figref idref="DRAWINGS">FIG. 1</figref> shows an example of the schematic configuration of the signal recognizing device <b>100</b>. The signal recognizing device <b>100</b> is mounted on a vehicle, and includes a camera <b>11</b>, a vehicle speed sensor, a system controller <b>20</b>, a data storage unit <b>36</b> and an output device <b>50</b>.

0030The camera <b>11</b> has a predetermined angle of view, and is an optical instrument which captures an object in the angle of view. In the embodiment, the camera <b>11</b> is directed toward the front of the vehicle (hereinafter referred to as “equipped vehicle”) equipped with the signal recognizing device <b>100</b>, and is provided on a position where the camera <b>11</b> can capture the traffic signal. The vehicle speed sensor <b>12</b> measures vehicle speed pulses including a pulse signal generated with the wheel rotation of the equipped vehicle.

0031The system controller <b>20</b> includes an interface <b>21</b>, a CPU (Central Processing Unit) <b>22</b>, a ROM (Read Only Memory) <b>23</b> and a RAM (Random Access Memory) <b>24</b>, and controls the entire signal recognizing device <b>100</b>. The system controller <b>20</b> corresponds to the signal display candidate detecting unit, the reference calculating unit, and the green lighting determination unit.

0032The interface <b>21</b> executes the interface operation with the camera <b>11</b> and the system controller <b>20</b>. Then, the interface <b>21</b> is supplied with an image (hereinafter referred to as “image F”) by the camera <b>11</b> at predetermined time intervals, and it inputs the image F into the system controller <b>20</b>. The interface <b>21</b> is supplied with the vehicle speed pulses from the vehicle speed sensor <b>12</b>, and it inputs the vehicle speed pulses into the system controller <b>20</b>.

0033The CPU <b>22</b> controls the entire system controller <b>20</b>. By executing a program prepared in advance, the CPU <b>22</b> executes the signal recognition process described below. The ROM <b>23</b> includes a non-volatile memory, not shown, in which a control program for controlling the system controller <b>20</b> is stored. The RAM <b>24</b> readably stores various kinds of data, and supplies a working area to the CPU <b>22</b>.

0034The system controller <b>20</b>, the data storage unit <b>36</b> and the output device <b>50</b> are connected to each other via a bus line <b>30</b>.

0035The data storage unit <b>36</b> includes HDD, for example, and stores various kinds of data. The output device <b>50</b> is a display or sound output device, for example, and outputs the signal recognition result recognized by the system controller <b>20</b>.

0036[Control Method]

0037Next, the process executed by the system controller <b>20</b> will be described. Summarily, in a state (hereinafter referred to as “red lit state”) where the red lamp part is lit, the system controller <b>20</b> extracts a candidate area (hereinafter, referred to as “green light candidate area Lb”) of the green lamp from the image F, and it calculates the moving average per pixel whose samples are limited only to pixels which meets predetermined conditions. Then, if the system controller <b>20</b> detects a state (hereinafter referred to as “red unlit state”) where the red lamp part is unlit, the system controller <b>20</b> determines whether or not the green lamp part is lit based on the above-mentioned moving average and the green lamp candidate area Lb shown after the red lamp part is unlit. Thereby, the system controller <b>20</b> certainly recognizes the lighting of the green lamp part even when the traffic signal is temporarily obscured or when the simultaneous lighting of the red lamp part and the green lamp is shown in the image F due to the exposure time.

0038The concrete description thereof will be given with reference to <figref idref="DRAWINGS">FIG. 2</figref>. <figref idref="DRAWINGS">FIG. 2</figref> is one example of a functional block of the signal recognizing device <b>100</b> according to the embodiment. As shown in <figref idref="DRAWINGS">FIG. 2</figref>, the system controller <b>20</b> includes a signal display candidate detecting unit <b>201</b>, a reference calculating unit <b>202</b> and a green lighting determination unit <b>203</b>.

0039First, the process executed by the signal display candidate detecting unit <b>201</b> will be described. The signal display candidate detecting unit <b>201</b> extracts the red lamp candidate area Lr and the green lamp candidate area Lb from the image F. The concrete description thereof will be given with reference to <figref idref="DRAWINGS">FIG. 3</figref>.

0040<figref idref="DRAWINGS">FIG. 3A</figref> shows a traffic signal <b>40</b>. From left to right, the traffic signal <b>40</b> includes a green lamp part <b>41</b>B, a yellow lamp part <b>41</b>Y and a red lamp part <b>41</b>R. The symbol “◯” in the figure indicates the center of the red lamp part <b>41</b>R, and the symbol “R” indicates the radius of the red lamp part <b>41</b>R.

0041First, the signal display candidate detecting unit <b>201</b> identifies the red lamp candidate area Lr from the image F. For example, when the vehicle speed of the equipped vehicle is 0, the signal display candidate detecting unit <b>201</b> calculates the brightness, the saturation and the hue of each pixel in the image F. Then, the signal display candidate detecting unit <b>201</b> extracts the pixels existing in the range of the possible values of the brightness, the saturation and the hue at the time when the red lamp part <b>41</b>R is lit. It is noted that the signal display candidate detecting unit <b>201</b> stores the range of the possible values of the brightness, the saturation and the hue at the time when the red lamp part <b>41</b>R is lit in the memory such as the data storage unit <b>36</b> in advance. The signal display candidate detecting unit <b>201</b> calculates the degree of circularity of the areas composed of the extracted pixels, and recognizes the area which has the highest degree of circularity as the red lamp candidate area Lr. The calculation method of the above-mentioned degree of circularity is explained in detail in Japanese Patent Application Laid-open under No. 2005-301519.

0042Next, the signal display candidate detecting unit <b>201</b> recognizes the green lamp candidate area Lb based on the position and the scale of the red lamp candidate area Lr. The concrete example thereof will be described below. Based on the red lamp candidate area Lr, the signal display candidate detecting unit <b>201</b> recognizes the position of the center ◯ and the radius R in the red lamp part <b>41</b>R shown in <figref idref="DRAWINGS">FIG. 3A</figref>. Then, the signal display candidate detecting unit <b>201</b> extracts the green lamp candidate area Lb based on the position of the center ◯ and the radius R. Concretely, as also shown in <figref idref="DRAWINGS">FIG. 3B</figref> to be described below, given that “A” and “B” are predetermined values for example, the signal display candidate detecting unit <b>201</b> sets the horizontal lines passing the positions which are vertically spaced from the center ◯ by “R×A” in the image F as the upper border line “Lbu” and the lower border line “Lbd” of the green lamp candidate area Lb. The signal display candidate detecting unit <b>201</b> sets the vertical line passing the center ◯ as the right border line “Lbr” of the green lamp candidate area Lb, and it sets the vertical line passing the positions which are spaced from the center ◯ by “R×B” in the left direction as the left border line “Lbl”.

0043<figref idref="DRAWINGS">FIG. 3B</figref> shows an image including the traffic signal <b>40</b> and an enlarged view of the green lamp candidate area Lb in the image. As shown in <figref idref="DRAWINGS">FIG. 3B</figref>, the green lamp candidate area Lb is set so that the green lamp candidate area Lb includes the green lamp part <b>41</b>B. In this way, the signal display candidate detecting unit <b>201</b> stores the numbers A and B set by experimental trials in its memory thereby to set the green lamp candidate area Lb including the green lamp part <b>41</b>B. Then, the signal display candidate detecting unit <b>201</b> supplies the reference calculating unit <b>202</b> with each pixel value (hereinafter referred to as “green candidate area pixel value Pb”) of the green lamp candidate area Lb.

0044Hereinafter, as one example, it is assumed that the signal display candidate detecting unit <b>201</b> recognizes the red lamp candidate area Lr and the green lamp candidate area Lb when the equipped vehicle is at a stop. It is also assumed that the signal display candidate detecting unit <b>201</b> keeps the positions, the scales and the ranges of the red lamp candidate area Lr and the green lamp candidate area Lb unchanged in the image F when the equipped vehicle is at a stop, and that the system controller <b>20</b> stops the traffic signal recognition process when the equipped vehicle is running.

0045The signal display candidate detecting unit <b>201</b> detects the red unlit state based on the red lamp candidate area Lr. For example, the signal display candidate detecting unit <b>201</b> determines that the red lamp part <b>41</b>R is unlit in a case where the red lamp candidate area Lr becomes unable to be detected or a case where the brightness of each pixel in the red lamp candidate area Lr varies by equal to or larger than predetermined value. In this case, the signal display candidate detecting unit <b>201</b> supplies the green lighting determination unit <b>203</b> with the green candidate area pixel value Pb in the unlit state of the red lamp part <b>41</b>R.

0046Next, a description will be given of the process executed by the reference calculating unit <b>202</b>. The reference calculating unit <b>202</b> calculates the moving averages of the pixels in the green lamp candidate area Lb while the traffic signal is in the red lit state. Concretely, until detecting the red unlit state, the reference calculating unit <b>202</b> calculates the moving average (hereinafter referred to as “moving average Pbref”) of the green candidate area pixel values Pb per pixel every time the green candidate area pixel value Pb is supplied, and stores the moving average Pbref in the ROM <b>23</b> or the data storage unit <b>36</b> (hereinafter simply referred to as “memory”). Here, the number of samples used for the purpose of calculating the moving average Pbref is set in advance to a value in consideration of the disturbance, for example. For example, the number of the samples may be set so that the samples include all the green candidate area pixel values Pb obtained by the reference calculating unit <b>202</b> in the red lit state. In this way, the reference calculating unit <b>202</b> calculates the moving average Pbref which is insusceptible to the disturbance as the representative pixel value of the green lamp part <b>41</b>B in the unlit state. Thereby, it is possible to improve the accuracy of the traffic signal recognition executed by the green lighting determination unit <b>203</b>.

0047Every time the image F is supplied from the camera <b>11</b>, the reference calculating unit <b>202</b> calculates the difference (hereinafter referred to as “first difference Dif<b>1</b>”) between the green candidate area pixel value Pb in the image F and the moving average Pbref stored in the memory with respect to each corresponding pixel. Then, the reference calculating unit <b>202</b> omits the green candidate area pixel values Pb where each first difference Dif<b>1</b> thereof is larger than a predetermined threshold (hereinafter referred to as “threshold T<b>1</b>”) from the samples for calculating the moving average Pbref. In other words, in this case, without recalculating the moving average Pbref for the above-mentioned pixels, the reference calculating unit <b>202</b> keeps the moving average Pbref which was used for calculating the first difference Dif<b>1</b> stored. The threshold T<b>1</b> is set by experimental trials to a value which is resistant to the influence caused by the obstruction and the red-green simultaneous lighting. Thereby, the reference calculating unit <b>202</b> can precisely calculate the moving average Pbref even if the obstruction and/or the red-green simultaneous lighting described below occur.

0048The description thereof will be given with reference to <figref idref="DRAWINGS">FIGS. 4A and 4B</figref>. <figref idref="DRAWINGS">FIG. 4A</figref> shows the image F in a case where the obstruction occurs and the enlarged view of the green lamp candidate area Lb thereof. In <figref idref="DRAWINGS">FIG. 4A</figref>, a part of the traffic signal <b>40</b> is obscured by the vehicle <b>50</b>. Thereby, a part of the vehicle <b>50</b> is shown in the green lamp candidate area Lb. Even in this case, by setting the threshold T<b>1</b>, the reference calculating unit <b>202</b> can calculate the moving average Pbref without the influence caused by the obstruction by the vehicle <b>50</b>. The description thereof will be given with reference to <figref idref="DRAWINGS">FIG. 4B</figref>.

0049<figref idref="DRAWINGS">FIG. 4B</figref> schematically shows the process executed by the reference calculating unit <b>202</b> according to the embodiment. As shown in <figref idref="DRAWINGS">FIG. 4B</figref>, the reference calculating unit <b>202</b> compares each pixel of “IMAGE IN CASE OBSTRUCTION OCCURS”, i.e., the image F which is supplied from the camera <b>11</b> and which shows the green lamp candidate area Lb at the occurrence time of the obstruction, to each pixel of “IMAGE OF MOVING AVERAGE Pbref BEFORE UPDATED”, i.e., an image composed of the moving averages Pbref. Then, with respect to pixels where each first difference Dif<b>1</b> thereof is equal to or smaller than the threshold T<b>1</b>, i.e., pixels without obstruction of the vehicle <b>50</b>, the reference calculating unit <b>202</b> recalculates the moving averages Pbref by using the pixel values thereof. In contrast, with respect to pixels where each first difference Dif<b>1</b> thereof is larger than the threshold T<b>1</b>, i.e., pixels obstructed by the vehicle <b>50</b>, the reference calculating unit <b>202</b> keeps using the previous moving average Pbref without recalculating the moving average Pbref. The image composed of the moving averages Pbref updated by the process thereof is “IMAGE OF MOVING AVERAGE Pbref AFTER UPDATED” shown in <figref idref="DRAWINGS">FIG. 4B</figref>. In this way, even in the case where the obstruction by the vehicle <b>50</b> occurs, the reference calculating unit <b>202</b> can calculate the moving average Pbref by eliminating the influence thereof.

0050Next, the description will be given of the process executed by the green lighting determination unit <b>203</b> with reference to <figref idref="DRAWINGS">FIG. 2</figref> again. When the signal display candidate detecting unit <b>201</b> detects the red unlit state, the green lighting determination unit <b>203</b> determines whether or not the green lamp part <b>41</b>B is lit based on both the green candidate area pixel value Pb in the image F obtained at the time of the detection or just after the detection and the moving average Pbref stored in the memory.

0051The description thereof will be given with reference <figref idref="DRAWINGS">FIG. 5</figref>. <figref idref="DRAWINGS">FIG. 5</figref> schematically shows the process executed by the green lighting determination unit <b>203</b>. First, as shown in <figref idref="DRAWINGS">FIG. 5</figref>, the green lighting determination unit <b>203</b> calculates the difference (herein after referred to as “second difference Dif<b>2</b>”) between the moving average Pbref stored in the memory and the green candidate area pixel value Pb in the image F obtained at the time of red unlit state. Then, the green lighting determination unit <b>203</b> extracts the pixels where each second difference Dif<b>2</b> thereof is larger than a predetermined threshold (hereinafter referred to as “threshold T<b>2</b>”) from the image F captured in the red unlit state. For example, the threshold T<b>2</b> is set to one half of the difference between the estimated pixel value at the time of the lit state of the green lamp part <b>41</b>B and the estimated pixel value at the time of the unlit state of the green lamp part <b>41</b>B, and it is stored in the memory.

0052Then, based on the shape and a pixel value or pixel values such as hue and brightness, the green lighting determination unit <b>203</b> determines whether or not the area (extracted area) composed of the extracted pixels shows the green lamp part <b>41</b>B at the time of the lit state. The concrete example thereof will be described below. For example, the green lighting determination unit <b>203</b> determines whether or not each pixel in the extracted area has a value in the range of the possible values of the hue, saturation, brightness in the green lamp part <b>41</b>B at the time of the lit state. It is noted that the green lighting determination unit <b>203</b> stores the information of the range of the possible values of the hue, saturation, brightness in the green lamp part <b>41</b>B at the time of the lit state in its memory in advance. The green lighting determination unit <b>203</b> also calculates the degree of circularity of the extracted area to determine whether or not the extracted area shows the green lamp part <b>41</b>B. The detail of the calculation method of the degree of the circularity will not be described since the detailed explanation has already been given in “Japanese Patent Application Laid-open under No. 2005-301519”.

0053[Process Flow]

0054Next, the description will be given of a procedure of the process according to the embodiment. <figref idref="DRAWINGS">FIG. 6</figref> is one example showing the procedure of the process executed by the system controller <b>20</b> according to the embodiment. The system controller <b>20</b> repeatedly executes the process of the flowchart shown in <figref idref="DRAWINGS">FIG. 6</figref> every time the system controller <b>20</b> obtains the image F from the camera <b>11</b>.

0055First, the system controller <b>20</b> detects the red lamp candidate area Lr in the image F (step S<b>101</b>). When the system controller <b>20</b> does not detect the red lamp candidate area Lr, the system controller <b>20</b> keeps trying to detect the red lamp candidate area Lr.

0056Next, the system controller <b>20</b> determines whether or not the red lamp part <b>41</b>R turns into the red unlit state from the red lit state (step S<b>102</b>). When the red lamp part <b>41</b>R turns into the red unlit state from the red lit state (step S<b>102</b>; Yes), the system controller <b>20</b> executes the process at step S<b>108</b> to S<b>110</b>. The description thereof will be given below.

0057In contrast, when the red lamp part <b>41</b>R does not turn into the red unlit state from the red lit state (step S<b>102</b>; No), the system controller <b>20</b> determines whether or not the system controller <b>20</b> newly detected the red lamp candidate area Lr (step S<b>103</b>). Namely, the system controller <b>20</b> determines whether or not it has not yet identified the green lamp candidate area Lb corresponding to the detected red lamp candidate area Lr. In a case where the system controller <b>20</b> newly detected the red lamp candidate area Lr (step S<b>103</b>; Yes), the system controller <b>20</b> determines the green lamp candidate area Lb based on the red lamp candidate area Lr (step S<b>104</b>). Concretely, as illustrated in the context of <figref idref="DRAWINGS">FIG. 3A</figref>, the system controller <b>20</b> recognizes the center ◯ and the radius R and identifies the green lamp candidate area Lb thereby. Then, the system controller <b>20</b> stores the green candidate area pixel values Pb in its memory (step S<b>105</b>). Thereafter, the system controller <b>20</b> uses the green candidate area pixel values Pb as the initial values of the moving average Pbref at the next time the system controller <b>20</b> obtains the image F and executes the process of the flowchart.

0058In contrast, when the system controller <b>20</b> does not newly detect the red lamp candidate area Lr (step S<b>103</b>; No), i.e., it has already detected the red lamp candidate area Lr, the system controller <b>20</b> calculates the first difference Dif<b>1</b> per pixel in the green lamp candidate area Lb based on the moving average Pbref (step S<b>106</b>). Concretely, per pixel in the green lamp candidate area Lb, the system controller <b>20</b> calculates the difference between the moving average Pbref stored in the memory and the green candidate area pixel value Pb in the image F obtained at the start time of the flowchart as the first difference Dif<b>1</b>.

0059Next, for the pixels where each first difference Dif<b>1</b> thereof is equal to or smaller than the threshold T<b>1</b>, the system controller <b>20</b> updates the moving average Pbref, and for the pixels where each first difference Dif<b>1</b> thereof is larger than the threshold T<b>1</b>, the system controller <b>20</b> keeps using the past moving average Pbref (step S<b>107</b>). In other words, with respect to the pixels each first difference Dif<b>1</b> of which is equal to or smaller than the threshold T<b>1</b>, the system controller <b>20</b> recalculates the moving average Pbref by adding the green candidate area pixel value Pb in the image F obtained at the start time of the flowchart. In contrast, with respect to the pixels each first difference Dif<b>1</b> of which is larger than the threshold T<b>1</b>, the system controller <b>20</b> does not use the green candidate area pixel value Pb for calculating the moving average Pbref. Thereby, even in the case where the traffic signal <b>40</b> is obstructed and/or the red-green simultaneous lighting occurs in the image F, the system controller <b>20</b> can calculate the moving average Pbref without the influence thereof.

0060In contrast, at step S<b>102</b>, when the system controller <b>20</b> determines that the traffic signal <b>40</b> turns into the red lit state to the red unlit state (step S<b>102</b>; Yes), the system controller <b>20</b> calculates the second difference Dif<b>2</b> per pixel in the green lamp candidate area Lb on the basis of the moving average Pbref (step S<b>108</b>). Concretely, the system controller <b>20</b> calculates the difference between the moving average Pbref stored in the memory and the green candidate area pixel value Pb in the image F obtained at the start time of the flowchart and sets it to the second difference Dif<b>2</b>.

0061Next, the system controller <b>20</b> identifies the pixels where each second difference Dif<b>2</b> thereof is larger than the threshold T<b>2</b> (step S<b>109</b>). Based thereon, the system controller <b>20</b> makes the determination of the green lit state (step S<b>110</b>). Concretely, based on the shape formed by the extracted pixels and the pixel value (s) thereof such as hue and brightness, the system controller <b>20</b> determines whether or not the green lamp part <b>41</b>B is lit.

0062As described above, the above signal recognizing device is mounted on a vehicle, and includes a camera, a storage unit, a signal display candidate detecting unit, a reference calculating unit and a green lighting determination unit. The signal display candidate detecting unit detects a red lamp candidate area and a green lamp candidate area of the traffic signal from an image obtained from the camera. The reference calculating unit calculates a moving average of a pixel value of each pixel existing in the green lamp candidate area, and stores the moving averages in the storage unit. At that time, the reference calculating unit omits unit area values of the green lamp candidate area from the samples for calculating the moving averages, provided that between the pixel values and the moving averages stored in the storage unit are larger than a predetermined threshold. In other words, for the unit areas having the above-mentioned each difference equal to or smaller than the threshold, the reference calculating unit updates the moving averages by using the unit area values thereof, and for the unit areas having the above-mentioned each difference larger than the threshold, it keeps using the moving averages thereof without updating them. In case of determining that the red lamp turns into the unlit state from the lit state, the green lighting determination unit determines whether or not a green lamp is lit on the basis of the image captured at the time of the unlit state of the red lamp and the moving averages stored in the storage unit. Thereby, the signal recognizing device can precisely recognize the green lit state even when the obstruction or the red-green simultaneous lighting occurs.

0063[Modification]

0064Each modification of the embodiment will be described below. Each modification can be applied to the above-mentioned embodiment in combination.

0065(First Modification)

0066In the explanation with respect to <figref idref="DRAWINGS">FIG. 1</figref>, the signal recognizing device <b>100</b> includes the camera <b>11</b>. The configuration to which the present invention can be applied, however, is not limited to the configuration. Instead of the above-mentioned configuration, the signal recognizing device <b>100</b> and the camera <b>11</b> may be separate devices from each other. In this case, for example, the signal recognizing device <b>100</b> is electromagnetically connected to the camera <b>11</b> by wired or wireless connection, and is supplied with the image F from the camera <b>11</b> at predetermined intervals.

0067(Second Modification)

0068In the explanation with respect to <figref idref="DRAWINGS">FIG. 3</figref>, the traffic signal <b>40</b> has the format (hereinafter referred to as “horizontally-lined traffic signal format”) in which the green lamp part <b>41</b>B, the yellow lamp part <b>41</b>Y and the red lamp part <b>41</b>R are lined horizontally to the ground. In addition to this format, the present invention can also be applied to a format (hereinafter referred to as “vertically-lined traffic signal format”) in which each display unit is lined vertically to the ground.

0069In case of the vertically-lined traffic signal format, the signal display candidate detecting unit <b>201</b> sets the vertical lines passing the positions spaced from the center ◯ in the right/left direction by “R×A” in the image F to the left border line Lbl and the right border line Lbr, for example. The signal display candidate detecting unit <b>201</b> also sets the horizontal lines passing the center ◯ to the upper border line Lbu of the green lamp candidate area Lb, and sets the horizontal lines passing the positions spaced from the center ◯ in the downward by “R×B” to the lower border line Lbd.

0070Even in case of the vertically-lined traffic signal format, by determining the green lamp candidate area Lb as described above, the signal recognizing device <b>100</b> can calculate the moving averages Pbref based on the green lamp candidate area Lb to recognize the green lit state similarly to the above-mentioned embodiment.

0071(Third Modification)

0072In the explanation of the reference calculating unit <b>202</b> in <figref idref="DRAWINGS">FIG. 2</figref>, the moving average Pbref is calculated based on the one kind of the pixel value. A method to which the present invention can be applied, however, is not limited to the method. Instead of the above-mentioned method, the reference calculating unit <b>202</b> may calculate the moving average Pbref based on at least two kinds of the pixel value.

0073Here, the concrete example will be given in a case where brightness and hue are used as the pixel values in the embodiment. In this case, the reference calculating unit <b>202</b> stores thresholds T<b>1</b> each suited to the brightness and the hue in advance. Hereinafter, the threshold T<b>1</b> for the brightness is especially referred to as “threshold T<b>1</b>L”, and the threshold T<b>1</b> for the hue is especially referred to as “threshold T<b>1</b>C”. Then, the reference calculating unit <b>202</b> calculates each first difference Dif<b>1</b> with respect to the brightness and the hue, respectively. Hereinafter, the first difference Dif<b>1</b> for the brightness is especially referred to as “first difference Dif<b>1</b>L”, and the first difference Dif<b>1</b> for the hue is especially referred to as “first difference Dif<b>1</b>C”. Next, for pixels where each first difference Dif<b>1</b>L thereof does not exceed the threshold T<b>1</b>L and where each first difference Dif<b>1</b>C thereof does not exceed the threshold TIC, the reference calculating unit <b>202</b> updates each moving average Pbref of the brightness and the hue. In contrast, for the other pixels, the reference calculating unit <b>202</b> does not update each moving average Pbref for the brightness and the hue. In another example, for pixels where each first difference Dif<b>1</b>L thereof does not exceed the threshold T<b>1</b>L or pixels where each first difference Dif<b>1</b>C thereof does not exceed the threshold T<b>1</b>C, the reference calculating unit <b>202</b> updates each moving average Pbref for the brightness and the hue, and for the other pixels, the reference calculating unit <b>202</b> does not updates each moving average Pbref.

0074In the above-mentioned cases, the green lighting determination unit <b>203</b> stores the thresholds T<b>2</b> each suited to the brightness and the hue in advance. Then, the green lighting determination unit <b>203</b> calculates each second difference Dif<b>2</b> based on each moving average Pbref for the brightness and the hue. Next, for example, the green lighting determination unit <b>203</b> identifies pixels where each second difference Dif<b>2</b> thereof for the brightness and the hue is larger than the threshold T<b>2</b>, respectively.

0075As described above, even in case of calculating the moving averages Pbref of multiple kinds of pixel values, the system controller <b>20</b> can calculate the moving averages Pbref without influence caused by the obstruction of the traffic signal <b>40</b> or the red-green simultaneous lighting in the image F.

0076[Fourth Modification]

0077In addition to the process according to the embodiment, the system controller <b>20</b> may store default values of the moving averages Pbref in its memory in advance. In this case, each default value is set by experimental trials to a typical value of the green candidate area pixel value Pb. In addition, in this case, the system controller <b>20</b> may consider that the green lamp part <b>41</b>B is not included in the green lamp candidate area Lb if the moving averages Pbref of all the pixels have never been updated from the default values in a predetermined time period after the recognition of the green lamp candidate area Lb. Namely, in this case, the system controller <b>20</b> considers that it misidentifies the object other than a traffic signal as the traffic signal. In addition, the system controller <b>20</b> may also consider that it misidentifies the traffic signal if the number of pixels where each moving average Pbref thereof has ever been updated from the default value is at most a predetermined number.

0078(Fifth Modification)

0079According to the embodiment, at the time of calculating the moving averages Pbref, the reference calculating unit <b>202</b> does not use the green candidate area pixel values Pb where each first difference Dif<b>1</b> is larger than the threshold T<b>1</b>. In addition to this, regardless of the first difference Dif<b>1</b>, the reference calculating unit <b>202</b> may not use pixels each of which does not belong to a predetermined range of the brightness and/or a predetermined range of the hue in the green lamp candidate area Lb at the time of calculating the moving average Pbref. The above-mentioned ranges are set by experimental trials to possible ranges of the brightness and the hue of the green lamp part <b>41</b>B at the unlit state. Thereby, the reference calculating unit <b>202</b> can more precisely calculate the moving averages Pbref which is representative values of the green lamp part <b>41</b>B at the unlit state.

0080(Sixth Modification)

0081A concrete description will be given of the process of setting the threshold T<b>1</b> in case of using the brightness as the pixel value. Here, a description will be given of the process of setting the threshold T<b>1</b> in such a way that the moving average Pbref can be precisely calculated even in the case where the red-green simultaneous lighting is shown in the image F.

0082In this case, the threshold T<b>1</b> is set on the basis of the brightness of the red lamp part <b>41</b>R. Concretely, given that the symbol “RLt” stands for brightness of the red lamp part <b>41</b>R at the lit state and the symbol “RLs” stands for brightness of the red lamp part <b>41</b>R at the unlit state, the threshold T<b>1</b> is determined according to the following equation (1).

0000<br /><i>T</i>1=(<i>RLt−RLs</i>)/2   (1)

0083Additionally, assuming the brightness RLs of the red lamp part <b>41</b>R at the unlit state is “0”, the threshold T<b>1</b> is determined according to the equation (2).

0000<br /><i>T</i>1<i>=RLt/</i>2   (2)

0084Here, for example, an estimate value of the brightness RLs according to the equation (1) is determined by experimental trials in advance, and stored in the memory. In addition, the system controller <b>20</b> may determine the brightness RLt based on brightness of pixels in the red lamp candidate area Lr. Concretely, the system controller <b>20</b> may set a representative brightness of the area such as the average brightness of the area to the brightness RLt. In this case, the threshold T<b>1</b> is a variable value which varies depending on each target traffic signal <b>40</b> or each target image F.

0085A supplemental explanation will be given of the effect according to the sixth modification. Generally, brightness of the red lamp part <b>41</b>R at a lit/unlit state is estimated to be almost identical to brightness of the green lamp part <b>41</b>B at a lit/unlit state, respectively. Thus, by determining the threshold T<b>1</b> based on the brightness RLt and RLs, the system controller <b>20</b> can omit pixels in the green lamp candidate area Lb at the lit state from the samples for calculating the moving average Pbref even in the case where the red-green simultaneous lighting is shown in the image F. In other words, the reference calculating unit <b>202</b> can more precisely calculate the moving average Pbref which is a representative pixel value of the green lamp part <b>41</b>B at the unlit state.

0086(Seventh Modification)

0087A description will be given of an actual process of setting the threshold T<b>2</b> in case of using brightness as the pixel value. In this case, the system controller <b>20</b> may determine the threshold T<b>2</b> based on the brightness RLt of the red lamp part <b>41</b>R at the lit state.

0088The concrete description thereof will be given below. The system controller <b>20</b> determines the threshold T<b>2</b> based on the following equation (3).

0000<br /><i>T</i>2=(<i>RLt−RLs</i>)/2   (3)

0089Here, the system controller <b>20</b> determines the brightness RLt based on the brightness of each pixel in the red lamp candidate area Lr. Actually, the system controller <b>20</b> sets a representative value of the brightness in the area such as the average to the brightness RLt. The system controller <b>20</b> stores an estimate value of the brightness RLs in its memory in advance.

0090As described above, by dynamically determining the threshold T<b>2</b> based on the brightness RLt of the red lamp part <b>41</b>R at the lit state, the system controller <b>20</b> can properly set the threshold T<b>2</b>.

0091(Eighth Modification)

0092The reference calculating unit <b>202</b> omits pixels where each first difference Dif<b>1</b> thereof is larger than the threshold T<b>1</b> from the calculation of the moving average Pbref. In addition, on detecting the red unlit sate, the reference calculating unit <b>202</b> may omit the green candidate area pixel values Pb in a predetermined number of images F obtained just before becoming the red unlit state from the samples for calculating the moving average Pbref. The above-mentioned predetermined number is set to an estimate value of the number of the image where the red-green simultaneous lighting could occur.

0093In this case, the reference calculating unit <b>202</b> stores not only the moving averages Pbref according to the embodiment in its memory but also the moving averages Pbref calculated before the acquisition of the predetermined number of image F. When the red lamp turns into the unlit state from the lit state, the green lighting determination unit <b>203</b> executes various kinds of process according to the embodiment by using the latter moving average Pbref.

0094Thereby, the reference calculating unit <b>202</b> can omit the pixels in the green lamp candidate area Lb at the lit state from the samples for calculating the moving average Pbref even in the case where the red-green simultaneous lighting is shown in the image F. In other words, the reference calculating unit <b>202</b> can more precisely calculate the moving average Pbref which is a representative pixel value of the green lamp part <b>41</b>B at the unlit state.

0095(Ninth Modification)

0096The reference calculating unit <b>202</b> may change the threshold T<b>1</b> based on the vehicle speed of the equipped vehicle. For example, the higher the vehicle speed is, the more the reference calculating unit <b>202</b> increases the threshold T<b>1</b> assuming that the variation of the traffic signal <b>40</b> in the image F becomes greater. Namely, when the vehicle speed is high, the reference calculating unit <b>202</b> considers that the variation of the traffic signal <b>40</b> in the image F is great, and changes the threshold T<b>1</b> to a more permissive value. In this case, the reference calculating unit <b>202</b> stores a map or an equation which associates each vehicle speed with the compatible threshold T<b>1</b> in its memory, and the sets the threshold T<b>1</b> based on the vehicle speed with reference to the above-mentioned map or the equation. Thereby, the signal recognizing device <b>100</b> can set the threshold T<b>1</b> according to the state of the equipped vehicle even if the equipped vehicle is moving.

0097(Tenth Modification)

0098In the explanation of <figref idref="DRAWINGS">FIG. 2</figref>, after determining the red lamp candidate area Lr and the green lamp candidate area Lb, the system controller <b>20</b> fixes positions, ranges and scales thereof. A method to which the present invention can be applied, however, is not limited to the method. Instead of this, the system controller <b>20</b> may determine the red lamp candidate area Lr and the green lamp candidate area Lb every time it obtains the image F.

0099Even in this case, the system controller <b>20</b> does not minify the range of the green lamp candidate area Lb. In other words, the system controller <b>20</b> determines the green lamp candidate area Lb so that the green lamp candidate area Lb includes the past green lamp candidate area Lb determined after the detection of the red lit state.

0100The description thereof will be given with reference to <figref idref="DRAWINGS">FIG. 7</figref>. <figref idref="DRAWINGS">FIG. 7</figref> schematically shows the process executed by the green lighting determination unit <b>203</b>. It is noted that <figref idref="DRAWINGS">FIG. 7</figref> shows a case (hereinafter referred to as “comparison example”) where the signal display candidate detecting unit <b>201</b> accordingly minifies the range of the green lamp candidate area Lb according to the red lamp candidate area Lr instead of the embodiment. Additionally, the moving average Pbref is calculated per pixel in the common green lamp candidate area Lb thereof at the red lit state.

0101In this case, the signal display candidate detecting unit <b>201</b> detects nothing more than a part of the red lamp part <b>41</b>R as the red lamp candidate area Lr. As a result, the target area of the moving averages Pbref does not include the whole area of the green lamp part <b>41</b>B. Thus, in this case, since the green lighting determination unit <b>203</b> can detect only the part of the green lamp part <b>41</b>B, it is possible that the green lit state cannot be properly recognized.

0102In consideration of the above-mentioned fact, without minifying the range of the green lamp candidate area Lb, the system controller <b>20</b> determines a new green lamp candidate area Lb so that the new green lamp candidate area Lb includes the past green lamp candidate area Lb in order to calculate the moving average Pbref with respect to whole range of the green lamp candidate area Lb. Thereby, the system controller <b>20</b> can more improve the recognition accuracy of the green lit state.

0103(Eleventh Modification)

0104In the embodiment, the reference calculating unit <b>202</b> calculates the moving average Pbref of the time-series candidate area pixel values Pb. A method to which the present invention can be applied, however, is not limited to the method. Instead of this, the reference calculating unit <b>202</b> may use a value (hereinafter referred to as “statistic”) obtained by statistical processing of the time-series green candidate area pixel values Pb other than the moving average Pbref. In this case, for example, according to a predetermined protocol, the reference calculating unit <b>202</b> executes the averaging procedure based on all or a part of the green candidate area pixel values Pb obtained in the past thereby to calculate the statistic.

0105(Twelfth Modification)

0106In the explanation of the embodiment, the reference calculating unit <b>202</b> calculates the moving average Pbref per green candidate area pixel value Pb, i.e., the pixel value of each pixel in the green lamp candidate area Lb, and based on the threshold T<b>1</b>, it determines whether or not it should use the green candidate area pixel value Pb as the samples for calculating the moving average Pbref. A method to which the present invention can be applied, however, is not limited to the method. Instead of this, the reference calculating unit <b>202</b> may calculate the green candidate area pixel value Pb per unit area (hereinafter simply referred to as “unit area”) which has a number of adjacent pixels to each other and which is determined on the basis of a predetermined protocol, and determine whether or not it should use the unit area for calculating the moving average Pbref on the basis of the threshold T<b>1</b>. In this case, the reference calculating unit <b>202</b> calculates a representative value (hereinafter referred to as “unit area value”) of each pixel in the unit area such as the average of the pixel values thereof, and it calculates the moving average Pbref based on the unit area value.

0107Thus, according to the twelfth modification, for example, when the unit area is set to the green lamp candidate area Lb, the reference calculating unit <b>202</b> calculates a moving average Pbref per green lamp candidate area Lb in order to determine whether or not the unit area should be used for calculating the moving average Pbref.

0108In case of the twelfth modification, the green lighting determination unit <b>203</b> calculates the second difference Dif<b>2</b> per unit area, for example. Namely, in this case, the green lighting determination unit <b>203</b> calculates a unit area value of each unit area of the green lamp candidate area Lb in the image F at the red unlit state, and it calculates the second difference Dif<b>2</b> per unit area by comparing the unit area value to the moving average Pbref calculated per unit area. In another example, the green lighting determination unit <b>203</b> calculates the second difference Dif<b>2</b> per pixel by comparing a pixel value of each pixel in the green lamp candidate area Lb in the image F at the red unlit state to the moving average Pbref of the unit area corresponding to the pixel.

0109(Thirteenth Modification)

0110In the explanation of the embodiment, every time the image F is supplied from the camera <b>11</b>, the reference calculating unit <b>202</b> calculates each first difference Dif<b>1</b> i.e., a difference between each green candidate area pixel value Pb in the image F and the moving average Pbref. A method to which the present invention can be applied, however, is not limited to the method.

0111instead of this, or in addition to this, every time the image F is supplied from the camera <b>11</b>, the reference calculating unit <b>202</b> may calculate each difference between each green candidate area pixel value Pb in the image F and the corresponding green candidate area pixel value Pb in the image F (hereinafter referred to as “past obtained image Fd”) obtained in the past as the first difference Dif<b>1</b>. The past obtained image Fd may be the image F obtained at the last minute or may be the image F obtained before a predetermined time. Even in this case, the reference calculating unit <b>202</b> omits a green candidate area pixel value Pb of a pixel where a first difference Diff calculated by use of the past obtained image Fd is larger than the threshold T<b>1</b> from the samples for calculating the above-mentioned moving average Pbref. Even in this way, the reference calculating unit <b>202</b> can precisely calculate the moving average Pbref even in the case where the obstruction or the red-green simultaneous lighting occurs.

INDUSTRIAL APPLICABILITY

0112This invention can be applied to a multipurpose device mounted on a vehicle, a navigation device and other kinds of devices which recognizes a traffic signal and which is mounted on a moving body. The term “multipurpose device” herein indicates a machine or a device which autonomously communicates with users and captures the scenery in the outside of the vehicle. The multipurpose device may also have a function to cooperate with a navigation device and a function to reproduce contents such as music and images if necessary.

BRIEF DESCRIPTION OF REFERENCE NUMBERS

0113<b>11</b> Camera

0114<b>12</b> Vehicle speed sensor

0115<b>20</b> System controller

0116<b>22</b> CPU

0117<b>36</b> Data storage unit

0118<b>40</b> Traffic signal

0119<b>50</b> Output device

0120<b>100</b> Signal recognizing device

0121<b>201</b> Signal display candidate detecting unit

0122<b>202</b> Reference calculating unit

0123<b>203</b> Green lighting determination unit