Image processing device, learning device, inference device and image processing method

The present invention relates to an image processing device, a learning device, an inference device, and an image processing method.

It is known that using neural networks to recognize the presence or absence of abnormalities in visual inspections using photographed images of manufactured products in manufacturing plants, and in medical diagnoses using medical images such as X-ray fluoroscopic images and CT images, generally provides higher recognition accuracy than other image processing methods. In addition, in such visual inspections and medical diagnoses, an abnormality is often only a small part of the entire recognition target image. For this reason, a technique is known in which the recognition target image is divided into multiple processing images and each of the divided multiple processing images is individually processed by a neural network. By using this technique, the processing amount of each neural network can be reduced compared to the processing amount when the recognition target image is processed as is.

By the way, in the learning of the neural network in the above technology, it is ideal to use a method of teaching the presence or absence of anomalies in each of the multiple processed images as a correct answer value. However, this method has a problem in that it takes more time to create correct answer data in proportion to the number of processed images to be divided than teaching the presence or absence of anomalies in the recognition target image. To address this problem, a technology is known in which the maximum value of each output obtained by processing the multiple processed images individually with a neural network is calculated, and the presence or absence of anomalies in the recognition target image before division is taught as a correct answer value for the estimated value of the presence or absence of anomalies obtained from this maximum value, thereby learning the individual neural network.

In the neural network learning process used in the above technology, each of a plurality of processed images is processed by the neural network, and all processing process data such as pixel values of the converted image in each processing process and weight parameters of the neural network at that time are stored in memory.Then, using the error of the estimated value obtained as the output of the neural network with respect to the correct value and the processing process data that contributes to learning, the weight parameters of the neural network are updated by backpropagation processing, and all processing process data is released.

However, in the above configuration, since it is necessary to hold all process data until the backpropagation process is performed, there is a problem that the memory capacity cannot be reduced.

<p><nplcit><text>Maximilian Ilse and 2 others, Attention-based Deep Multiple Instance Learning, Volume 80: International Conference on Machine Learning, (Sweden), 2018, PMLR 80: 2127-2136.</text></nplcit></p>

<p>An object of the present invention is to provide an image processing device, a learning device, an inference device, and a method that can reduce the memory capacity required for image processing using a neural network.</p>

<p>An image processing device according to an embodiment includes a feature extraction unit, a memory, a maximum feature selection unit, and an optimization unit. The feature extraction unit generates N features by performing feature extraction processing using a neural network for N (N3) processed images based on an input image. The memory holds processing process data generated in the process of the feature extraction processing. The maximum feature selection unit selects the maximum feature by performing two or more comparisons with M combinations of 2 to N-1 of the N features. After each of the two or more comparisons, the optimization unit releases M-1 or less processing process data corresponding to the M-1 or less features that were not selected from the memory.</p>

<figref num="1">FIG. 1 is a block diagram illustrating the configuration of a learning device including an image processing device according to a first embodiment.</figref><figref num="2">FIG. 13 is an explanatory diagram showing an example of dividing an input image into three processed images by cropping.</figref><figref num="3">FIG. 2 is a block diagram illustrating a detailed configuration of the image processing device in FIG.</figref><figref num="4">FIG. 13 is an explanatory diagram showing an example of dividing an input image into four processed images by cropping.</figref><figref num="5">FIG. 13 is an explanatory diagram showing an example of dividing an input image into four processed images by overlappingly cutting out the input image;</figref><figref num="6">4 is a block diagram showing a first alternative example of the configuration of the feature amount extraction unit and the maximum feature amount selection unit in the image processing device of FIG. 3; </figref><figref num="7">4 is a flowchart illustrating the operation of the image processing device according to the first embodiment. </figref><figref num="8">4 is a block diagram showing a second alternative example of the configuration of the feature amount extraction unit and the maximum feature amount selection unit in the image processing device of FIG. 3. </figref><figref num="9">FIG. 13 is an explanatory diagram showing an example of generating two processed images by reducing an input image.</figref><figref num="10">FIG. 2 is an explanatory diagram illustrating a transformed image, an intermediate image, and a receptive field in two convolution processes on an input image.</figref><figref num="11">FIG. 2 is a block diagram showing another example of the configuration of the image processing device in FIG. 1. </figref><figref num="12">9 is a flowchart illustrating another operation of the image processing device according to the first embodiment. </figref><figref num="13">7 is a block diagram showing a third example of another configuration of the feature amount extraction unit and the maximum feature amount selection unit in the image processing device of FIG. 6. </figref><figref num="14">FIG. 1 is an explanatory diagram illustrating multiple transformed images, intermediate images, and receptive fields in a convolution process for an input image.</figref><figref num="15">Intermediate images divided into processing units for convolution processing.</figref><figref num="16">11 is a diagram illustrating an example of a relationship between an intermediate image having a plurality of channels and feature amounts for each channel;</figref><figref num="17">FIG. 11 is a block diagram illustrating the configuration of an inference device including an image processing device according to a second embodiment.</figref><figref num="18">11 is an explanatory diagram illustrating an example of the relationship between a partial image that has undergone convolution processing and partial image data stored in a memory;</figref><figref num="19">11 is an explanatory diagram illustrating partial image data released from a memory.</figref><figref num="20">FIG. 11 is an explanatory diagram illustrating new partial image data stored in a memory. </figref><figref num="21">Interpolated intermediate images generated from subimages.</figref><figref num="22">FIG. 1 is a block diagram illustrating a hardware configuration of a computer according to an embodiment.</figref>

Hereinafter, embodiments of a learning device and an inference device including an image processing device will be described in detail with reference to the drawings.

(First embodiment) In the first embodiment, learning of a neural network that recognizes whether or not an image contains a recognition target object is described. The above object is assumed to be, for example, a crack or stain on a manufactured product in a visual inspection, or a tumor or a blood vessel with internal bleeding in a medical diagnosis.

1 is a block diagram illustrating the configuration of a learning device 100 including an image processing device 110 according to the first embodiment. The learning device 100 includes an image processing device 110 (image processing unit), an error calculation unit 120, and a learning unit 130. The image processing device 110 includes a feature extraction unit 111, a memory 112, a maximum feature selection unit 113, and an optimization unit 114.

The learning device 100 may include an acquisition unit that acquires a learning data set that is a set of an input image required for learning the neural network and a correct label (correct value) corresponding to the input image. The learning device 100 may also include a control unit that controls each unit.

The feature extraction unit 111 receives an input image from another device (not shown). The feature extraction unit 111 generates N features by performing feature extraction processing using a neural network on N (N3) processed images based on the input image. The feature extraction unit 111 outputs processing process data generated in the process of the feature extraction processing to the memory 112, and outputs the N features to the maximum feature selection unit 113.

Specifically, the feature extraction unit 111 sequentially performs feature extraction processing on each of the N processed images. That is, after the feature extraction unit 111 finishes processing on a first processed image, it performs processing on the subsequent second processed image, and repeats this up to the Nth processed image. Moreover, the feature extraction unit 111 outputs processing process data to the memory 112 and outputs the feature to the maximum feature selection unit 113 every time feature extraction processing is performed on a processed image.

The feature extraction process includes a convolution process, an activation process, a full connection process, and a pooling process. Specifically, the feature extraction unit 111 performs a conversion such as a convolution process and an activation process on the processed image, and then converts the processed image into a scalar value by a full connection process or a pooling process to generate a feature. In other words, the feature extraction unit 111 is configured by a neural network that outputs a feature when a processed image is input. The feature extraction unit 111 has N neural networks corresponding to the N processed images, respectively. Each of the N neural networks may be called an individual neural network.

In addition, the feature extraction process may perform a final activation process immediately before generating features. This final activation process is a process of converting values from "0" to "1" by applying a sigmoid function, for example. If the final activation process is not performed in the feature extraction process, the final activation process is performed in an arbitrary unit after the maximum feature is selected in the maximum feature selection unit 113 (described later) and before the error calculation process is performed in the error calculation unit 120. In addition, an activation unit that performs the final activation process may be provided as an arbitrary unit between the maximum feature selection unit 113 and the error calculation unit 120.

The above-mentioned processing data is, for example, data values such as pixel values of the processed image after conversion (converted image or intermediate image), values of weight parameters set for the processing at the time of conversion, and values of shift parameters set for the processing at the time of conversion. This processing data is used during learning of the neural network described later, and therefore may be rephrased as data necessary for learning.

Furthermore, the feature extraction unit 111 may generate N processed images based on the input image. For example, the feature extraction unit 111 generates N processed images by cutting out a part of the input image. The relationship between the input image and the N processed images will be described with reference to FIG. 2.

2 is an explanatory diagram showing an example of dividing an input image 200 into three processed images 210 to 230 by cutting the input image 200. The feature extraction unit 111 generates the three processed images 210 to 230 by cutting the input image 200 into images of a predetermined size. The three processed images 210 to 230 may be of the same size or different sizes.

The process of cutting out the input image may be rephrased as a process of selecting specific regions from the input image. That is, the feature extraction unit 111 generates N processed images by selecting different regions from the input image.

The memory 112 receives the processing process data from the feature extraction unit 111 and stores it. The memory 112 also receives release instruction information from the optimization unit 114. The memory 112 releases unnecessary processing process data (hereinafter referred to as unnecessary data) from among the multiple processing process data stored in accordance with the release instruction information. After a series of processes related to the processed image in each of the other units are completed, the memory 112 outputs the final stored processing process data, i.e., the processing process data related to the maximum feature amount, to the learning unit 130.

Specifically, the memory 112 holds the processing process data sequentially input from the feature extraction unit 111, while releasing the unnecessary data in accordance with the release instruction information sequentially input from the optimization unit 114. With this operation, the memory 112 always releases the unnecessary data, and therefore does not need to hold all the processing process data.

The unnecessary data is processing data corresponding to features that are not selected in the selection process described below. Since the features that are not selected are not taken into consideration when training the neural network, the unnecessary data can be said to be processing data that does not contribute to training the neural network.

The maximum feature selection unit 113 inputs N features from the feature extraction unit 111. The maximum feature selection unit 113 selects the maximum feature by performing two or more comparisons for M combinations, which is 2 to N-1 of the N features. The maximum feature selection unit 113 generates non-selection information regarding features not selected by the selection process and outputs it to the optimization unit 114, and outputs the maximum feature to the error calculation unit 120.

Specifically, maximum feature selection unit 113 inputs features sequentially from feature extraction unit 111, and after the number of input features reaches the number required for selection processing (the above M), performs selection processing and selects the largest feature. Thereafter, maximum feature selection unit 113 again inputs features sequentially from feature extraction unit 111, and after the number of input features reaches the number required for selection processing again, performs selection processing, and thereafter repeats this process.

The feature amount selected by the selection process is used again in the subsequent selection process if there is a subsequent selection process, and is output as a maximum feature amount to the error calculation unit 120 if there is no subsequent selection process. In addition, the maximum feature amount selection unit 113 generates non-selection information and outputs it to the optimization unit 114 every time a selection process is performed.

Note that the two or more comparisons may include comparisons with different numbers of combinations, in which case the maximum feature quantity selection unit 113 performs the comparisons. For example, when performing two comparisons in total, the maximum feature quantity selection unit 113 may compare three feature quantities in the first comparison and compare two feature quantities in the subsequent second comparison.

The optimization unit 114 receives non-selection information from the maximum feature selection unit 113. The optimization unit 114 generates release instruction information based on the non-selection information and outputs it to the memory 112. The release instruction information is information for releasing unnecessary data held in the memory 112. In other words, the optimization unit 114 releases, from the memory, M-1 or less pieces of processing process data corresponding to the M-1 or less features that were not selected for each of two or more comparisons in the maximum feature selection unit 113 (i.e., each comparison for selecting the maximum feature).

The error calculation unit 120 inputs the maximum feature amount from the maximum feature amount selection unit 113, and inputs a correct answer value (correct answer feature amount) corresponding to the input image from another device. The error calculation unit 120 calculates an error value based on the maximum feature amount and the correct answer feature amount. The error calculation unit 120 outputs the error value to the learning unit 130.

Specifically, the error calculation unit 120 compares the maximum feature amount with the correct feature amount, and calculates an error value represented by binary cross entropy, etc. The correct feature amount is, for example, a value of "1" if the input image includes the object to be recognized, and a value of "0" if it does not include the object.

The learning unit 130 inputs the processing process data related to the maximum feature amount from the memory 112, and inputs the error value from the error calculation unit 120. The learning unit 130 trains the neural network constituting the feature amount extraction unit 111 based on the processing process data related to the maximum feature amount and the error value.

Specifically, the learning unit 130 uses the processing process data and error values related to the maximum feature amount to learn the individual neural network from which the maximum feature amount was extracted by the backpropagation method. This backpropagation method of learning is performed by tracing backward the data connections in the forward processing from when the input image is input to when the maximum feature amount is obtained, and sequentially updating the values of the weight parameters and shift parameters set for various processes in the individual neural network. For this reason, the individual neural networks corresponding to the unselected feature amount are not the learning target because the data connections are interrupted midway. In other words, the processing process data related to the unselected feature amount does not contribute to the learning of the individual neural network from which the maximum feature amount was extracted.

The configuration of the learning device 100 including the image processing device 110 according to the first embodiment has been described above. Next, a detailed configuration of the image processing device 110 will be described with reference to Fig. 3. Note that it is assumed that the image processing device 110 in Fig. 3 uses three processed images as shown in Fig. 2.

Fig. 3 is a block diagram illustrating a detailed configuration of the image processing device 110. The feature extraction unit 111 in Fig. 3 includes a processed image generation unit 310, a first extraction unit 320-1, a second extraction unit 320-2, and a third extraction unit 320-3. The maximum feature selection unit 113 in Fig. 3 includes a first selection unit 330-1 and a second selection unit 330-2.

The processed image generating unit 310 generates three processed images based on the input image. The processed image generating unit 310 outputs the first processed image of the three processed images to the first extraction unit 320-1, the second processed image to the second extraction unit 320-2, and the third processed image to the third extraction unit 320-3.

The first extraction unit 320-1 inputs the first processed image from the processed image generation unit 310. The first extraction unit 320-1 generates a first feature by performing a first extraction process corresponding to a feature extraction process on the first processed image. The first extraction unit 320-1 outputs the first feature to the first selection unit 330-1, and outputs first processing process data generated in the process of the first extraction process to the memory 112.

After the first feature amount is extracted, the first processing process data is input from the first extraction unit 320-1 and stored in the memory 112. At this point, the memory 112 stores one piece of processing process data.

The second extraction unit 320-2 inputs the second processed image from the processed image generation unit 310. The second extraction unit 320-2 generates a second feature by performing a second extraction process corresponding to the feature extraction process on the second processed image. The second extraction unit 320-2 outputs the second feature to the first selection unit 330-1, and outputs second processing data generated in the process of the second extraction process to the memory 112.

After the second feature amount is extracted, the memory 112 receives the second processing process data from the second extraction unit 320-2 and stores the data. At this point, the memory 112 stores two pieces of processing process data.

The first selection unit 330-1 receives the first feature from the first extraction unit 320-1 and receives the second feature from the second extraction unit 320-2. The first selection unit 330-1 compares the first feature with the second feature and selects the larger one as the first selected feature. The first selection unit 330-1 generates first non-selection information regarding the feature that was not selected and outputs it to the optimization unit 114, and outputs the first selected feature to the second selection unit 330-2.

After the first non-selection information is generated, the optimization unit 114 receives the first non-selection information from the first selection unit 330-1. The optimization unit 114 generates first release instruction information based on the first non-selection information and outputs it to the memory 112.

After the first release instruction information is generated, the memory 112 receives the first release instruction information. The memory 112 releases the unnecessary data of the two pieces of processing in-progress data held in accordance with the first release instruction information. At this point, the memory 112 holds one piece of processing in-progress data.

The third extraction unit 320-3 inputs the third processed image from the processed image generation unit 310. The third extraction unit 320-3 generates a third feature by performing a third extraction process corresponding to a feature extraction process on the third processed image. The third extraction unit 320-3 outputs the third feature to the second selection unit 330-2, and outputs third processing data generated in the process of the third extraction process to the memory 112.

The feature extraction process in the third extraction unit 320-3 is performed at a timing when only one piece of processing process data is held in the memory 112 when the third processing process data is output to the memory 112. Alternatively, the feature extraction process in the third extraction unit 320-3 is performed in a state where one piece of processing process data is held in the memory 112.

After the third feature amount is extracted, the memory 112 receives the third processing process data from the third extraction unit 320-3 and stores the data. At this point, the memory 112 stores two pieces of processing process data.

The second selection unit 330-2 receives the first selected feature from the first selection unit 330-1 and receives the third feature from the third extraction unit 320-3. The second selection unit 330-2 compares the first selected feature with the third selected feature and selects the larger one as the second selected feature. The second selection unit 330-2 generates second non-selection information regarding the feature that was not selected and outputs it to the optimization unit 114, and outputs the second selected feature to the error calculation unit 120 as the maximum feature.

After the second non-selection information is generated, the optimization unit 114 receives the second non-selection information from the second selection unit 330-2. The optimization unit 114 generates second release instruction information based on the second non-selection information and outputs it to the memory 112.

After the second release instruction information is generated, the memory 112 receives the second release instruction information. The memory 112 releases the unnecessary data of the two pieces of processing process data held in accordance with the second release instruction information. At this point, the memory 112 holds only the processing process data related to the maximum feature amount. The memory 112 then outputs the processing process data related to the maximum feature amount to the learning unit 130.

3, the memory 112 holds only two pieces of processing process data corresponding to two feature amounts that are the targets of selection processing in the first selection unit 330-1 or the second selection unit 330-2. That is, the memory 112 holds two pieces of processing process data as an upper limit. Although a total of three pieces of processing process data are generated in the feature extraction unit 111, the memory 112 does not need to hold all three pieces of processing process data because it releases unnecessary data for each selection process, and this makes it possible to reduce memory capacity.

An example of processing using three processed images generated from an input image has been described above. Below, an example of generating four processed images from an input image will be described with reference to FIG.

4 is an explanatory diagram showing an example of dividing an input image 400 into four processed images 410 to 440 by cropping the input image 400. The feature extraction unit 111 generates the four processed images 410 to 440 by cropping the input image 400 to a predetermined size.

In Fig. 4, four processed images 410 to 440 are generated by simply dividing an input image 400. However, if an object to be recognized exists near the boundary between adjacent processed images, the object to be recognized may be divided at the boundary. Therefore, the process of cutting out multiple processed images so that they overlap each other will be described with reference to Fig. 5.

5 is an explanatory diagram showing an example of dividing an input image 500 into four processed images 510 to 540 by cutting out overlapping images. The feature extraction unit 111 generates the four processed images 510 to 540 by cutting out the input image 500 so that the multiple processed images overlap each other.

5 shows a processed image 510 including the upper left vertex of the input image 500, a processed image 520 including the upper right vertex, a processed image 530 including the lower left vertex, and a processed image 540 including the lower right vertex. These four processed images 510 to 540 have some overlapping areas. In this way, by overlapping a plurality of processed images, even if an object to be recognized is divided in one processed image, it is possible to prevent the object to be recognized from being divided in the other processed image. Therefore, the image processing device 110 can prevent non-detection of the object to be recognized.

An example of generating four processed images from an input image has been described above. However, the number of divided processed images is not limited to 3 or 4. Below, a configuration example of an image processing device in which the number of divided processed images is expanded to N will be described with reference to FIG. 6.

Fig. 6 is a block diagram showing a first other configuration example of the feature amount extraction unit 111 and maximum feature amount selection unit 113 in the image processing device 110 in Fig. 3. In the first configuration example, the processing using three processed images shown in Fig. 3 is expanded to processing using N processed images. Therefore, in Fig. 6, the feature amount extraction unit 111 will be described as feature amount extraction unit 111A, and the maximum feature amount selection unit 113 will be described as maximum feature amount selection unit 113A. Note that in Fig. 6, the memory 112 and optimization unit 114 in the image processing device 110 are omitted from illustration.

The feature extraction unit 111A includes a processed image generation unit 610 and a first extraction unit 620-1 to an N-th extraction unit 620-N. The maximum feature selection unit 113A includes a first selection unit 630-1 to an L-th selection unit 630-L, where L is N-1.

The processed image generation unit 610 generates N processed images based on the input image. The processed image generation unit 610 outputs the N processed images to the first extraction unit 610-1 to the Nth extraction unit 610-N, respectively.

The first extraction unit 620-1, the second extraction unit 620-2, the first selection unit 630-1, the third extraction unit 620-3, and the second selection unit 630-2 perform the same processing as the first extraction unit 320-1, the second extraction unit 320-2, the first selection unit 330-1, the third extraction unit 320-3, and the second selection unit 630-2 in FIG. 3, and therefore their explanation will be omitted.

The fourth extraction unit 620-4 and the third selection unit 630-3 perform substantially the same processing as the third extraction unit 620-3 and the second selection unit 630-2. The same also applies to the subsequent extraction units and selection units.

To summarize the configuration in Fig. 6, the selection units after the second selection unit 630-2 are configured to sequentially compare two feature amounts, the selected feature amount selected by the previous selection unit and an unselected feature amount. The extraction units after the third extraction unit 620-3 are performed at a timing when only one processing process data is held in the memory 112 when outputting the processing process data to the memory 112. Alternatively, the feature amount extraction process in the extraction units after the third extraction unit 620-3 is performed in a state where one processing process data is held in the memory 112. In other words, even if the number of processed images is expanded to N, the memory 112 only needs to hold two processing process data as an upper limit.

The above describes an example of the configuration of the image processing device when the number of images to be divided is expanded to N. Next, the operation of the image processing device 110 according to the first embodiment using N images to be processed will be described with reference to FIG.

Fig. 7 is a flowchart illustrating the operation of the image processing device according to the first embodiment. The flowchart in Fig. 7 shows a series of flows of maximum feature amount selection processing for one input image. The flowchart in Fig. 7 is premised on a configuration in which two feature amounts are compared in the selection unit as shown in Fig. 6. The following description will be given with reference to each unit in Fig. 1 and Fig. 6.

(Step ST701) When the image processing device 110 acquires an input image, the processed image generating unit 610 generates N (N3) processed images based on the input image.

(Step ST702) The first extraction unit 620-1 generates a first feature amount by performing a first extraction process on the first processed image.

(Step ST703) The memory 112 holds the first processing process data generated in the course of the first extraction processing.

(Step ST704) The second extraction unit 620-2 performs a second extraction process on the second processed image to generate a second feature amount.

(Step ST705) The memory 112 holds the second processing process data generated in the course of the second extraction processing. At this time, the memory 112 holds two pieces of processing process data.

(Step ST706) The first selection unit 630-1 compares the first feature amount with the second feature amount and selects the larger one as a first selected feature amount.

(Step ST707) The optimization unit 114 releases the processing process data corresponding to the feature amount not selected in the comparison between the first feature amount and the second feature amount from the memory 112. As a result, the memory 112 holds one processing process data.

(Step ST708) The image processing apparatus 110 defines variables i and j, and assigns the values 3 and 1 to them, respectively.

(Step ST709) The ith extraction unit 620-i generates the ith feature amount by performing the ith extraction process on the ith processed image.

(Step ST710) The memory 112 holds the ith processing progress data generated in the process of the ith extraction process. At this time, the memory 112 holds two processing progress data.

(Step ST711) The (i-1)th selection unit 630-(i-1) compares the j-th selected feature amount with the i-th feature amount and selects the larger one as the (j+1)-th selected feature amount.

(Step ST712) The optimization unit 114 releases the processing process data corresponding to the feature not selected in the comparison between the j-th selected feature and the i-th feature from the memory 112. As a result, the memory 112 holds only one processing process data.

(Step ST713) The image processing device 110 judges whether or not the variable i is N. If the variable i is not N, the process proceeds to step ST714. On the other hand, if the variable i is N, the image processing device 110 outputs the selected feature selected in the immediately preceding selection process as the maximum feature to the error calculation section 120, outputs processing process data related to the maximum feature to the learning section 130, and the process ends.

(Step ST714) The image processing apparatus 110 adds 1 to each of the variables i and j. After the process of step ST714, the process returns to step ST709.

The operation of the image processing device 110 according to the first embodiment using N processed images has been described above. The configuration in which two feature amounts are compared in the selection unit has been described above. However, the number of feature amounts to be compared is not limited to two. Below, an example in which three feature amounts are compared in the selection unit will be described with reference to FIG. 8. Note that when describing FIG. 8, reference will be made to FIG. 6, which shows a configuration in which two feature amounts are compared in the selection unit, as the comparison targets.

Fig. 8 is a block diagram showing another second configuration example of the feature amount extraction unit and maximum feature amount selection unit in the image processing device in Fig. 3. In the second configuration example, the comparison of two feature amounts in the selection unit shown in Fig. 3 or Fig. 6 is expanded to a comparison of three feature amounts. Therefore, in Fig. 8, the feature amount extraction unit 111 will be described as feature amount extraction unit 111B, and the maximum feature amount selection unit 113 will be described as maximum feature amount selection unit 113B. Note that the memory 112 and optimization unit 114 in the image processing device 110 are omitted from Fig. 8.

The feature extraction unit 111B includes a processed image generation unit 810 and a first extraction unit 820-1 through an N-th extraction unit 820-N. The maximum feature selection unit 113B includes a first selection unit 830-1 through an L-th selection unit 830-L, where L is (N-1)/2.

As described above, the difference between Fig. 8 and Fig. 6 is the number of feature amounts compared in the selection unit. Specifically, the first selection unit 830-1 in Fig. 8 inputs the first feature amount from the first extraction unit 820-1, the second feature amount from the second extraction unit 820-2, and the third feature amount from the third extraction unit 820-3. The first selection unit 830-1 then compares the three feature amounts from the first feature amount to the third feature amount, and selects the largest feature amount as the first selected feature amount.

8, the number of processing process data held by memory 112 is also different. For example, since three feature amounts are compared in first selection unit 830-1, at the time when the third feature amount is extracted, memory 112 holds three processing process data.

When the first selection unit 830-1 selects the first selected feature, the memory 112 releases unnecessary data (here, two pieces of processing-process data corresponding to the two features not selected) from among the three pieces of processing-process data held by the memory 112. At this point, the memory 112 holds one piece of processing-process data.

Next, the fourth extraction unit 820-4 extracts a fourth feature amount, and the fifth extraction unit 820-5 extracts a fifth feature amount, so that the memory 112 again becomes in a state where it holds three pieces of processing process data.

Furthermore, the second selection unit 830-2 receives the first selected feature from the first selection unit 830-1, the fourth feature from the fourth extraction unit 820-4, and the fifth feature from the fifth extraction unit 820-5. The second selection unit 830-2 compares the first selected feature, the fourth feature, and the fifth feature to select the largest feature as the second selected feature.

When the second selected feature is selected, the memory 112 releases unnecessary data from among the three pieces of processing process data held by the memory 112. At this point, the memory 112 is again in a state where it holds only one piece of processing process data.

8, the selection units after the second selection unit 830-2 are configured to sequentially compare three feature amounts, namely, the selected feature amount selected by the previous selection unit with two unselected feature amounts. Also, the extraction units after the fourth extraction unit 820-4 are performed in a state where at most two pieces of processing process data are held in the memory 112 when outputting the processing process data to the memory 112. That is, the memory 112 holds three pieces of processing process data as an upper limit.

8, the image processing device 110 is configured to compare three feature amounts in the selection unit, but is not limited to this. For example, the image processing device 110 may be configured to compare four or more feature amounts in the selection unit.

As described above, by matching the upper limit of processing process data stored in memory 112 with the number of features to be compared in the selection unit, regardless of the number of images to be processed, memory 112 only needs to store an upper limit of processing process data equal to the number of features to be compared in the selection unit.

Furthermore, in the configuration of Fig. 8, the extraction processes of the first extraction unit 820-1, the second extraction unit 820-2, and the third extraction unit 820-3 may be performed in parallel. Thereafter, after the selection process by the first selection unit 830-1 is completed and unnecessary data is deleted from the memory 112, the extraction processes of the fourth extraction unit 820-4 and the fifth extraction unit 820-5 may also be performed in parallel, and so on. Thus, according to the configuration of Fig. 8, the extraction processes can be performed in parallel, so that the image processing device 110 can reduce the overall processing time compared to the configuration of Fig. 6.

The parallelization of the extraction processes in Fig. 8 remains the same even if the number of feature amounts to be compared in the selection unit increases. For example, when comparing M feature amounts in the selection unit, the image processing device 110 may perform in parallel the extraction processes related to the M feature amounts used in the first selection process, and perform in parallel the extraction processes related to the M-1 feature amounts to be used in the second selection process and thereafter. In other words, when there are multiple feature amounts immediately after the extraction processes to be input to the selection unit, the image processing device 110 may generate these multiple feature amounts simultaneously.

(Another Example of Processed Images) In the above, an example of generating multiple processed images by dividing an input image has been described. However, the multiple processed images are not limited to dividing the input image. Below, an example of generating multiple processed images by reducing an input image will be described with reference to FIG. 9.

9 is an explanatory diagram showing an example in which two processed images 910 and 920 are generated by reducing an input image 900. The feature extraction unit 111 generates two processed images 910 and 920 with different reduction ratios by reducing the input image 900 through reduction processing. For example, the feature extraction unit 111 generates the processed image 910 by reducing the input image 900 to 1/2, and generates the processed image 920 by reducing the input image 900 to 1/4. Note that the input image 900 may be used as the processed image if it satisfies the image size that can be processed by the image processing device 110.

The reduction process may be performed separately from the neural network process using a fixed filter such as a bilinear or bicubic filter, or may be performed as part of the neural network using the fixed filter as a convolution filter. In the latter case, since the parameters of the convolution filter must be newly learned, the learning speed of the entire neural network is reduced, but it is expected to improve the recognition accuracy compared to the former case.

Next, a description will be given of the advantages of using processed images with different reduction ratios as shown in Fig. 9. The following describes two aspects.

The first point is the advantage of learning the reduced image itself. For example, when an input image contains an object to be recognized that is different in size from that in learning during inference using the image processing device 110, a neural network trained using multiple processed images that do not use reduced images may not be able to recognize the object.

9, for example, an object is learned at size A in input image 900 as a processed image, an object is learned at size A/2 in processed image 910, and an object is learned at size A/4 in processed image 920. In this case, if the parameters of the individual neural networks are shared, the image processing device 110 can recognize objects of any of the above sizes.

Furthermore, even if an object of size 2A appears in the input image 900 as the processed image, the size is reduced to size A in the processed image 910, so that the image processing device 110 can also recognize the object of size 2A. This is also true in the case where an object of size 4A appears in the input image 900 as the processed image.

Therefore, the image processing device 110 can share parameters of individual neural networks during learning and inference using processed images with different reduction ratios, thereby enabling the recognition of objects of different sizes during inference compared to those during learning.

The second viewpoint is the merit of including convolution processing in an individual neural network. To this end, we first explain the concept of receptive fields in convolution processing by neural networks using Figure 10.

FIG. 10 is an explanatory diagram illustrating a transformed image 1020, an intermediate image 1030, and a receptive field in two convolution processes for an input image 1010. FIG. 10 shows an example in which a 3×3 pixel convolution process is performed on the input image 1010 to generate a transformed image 1020, and a 3×3 pixel convolution process is also performed on the transformed image 1020 to generate an intermediate image 1030. Note that activation processes and the like are omitted from the illustration. Here, the receptive field refers to the pixel range 1021 of the transformed image 1020 and the pixel range 1011 of the input image 1010 that affect one pixel (for example, pixel 1031) of the intermediate image 1030. FIG. 10 shows an example in which the pixel range 1021 is 3×3 pixels and the pixel range 1011 is 5×5 pixels. The pixel 1031 depends only on the pixel range 1021 and the pixel range 1011, which are the receptive fields, so there is no effect no matter how much the pixel values outside this receptive field change.

In addition, when the size of the intermediate image 1030 and the kernel of the convolution process (e.g., 3 × 3 pixels) do not change, the receptive field (pixel range 1011) of the input image 1010 becomes wider as the number of convolution processes that generate a transformed image is increased, i.e., as the number of convolution layers is increased.

Considering the above, let us consider the convolution process for each of the processed images with different reduction ratios as shown in Fig. 9. First, each of the three individual neural networks is composed of multiple convolution layers. In addition, in these multiple convolution layers, the final converted image that has been convolutionally processed is called an intermediate image. The image processing device 110 applies a fully connected process or a global pooling process to the intermediate image to generate features.

In the neural network configured as above, generally, the feature amount of the input image is extracted in the processing up to the intermediate image, and the feature amount is used in the subsequent processing (the above-mentioned fully connected processing and global pooling processing) to perform classification. Therefore, it is desirable that the feature of the object to be recognized is sufficiently reflected in each pixel of the intermediate image. However, for example, if an object larger than that at the time of learning is included in the input image, the object may become larger than the receptive field of the input image, and the feature of the object to be recognized may not be sufficiently reflected in the intermediate image. To address this, it is possible to widen the receptive field area of the input image by increasing the number of convolution layers, but there is a problem that the amount of processing increases by the amount of the convolution layers.

On the other hand, by training the neural network using a reduced image, if the structures of the individual neural networks are the same, the size of the receptive field for the input image does not change in each individual neural network. Therefore, even if an object larger than that at the time of training is included in the input image, the image processing device 110 can cause any of the individual neural networks to recognize the object to be recognized.

In addition, as described above, when the parameters of the convolution filter used to reduce an image are also trained in an individual neural network, the reduced image may have a different appearance from the input image. Therefore, it is preferable that the image processing device 110 optimizes the parameters of the individual neural network for each image reduction ratio by changing the parameters of the individual neural network.

To summarize the above, the image processing device 110 can recognize objects of a different size than the size of the object to be recognized that was assumed during learning, by using multiple processed images that are reduced from the input image.

Furthermore, if the size of the processed image is reduced, the amount of processing data can be reduced. Therefore, the image processing device 110 can reduce the amount of data stored in the memory 112 by performing feature extraction processing from the processed image with the highest reduction ratio.

(Another configuration example of the maximum feature quantity selection unit) In the above, a configuration in which the respective selection units in the maximum feature quantity selection unit compare outputs from different selection units is not illustrated. Below, a configuration of the maximum feature quantity selection unit including a selection unit that compares outputs from different selection units will be explained with reference to Fig. 11. Note that, in Fig. 11, in order to simplify the explanation, it is assumed that four processed images are used.

Fig. 11 is a block diagram showing another example of the configuration of the image processing device 110 in Fig. 1. The other example of the configuration includes a configuration in which outputs from different selection units are compared with each other. Therefore, in Fig. 11, the feature extraction unit 111, the memory 112, the maximum feature selection unit 113, and the optimization unit 114 will be described as a feature extraction unit 111C, a memory 112C, a maximum feature selection unit 113C, and an optimization unit 114C, respectively.

The feature amount extraction unit 111C includes a processed image generation unit 1110 and a first extraction unit 1120-1 to a fourth extraction unit 1120-4. The maximum feature amount selection unit 113C includes a first selection unit 1130-1 to a third selection unit 1130-3.

The processed image generation unit 1110 generates four processed images based on the input image. The processed image generation unit 1110 outputs a first processed image of the four processed images to a first extraction unit 1120-1, outputs a second processed image to a second extraction unit 1120-2, outputs a third processed image to a third extraction unit 1120-3, and outputs a fourth processed image to a fourth extraction unit 1120-4.

The first extraction unit 1120-1 inputs the first processed image from the processed image generation unit 1110. The first extraction unit 1120-1 generates a first feature by performing a first extraction process corresponding to a feature extraction process on the first processed image. The first extraction unit 1120-1 outputs the first feature to the first selection unit 1130-1, and outputs first processing data generated in the process of the first extraction process to the memory 112C.

After the first feature amount is extracted, the memory 112C receives the first processing process data from the first extraction unit 1120-1 and stores it. At this point, the memory 112C stores one piece of processing process data.

The second extraction unit 1120-2 inputs the second processed image from the processed image generation unit 1110. The second extraction unit 1120-2 generates a second feature by performing a second extraction process corresponding to the feature extraction process on the second processed image. The second extraction unit 1120-2 outputs the second feature to the first selection unit 1130-1, and outputs second processing data generated in the process of the second extraction process to the memory 112C.

The second extraction process in second extraction section 1120-2 may be performed at the same timing as the first extraction process in first extraction section 1120-1.

After the second feature amount is extracted, the memory 112C receives the second processing process data from the second extraction unit 1120-2 and stores the data. At this point, the memory 112C stores two pieces of processing process data.

The first selection unit 1130-1 inputs the first feature from the first extraction unit 1120-1 and the second feature from the second extraction unit 1120-2. The first selection unit 1130-1 compares the first feature with the second feature and selects the larger one as the first selected feature. The first selection unit 1130-1 generates first non-selection information regarding the feature that was not selected and outputs it to the optimization unit 114C, and outputs the first selected feature to the third selection unit 1130-3.

After the first non-selection information is generated, the optimization unit 114C receives the first non-selection information from the first selection unit 1130-1. The optimization unit 114C generates first release instruction information based on the first non-selection information, and outputs the first release instruction information to the memory 112C.

After the first release instruction information is generated, the memory 112C receives the first release instruction information. The memory 112C releases the unnecessary data of the two pieces of processing in progress data held in accordance with the first release instruction information. At this point, the memory 112C holds one piece of processing in progress data.

The third extraction unit 1120-3 inputs the third processed image from the processed image generation unit 1110. The third extraction unit 1120-3 generates a third feature by performing a third extraction process corresponding to a feature extraction process on the third processed image. The third extraction unit 1120-3 outputs the third feature to the second selection unit 1130-2, and outputs third processing data generated in the process of the third extraction process to the memory 112C.

The feature extraction process in the third extraction unit 1120-3 is performed at a timing when only one piece of processing process data is held in the memory 112C when the third processing process data is output to the memory 112C. Alternatively, the feature extraction process in the third extraction unit 1120-3 is performed in a state where one piece of processing process data is held in the memory 112C.

After the third feature amount is extracted, the memory 112C receives the third processing process data from the third extraction unit 1120-3 and stores the data. At this point, the memory 112C stores two pieces of processing process data.

The fourth extraction unit 1120-4 inputs the fourth processed image from the processed image generation unit 1110. The fourth extraction unit 1120-4 generates a fourth feature by performing a fourth extraction process corresponding to a feature extraction process on the fourth processed image. The fourth extraction unit 1120-4 outputs the fourth feature to the second selection unit 1130-2, and outputs fourth processing data generated in the process of the fourth extraction process to the memory 112C.

The fourth extraction process in the fourth extraction unit 1120-4 may be performed at the same timing as the third extraction process in the third extraction unit 1120-3.

After the fourth feature amount is extracted, the memory 112C receives the fourth processing process data from the fourth extraction unit 1120-4 and stores the data. At this point, the memory 112C stores three processing process data.

To summarize the configuration in Fig. 11, although it is a combination of configurations in which two feature amounts are compared in a selection unit as in Fig. 6, it also includes a configuration in which outputs from different selection units (i.e., two selected feature amounts) are compared with each other. As a result, the memory 112C holds three processing step data, but the feature amount extraction unit 111C can simultaneously perform extraction processing using two extraction units. In other words, the feature amount extraction unit 111C can generate four feature amounts for each pair of feature amounts.

Furthermore, if the selection unit is expanded to compare a plurality of feature amounts for N processed images, the feature amount extraction unit 111C can generate N feature amounts for each of the plurality of feature amounts. This enables the image processing device 110 to improve the throughput of the feature amount extraction process while reducing memory usage compared to the conventional method.

The configuration of the maximum feature quantity selection unit including a selection unit that compares outputs from different selection units has been described above. Next, the operation of the image processing device 110 having such a configuration will be described with reference to FIG.

Fig. 12 is a flowchart illustrating another operation of the image processing device 110 according to the first embodiment. The flowchart in Fig. 12 shows a series of flows of maximum feature selection processing for one input image. The flowchart in Fig. 12 is premised on a configuration including a comparison of two selected features in the selection unit as shown in Fig. 11, and the number of images to be processed is expanded to N. The following description will be given with reference to the respective units in Figs. 1 and 11.

(Step ST1201) When the image processing device 110 acquires an input image, the processed image generation unit 1110 generates N processed images (N4) based on the input image.

(Step ST1202) The process of step ST1202 is the same as the process of steps ST702 to ST707 in FIG. 7. Specifically, the first extraction unit 1120-1 generates a first feature by performing a first extraction process on the first processed image. The memory 112C holds the first processing process data generated in the process of the first extraction process. The second extraction unit 1120-2 generates a second feature by performing a second extraction process on the second processed image. The memory 112C holds the second processing process data generated in the process of the second extraction process. The first selection unit 1130-1 compares the first feature with the second feature and selects the larger one as the first selected feature. The optimization unit 114C releases the processing process data corresponding to the feature not selected in the comparison between the first feature and the second feature from the memory 112C.

(Step ST1203) The image processing apparatus 110 defines variables i and j, and assigns the values 3 and 2 to them, respectively.

(Step ST1204) The ith extraction unit 620-i generates the ith feature amount by performing the ith extraction process on the ith processed image.

(Step ST1205) The memory 112C holds the ith processing progress data generated in the process of the ith extraction process. At this time, the memory 112C holds two processing progress data.

(Step ST1206) The (i+1)th extraction unit 620-(i+1) generates the (i+1)th feature amount by performing the (i+1)th extraction process on the (i+1)th processed image.

(Step ST1207) The memory 112C holds the (i+1)th processing progress data generated in the course of the (i+1)th extraction processing. At this time, the memory 112C holds three processing progress data.

(Step ST1208) The (i-1)th selection unit 1130-(i-1) compares the ith feature amount with the (i+1)th feature amount and selects the larger one as the jth selected feature amount.

(Step ST1209) The optimization unit 114C releases from the memory 112C the processing process data corresponding to the feature that was not selected in the comparison between the i-th feature and the (i+1)-th feature. As a result, the memory 112C holds two processing process data.

(Step ST1210) The i-th selection unit 1130-i compares the (j-1)-th selected feature amount with the j-th selected feature amount and selects the larger one as the (j+1)-th selected feature amount.

(Step ST1211) The optimization unit 114C releases from the memory 112C the processing process data corresponding to the feature not selected in the comparison between the (j-1)th selected feature and the jth selected feature. As a result, the memory 112C holds only one processing process data.

(Step ST1212) The image processing device 110 judges whether or not the variable i is N-1. If the variable i is not N-1, the process proceeds to step ST1213. On the other hand, if the variable i is N-1, the image processing device 110 outputs the selected feature selected in the immediately preceding selection process as the maximum feature to the error calculation unit 120, outputs processing process data related to the maximum feature to the learning unit 130, and ends the process.

(Step ST1213) The image processing apparatus 110 adds 2 to each of the variables i and j. After step ST1213, the process returns to step ST1204.

The processes in steps ST1204 and ST1206 may be performed at the same timing.

(Another embodiment of the feature extraction unit) In the above, a plurality of processed images based on an input image are used to perform feature extraction processing for each of the plurality of processed images. In other words, the above configuration uses an individual neural network for each of the plurality of processed images. Below, a configuration that can reduce memory capacity compared to the conventional configuration while processing an input image using one neural network will be described with reference to Figs. 13 to 15.

Fig. 13 is a block diagram showing another third example of the configuration of the feature amount extraction unit 111A and maximum feature amount selection unit 113A in the image processing device 110 in Fig. 6. The third example of the configuration is based on the processing using the N feature amounts shown in Fig. 6, and the handling of the N processed images is changed. Therefore, in Fig. 13, the feature amount extraction unit 111A will be described as feature amount extraction unit 111D, and the maximum feature amount selection unit 113A will be described as maximum feature amount selection unit 113D. Note that in Fig. 13, the memory 112 and optimization unit 114 in the image processing device 110 are omitted from illustration.

The feature extraction unit 111D includes a convolution processing unit 1310. The maximum feature selection unit 113D includes a first selection unit 1320-1 to an L-th selection unit 1320-L, where L is N-1.

The convolution processing unit 1310 generates an intermediate image from the input image by performing a convolution process as a feature extraction process, decomposes the intermediate image into N blocks of at least one pixel vertically and horizontally, and generates N feature amounts for each of the N blocks. At this time, the convolution processing unit 1310 does not perform the convolution process on the entire input image at once, but performs it on a specific region basis. A specific region refers to a region in the input image that affects a block in the intermediate image. The relationship between the input image and the intermediate image will be described below with reference to FIG. 14.

14 is an explanatory diagram illustrating a plurality of transformed images 1420, intermediate images 1430, and receptive fields 1440 in a convolution process for an input image 1410. Usually, a transformed image 1420 is generated from the input image 1410 by a convolution process, and an intermediate image 1430, which is the final transformed image generated by repeating the convolution process, is generated. At this time, a block 1431 in the intermediate image 1430 has a corresponding relationship (receptive field 1440) with an area in the transformed image 1420 and an area in the input image 1410. An example of an intermediate image will be described with reference to FIG. 15.

FIG. 15 shows an intermediate image 1500 divided into processing units of the convolution process. FIG. 15 shows an example of an intermediate image 1500 divided vertically and horizontally into 4×6 blocks. Each divided block corresponds to the specific region described above. That is, the convolution processing unit 1310 generates the intermediate image 1500 generated by normal convolution processing for each specific block. This allows the convolution processing unit 1310 to perform, with one neural network, processing equivalent to processing performed on a plurality of processed images with separate neural networks.

Specifically, the convolution processing unit 1310 identifies a region of the input image based on the receptive field of block 1510 in the intermediate image 1500, and performs feature extraction processing by regarding this as a first processed image. Block 1520 following block 1510 corresponds to the second processed image, and block 1530 following further corresponds to the third processed image. Then, the convolution processing unit 1310 performs feature extraction processing on the 24th processed image corresponding to the last block 1540, and then ends processing on the input image.

Note that the feature extraction unit 111D of FIG. 13 described above differs from the other feature extraction units only in the feature extraction method, and the subsequent processing by the maximum feature selection unit 113D may be similar to that of the maximum feature selection unit 113A of FIG. 6, for example.

In addition, the relationship between the intermediate image and the receptive field for the input image can be set arbitrarily. For example, by setting the receptive field for each adjacent block of the intermediate image so that the area of the input image overlaps, the intermediate image can be regarded as a processed image as described in FIG. 5 and the feature extraction process can be performed.

(Another Example of Input Image) In the above, a one-channel input image (for example, a black-and-white image) has been assumed. However, the input image may be an RGB color image. When the input image is a color image, the image processing device 110 treats one input image as three images having the same number of pixels in the vertical and horizontal directions for the Red, Green, and Blue components, that is, a so-called three-channel image. In this case, the image processing device 110 uses a three-dimensional kernel such as 3×3 pixels×3 channels. In addition, the image processing device 110 may perform conversion processing of two or more channels in the feature extraction processing. In image recognition processing using a neural network, it is generally known that the recognition accuracy increases as the number of channels of the converted image increases. Therefore, in this embodiment, the number of channels may be set as necessary.

(Other Examples of Feature Amounts) In the above description, it is assumed that feature amounts are generated as scalar values. However, feature amounts may be vectors having multiple elements. For example, when different types of objects such as cracks and stains are to be distinguished and simultaneously recognized, the image processing device 110 generates, as feature amounts, vectors with the same number of dimensions as the number of types of objects to be recognized.

Specifically, when performing full connection in the final process of the individual neural network, the image processing device 110 matches the number of channels of the output of the full connection to the number of types to be recognized, and arranges them to obtain the feature. Alternatively, when performing average value pooling or maximum value pooling in the final process of the individual neural network, the image processing device 110 matches the number of channels of the intermediate image to the number of types to be recognized, and arranges the pooled values for each channel to obtain the feature vector. An intermediate image having multiple channels and each feature will be described with reference to FIG. 16.

Fig. 16 is an explanatory diagram illustrating the relationship between an intermediate image having a plurality of channels and feature amounts for each channel. Fig. 16 shows an intermediate image having four channels 1610 to 1640. The image processing device 110 generates a feature amount for this intermediate image as a vector in which an individual feature amount 1611 corresponding to channel 1610, an individual feature amount 1621 corresponding to channel 1620, an individual feature amount 1631 corresponding to channel 1630, and an individual feature amount 1641 corresponding to channel 1640 are arranged.

Next, the processing in the selection unit, optimization unit, and error calculation unit when the feature is a vector will be described. In the following description, the case where the feature is a vector having two elements will be described. For example, when comparing two feature amounts, the selection unit compares each element of the vector of each feature amount, and outputs the vector in which the larger element is selected as the selected feature amount. At this time, the optimization unit releases the processing data related to the feature amount in which none of the elements of the vector was selected from the memory. In addition, the error calculation unit calculates an error value represented by a vector based on each element of the maximum feature amount and the correct feature amount corresponding to each element of the vector.

As described above, by converting the feature amount into a vector, it becomes possible to distinguish between different types of objects and recognize them simultaneously. Also in this case, the image processing device 110 does not need to store all the processing data in memory as in the past, and the memory capacity can be reduced.

In addition, if each channel in the individual neural network is configured independently so that the data values of each channel do not affect the other, the processing data for the unselected element of the two feature values in the element-by-element comparison in the selection unit is released from the memory, which can further reduce the memory capacity.

As described above, the image processing device of the first embodiment generates N features by performing feature extraction processing using a neural network on N (N3) processed images based on an input image, stores processing process data generated during the feature extraction processing in memory, selects the maximum feature by performing two or more comparisons with M combinations of 2 to N-1 of the N features, and releases M-1 or less pieces of processing process data corresponding to the M-1 or less features that were not selected from the memory after each comparison.

Therefore, the image processing device of the first embodiment can release unnecessary processing data from memory at any time during the process of extracting the maximum feature amount in the input image, thereby reducing the memory capacity required for image processing using a neural network.

In addition, the learning device including the image processing device related to the first embodiment calculates an error value based on the maximum feature and the correct feature corresponding to the input image, and learns a neural network based on the processing process data related to the maximum feature finally held in the memory and the error value.

Therefore, the learning device can reduce the memory capacity required when learning a neural network.

(Second embodiment) In the first embodiment, a learning device including an image processing device is described. On the other hand, in the second embodiment, an inference device including an image processing device is described. The configuration of the image processing device according to the second embodiment is substantially similar to the configuration of the image processing device according to the first embodiment. However, the image processing device according to the second embodiment differs from the image processing device according to the first embodiment in the type of processing process data stored in memory.

17 is a block diagram illustrating a configuration of an inference device 1700 including an image processing device 1710 according to the second embodiment. The inference device 1700 includes an image processing device 1710 (image processing unit) and an output unit 1720. The image processing device 1710 includes a feature extraction unit 1711, a memory 1712, a maximum feature selection unit 1713, and an optimization unit 1714.

Inference device 1700 may include an acquisition unit that acquires an input image used for inference by a neural network. Inference device 1700 may also include a control unit that controls each unit.

The feature extraction unit 1711, memory 1712, maximum feature selection unit 1713, and optimization unit 1714 have substantially the same configuration as, for example, the feature extraction unit 111, memory 112, maximum feature selection unit 113, and optimization unit 114 in FIG. 1, and therefore redundant explanations will be omitted.

1 in that the memory 1712 outputs processing data relating to the maximum feature amount to the output unit 1720. The maximum feature amount selection unit 1713 differs from the maximum feature amount selection unit 113 in FIG.

The output unit 1720 inputs the maximum feature from the maximum feature selection unit 1713, and inputs processing data related to the maximum feature from the memory 1712. The output unit 1720 generates an inference result based on the maximum feature, and outputs it to another device. The inference result is, for example, information indicating whether or not an object to be recognized is present in the input image.

Specifically, the output unit 1720 generates an inference result by comparing the maximum feature amount with a threshold value. For example, when the maximum feature amount is equal to or less than the threshold value, the output unit 1720 outputs an inference result indicating that the object to be recognized does not exist in the input image, and when the maximum feature amount is greater than the threshold value, the output unit 1720 outputs an inference result indicating that the object to be recognized exists in the input image. Note that when the maximum feature amount is expressed as a value between "0" and "1", the threshold value is, for example, "0.5".

Next, the types of processing process data handled by the image processing device 1710 will be described. The processing process data in the second embodiment is, for example, a part of an intermediate image. This processing process data is used when presenting an inference result, which will be described later, and may be rephrased as data necessary for presenting the inference result. Note that the processing process data in the second embodiment may further include a processed image.

The significance of storing intermediate images is that the intermediate image corresponding to the maximum feature amount corresponds to the object position in the input image, and its pixel value is known to be large. This is shown, for example, in the non-patent document "Neural Networks for Small Object Detection" (Vision Technology Practical Use Workshop, IS1-03, pp.32-37, Dec. 2020.). Therefore, by storing intermediate images in memory, when presenting the inference result, it is possible to display the input image showing the part where the pixel value is large in the intermediate image. This display allows the user to visually confirm the recognition result, thereby improving the explainability of the neural network.

Next, the relationship between the intermediate image as processing-in-progress data and the partial images of the intermediate image stored in the memory will be described with reference to FIGS.

Fig. 18 is an explanatory diagram illustrating the relationship between partial images that have been subjected to the convolution process and partial image data stored in memory. Fig. 18 shows a state after feature extraction processing is performed on the first two partial images 1810 and 1820 of an intermediate image 1800. In this example, partial image 1820 includes an object to be recognized, and the pixel value where the object is located is large. At this time, partial image data 1811 corresponding to partial image 1810 and partial image data 1821 corresponding to partial image 1820 are stored in memory 1712. After that, maximum feature selection unit 1713 selects the feature corresponding to partial image 1820 by selection processing.

Fig. 19 is an explanatory diagram illustrating an example of partial image data released from memory. Fig. 19 shows only partial image 1820 selected by the selection process. At this time, only partial image data 1821 is held in memory 1712, and free space 1900 is created by releasing partial image data 1811 corresponding to partial image 1810 not selected by the selection process. Thereafter, feature extraction unit 1711 performs feature extraction processing on the new partial image.

Fig. 20 is an explanatory diagram illustrating new partial image data stored in memory. Fig. 20 shows a state after feature extraction processing has been performed on partial image 1830, which follows partial image 1820, of intermediate image 1800. At this time, partial image data 1811 and partial image data 1831 corresponding to partial image 1830 are stored in memory 1712.

Thereafter, the image processing device 1710 continues processing other partial images of the intermediate image 1800 by repeatedly storing and releasing the partial image data in the memory 1712. Then, the image processing device 1710 finally outputs the partial image data stored in the memory 1712 to the output unit 1720 as processing-in-progress data.

The output unit 1720 may generate a composite image based on the processing process data including the partial image data and the input image. The composite image is, for example, an image in which the pixels of the object to be recognized that appears in the input image are emphasized by blend synthesis. In other words, the composite image is an image in which the inference result is visualized and reflected in the input image. At this time, the output unit 1720 may generate an interpolated intermediate image by interpolating partial image data that is not held in the memory 1712. The interpolated intermediate image will be described with reference to FIG. 21.

21 shows an interpolated intermediate image 2100 generated from the partial image 1820. The output unit 1720 generates the interpolated intermediate image 2100 by, for example, performing zero padding on an area 2110 other than the partial image 1820. This makes it possible to restore an intermediate image corresponding to the input image, and therefore the output unit 1720 can perform synthesis. Note that the partial image and the interpolated intermediate image can be considered to represent the contents of the inference result, and therefore may be called an inference image that visualizes the inference result.

(Another Configuration Example of the Maximum Feature Selection Unit) Unlike the learning device described in the first embodiment, the memory in the inference device may hold processing process data related to features that are not selected by satisfying a predetermined condition. The predetermined condition is when the feature is equal to or greater than a threshold value. Specifically, each selection unit of the maximum feature selection unit 1713 performs a comparison process with a threshold value for the feature that is not selected. Then, when the feature that is not selected is equal to or greater than the threshold value, the maximum feature selection unit 1713 does not generate non-selection information related to the feature. As a result, multiple processing process data are held in the memory 1712, so that the inference device 1700 can handle the case where the input image includes multiple objects to be recognized. In this case, the inference device 1700 may output an inference result based on the maximum feature and the feature equal to or greater than the threshold value.

As described above, the image processing device of the second embodiment, like the image processing device of the first embodiment, generates N features by performing feature extraction processing using a neural network on N (N3) processed images based on an input image, stores processing process data generated in memory during the feature extraction processing, selects the maximum feature by performing two or more comparisons with M combinations of 2 to N-1 of the N features, and releases M-1 or less pieces of processing process data corresponding to the M-1 or less features that were not selected from the memory after each comparison.

Therefore, the image processing device according to the second embodiment is expected to have the same effects as the image processing device according to the first embodiment.

Moreover, an inference device including an image processing device according to the second embodiment outputs an inference result indicating whether or not an object to be recognized exists in an input image based on the maximum feature amount. Furthermore, the inference device further compares each of the M feature amounts with a threshold value for each of two or more comparisons in the image processing device, and does not release from memory, for each of two or more comparisons, processing process data corresponding to feature amounts equal to or greater than the threshold value among the M-1 or less feature amounts not selected. Furthermore, the inference device outputs an inference result based on the maximum feature amount and the feature amount equal to or greater than the threshold value. Furthermore, when the processing process data is an inference image that visualizes the inference result, the inference device further outputs an image in which pixels of the object to be recognized appearing in the input image are emphasized based on the input image and the inference image.

Therefore, the inference device can reduce the memory capacity required when performing inference using a neural network.

22 is a block diagram illustrating a hardware configuration of a computer 2200 according to an embodiment. The computer 2200 includes, as hardware, a central processing unit (CPU) 2210, a random access memory (RAM) 2220, a program memory 2230, an auxiliary storage device 2240, and an input/output interface 2250. The CPU 2210 communicates with the RAM 2220, the program memory 2230, the auxiliary storage device 2240, and the input/output interface 2250 via a bus 2260.

The CPU 2210 is an example of a general-purpose processor. The RAM 2220 is used by the CPU 2210 as a working memory. The RAM 2220 includes a volatile memory such as a Synchronous Dynamic Random Access Memory (SDRAM). The program memory 2230 stores various programs including a program related to the maximum feature selection process (maximum feature selection program). As the program memory 2230, for example, a Read-Only Memory (ROM), a part of the auxiliary storage device 2240, or a combination thereof is used. The auxiliary storage device 2240 stores data non-temporarily. The auxiliary storage device 2240 includes a non-volatile memory such as an HDD or SSD.

The input/output interface 2250 is an interface for connecting to other devices. The input/output interface 2250 is used, for example, for connecting to other devices.

Each program stored in the program memory 2230 includes computer-executable instructions. When executed by the CPU 2210, the programs (computer-executable instructions) cause the CPU 2210 to execute a predetermined process. For example, when executed by the CPU 2210, a maximum feature selection program causes the CPU 2210 to execute the series of processes described with respect to each part of Figures 1, 3, 6, 8, 11, 13, and 17.

The program may be provided to the computer 2200 in a state where it is stored in a computer-readable storage medium. In this case, for example, the computer 2200 further includes a drive (not shown) for reading data from the storage medium, and acquires the program from the storage medium. Examples of the storage medium include a magnetic disk, an optical disk (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), a magneto-optical disk (MO, etc.), and a semiconductor memory. Also, the program may be stored in a server on a communication network, and the computer 2200 may download the program from the server using the input/output interface 2250.

The processing described in the embodiment is not limited to being performed by a general-purpose hardware processor such as the CPU 2210 executing a program, but may be performed by a dedicated hardware processor such as an ASIC (Application Specific Integrated Circuit). The term processing circuit (processing unit) includes at least one general-purpose hardware processor, at least one dedicated hardware processor, or a combination of at least one general-purpose hardware processor and at least one dedicated hardware processor. In the example shown in FIG. 22, the CPU 2210, the RAM 2220, and the program memory 2230 correspond to the processing circuit.

Therefore, according to each of the above embodiments, it is possible to reduce the memory capacity required for image processing using a neural network.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims.

100...Learning device, 110...Image processing device, 111...Feature extraction unit, 112...Memory, 113...Maximum feature selection unit, 114...Optimization unit, 120...Error calculation unit, 130...Learning unit, 200, 400, 500, 900, 1010, 1410...Input image, 210, 220, 230, 410, 420, 430, 440, 510, 520, 530, 540, 910, 920...Processed image, 1011, 1021...Pixel range, 1020, 1420...Transformed image, 1030, 1430, 1500, 1800...Intermediate image, 1031...Pixel, 1431, 1510, 1520, 1530, 1540...Block, 1440...Receptive field , 1610, 1620, 1630, 1640...channels, 1611, 1621, 1631, 1641...individual features, 1700...inference device, 1710...image processing device, 1711...feature extraction unit, 1712...memory, 1713...maximum feature selection unit, 1714...optimization unit, 1720...output unit, 1810, 1820, 1830...partial images, 1811, 1821, 1831...partial image data, 1900...area, 2100...interpolated intermediate image, 2110...area, 2200...computer, 2230...program memory, 2240...auxiliary storage device, 2250...input/output interface, 2260...bus.