Method for classifying and annotating image, device, electronic device, and storage medium

Image classification and labeling method, device, electronic device and storage medium

METHOD FOR CLASSIFYING AND ANNOTATING IMAGE, DEVICE, ELECTRONIC DEVICE, AND STORAGE MEDIUM

The invention relates to the field of data labeling, in particular to an image classification labeling method, device, electronic device and storage medium.

The surface mounting process of the Printed Circuit Board (PCB) is delicate and complicated, and therefore, various defects are easily generated during the production process. At present, researchers are trying to use the neural network training method to realize automatic defect detection on PCB boards. However, in the training process of the neural network, a large number of PCB board defect image data are required as negative samples, and normal PCB board image data are required as positive samples. In this way, in the face of a wide variety of PCB board defect data, if manual marking is used, a lot of manpower and time will be wasted, and the cost will be expensive.

In view of the above content, it is necessary to provide an image classification and labeling method, apparatus, electronic device and storage medium to solve the above problems.

An image classification and labeling method, the image classification and labeling method comprises: (a) acquiring an image to be tested; (b) comparing the image to be tested with a reference image to generate a corresponding image a mask, the image mask includes a complex connected domain; (c) performing flaw detection on the image to be tested; (d) acquiring the at least one flaw when the image to be tested has at least one flaw Corresponding flaw coordinates; (e) According to the center coordinates of the connected domain and the flaw coordinates, determine that the corresponding connected domain is a flawed connected domain or a normal connected domain; (f) respectively generate corresponding first connected domains for the flawed connected domain and the normal connected domain an image mask and a second image mask; (g) respectively processing the first image mask and the second image mask with the image to be tested, so as to obtain communication with the defect respectively A defective component image corresponding to the domain and a normal component image corresponding to the normal connected domain.

Further, step (b) includes the following sub-steps: Step S21: Perform grayscale processing on the reference image and the image to be tested, respectively, to obtain the corresponding first image and second image; Step S22: Comparing the first image and the second image to obtain a third image; step S23: performing binarization processing on the third image to obtain a fourth image; step S24: performing a binarization process on the third image The fourth image is marked with connected domains to obtain the image mask, and the image mask includes a plurality of the connected domains.

Further, by calculating the mean square error of the first image and the second image or calculating the structural similarity index of the first image and the second image, to obtain the third image.

Further, step (e) includes the following sub-steps: Step S51: Calculate the center coordinates of the complex connected domain to obtain the complex center coordinates; Step S52: Calculate the Euclidean of each flaw coordinate and the complex center coordinates Calculate the Euclidean distance to obtain a complex Euclidean distance; Step S53: Select the defect connected domain according to the default rule, and record other connected domains on the image mask as the normal connected domain.

Further, the center coordinate is the barycentric coordinate of the connected domain.

Further, the default rule is: respectively select the minimum value in the Euclidean distance of the complex group, obtain the complex minimum value, select the complex connected domain corresponding to the complex minimum value, and record it as the defect connected domain. .

Further, step (f) includes the following sub-steps: Step S61 : set the pixel value of the pixel in the normal connected region in the image mask to 0 to obtain the first image mask; Step S62 : repeat In the above-mentioned steps, the pixel value of the pixel in the connected region of the defect in the image mask is set to 0, so as to obtain the second image mask.

Further, step (g) includes the following sub-steps: Step S71 : multiply the first image mask by the image to be tested to obtain the image of the defective component, and add the image of the defective component to the image of the defective component. The flaw coordinates are marked on the top; Step S72 : multiply the second image mask by the image to be tested to obtain the normal component image.

Further, when the detection result of step (c) is that the image to be tested is flawless, that is, when the image mask is the second image mask, step (g) is directly performed.

An image classification and labeling device, the image classification and labeling device comprises: an acquisition module for acquiring an image to be measured; a comparison module for comparing the to-be-measured image with a reference image, To generate a corresponding image mask, the image mask includes a complex connected domain; a defect detection module is used to detect defects on the image to be tested; a coordinate acquisition module is used to detect the defects when the When the detection result of the module is that there is at least one flaw in the image to be tested, the flaw coordinate corresponding to the at least one flaw is obtained; The connected domain is a defective connected domain or a normal connected domain; A mask generation module is used to generate corresponding first image masks and second image masks respectively for the defect connected region and the normal connected region; a processing module is used to generate the first image mask The image mask and the second image mask are respectively processed with the image to be tested to obtain a defective component image corresponding to the defective connected area and a normal component image corresponding to the normal connected area, respectively.

An electronic device, the electronic device comprises: a memory, which stores at least one instruction; and a processor, which executes the instructions stored in the memory to implement the image classification and labeling method.

A storage medium, wherein at least one instruction is stored in the storage medium, and the at least one instruction is executed by a processor in an electronic device to implement the image classification and labeling method.

The image classification and labeling method provided by the present invention, combined with the image mask and the defect detection algorithm, can automatically obtain and label the normal component images and the defective component images, thereby effectively reducing the labor cost during data labeling.

<p>100: Image classification and annotation device</p><p>101: Get Mods</p><p>102: Compare Mods</p><p>103: Defect detection module</p><p>104: Coordinate acquisition module</p><p>105: Determine the module</p><p>106: Mask Generation Module</p><p>107: Processing modules</p><p>200: Electronic Equipment</p><p>201: Memory</p><p>202: Processor</p><p>203: Computer Programs</p><p>1, 2: Connected Domain</p><p>a, b: center coordinates</p><p>(1), (2): Euclidean distance</p><p>S1~S6, S21~S24, S51~S53, S61~S62, S71~S72: Steps</p>

FIG. 1 is a flowchart of an embodiment of an image classification and labeling method according to the present invention.

FIG. 2 is a sub-step of step S2 shown in FIG. 1 .

FIG. 3 is a sub-step of step S5 shown in FIG. 1 .

FIG. 4 is a sub-step of step S6 shown in FIG. 1 .

FIG. 5 is a sub-step of step S7 shown in FIG. 1 .

FIG. 6 is a schematic diagram of an application scenario of the image classification and labeling method shown in FIG. 1 .

FIG. 7 is a functional module diagram of an embodiment of the image classification and labeling apparatus of the present invention.

FIG. 8 is a schematic structural diagram of an electronic device for implementing an image classification and labeling method according to an embodiment of the present invention.

In order to understand the above objects, features and advantages of the present invention more clearly, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features of the embodiments may be combined with each other without conflict.

In the following description, many specific details are set forth in order to facilitate a full understanding of the present invention, and the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the protection scope of the present invention.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terms used herein in the description of the present invention are for the purpose of describing specific embodiments and are not intended to limit the present invention.

Please refer to FIG. 1 . FIG. 1 is a flowchart of an embodiment of an image classification and labeling method of the present invention. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

Step S1, acquiring an image to be tested.

In step S2, the image to be tested is compared with a reference image to generate a corresponding image mask, and the image mask includes a complex connected domain.

Wherein, when step S2 is executed, please refer to FIG. 2 , step S2 includes the following sub-steps: step S21 : respectively performing grayscale processing on the image to be tested and the reference image to obtain the corresponding first image and second image; step S22: comparing the first image and the second image to obtain a third image; step S23: performing binarization processing on the third image to obtain a fourth image picture; Step S24: Marking the connected domain on the fourth image to obtain the image mask, where the image mask includes a plurality of the connected domains.

It can be understood that in step S21, the grayscale processing refers to converting the reference image and the image to be tested from color images into grayscale images with only one pixel value per pixel.

It can be understood that in step S22, the mean-square error (MSE) of the first image and the second image can be calculated or the first image and the second image can be calculated by calculating the mean-square error (MSE) of the first image and the second image. Structural similarity (SSIM) index of the images to obtain the third image.

It can be understood that step S23 further includes the following sub-steps: obtaining a threshold value; setting the gray value of the pixel greater than or equal to the threshold value in the third image as the first pixel value, The gray value of the pixel at the threshold is set as the second pixel value to obtain the fourth image.

The present invention does not limit the threshold, for example, the threshold in step S23 may be a global fixed threshold or a regional adaptive threshold. Those skilled in the art can select different thresholds by selecting different binarization algorithms.

It can be understood that the connected domain in step S24 refers to an image area composed of adjacent pixels with the same pixel value in the fourth image. The connected domain labeling refers to finding and labeling each connected domain in the fourth image, and recording all the coordinate values corresponding to each label value.

It can be understood that the pixel value of the plurality of pixels in the connected region on the image mask is the first pixel value. The pixel values of the pixels on the image mask other than the plurality of connected regions are the second pixel values.

In one embodiment, the value of the first pixel in step S23 is 1, and the value of the second pixel is 0.

It can be understood that there are various connected domain labeling algorithms in step S24, and the present invention does not limit the connected domain labeling algorithm. For example, the connected domain labeling algorithm may be a travel-based connected domain labeling algorithm or a contour-based connected domain labeling algorithm. It can be understood that the connected domain labeling algorithm is in the prior art, and the detailed steps of the connected domain labeling algorithm will not be described in detail in the present invention.

In one embodiment, the image to be tested includes an image of a Printed Circuit Board Assembly (PCBA) to which electronic components have been added. The reference image includes an image of a Printed Circuit Board (PCB) that has not been mounted with any of the electronic components.

It can be understood that each of the connected domains on the mask corresponds to the electronic element.

It can be understood that, before performing the step S21, the reference image and the image to be tested can also be preprocessed.

In at least one embodiment of the present invention, the preprocessing includes: translating, rotating or scaling the reference image and the image to be tested, so that the reference image and the plurality of images to be tested are The directions are the same; the reference image and the image to be tested are adjusted to standard size images.

It can be understood that the dimensions of the reference image, the image to be tested, the first image, the second image, the third image, the fourth image and the image mask are the same.

Step S3, performing flaw detection on the image to be tested.

In at least one embodiment of the present invention, a trained neural network model can be used to perform flaw detection on the image to be tested.

In the present invention, the image to be tested can also be detected by other methods, such as image erosion and/or image dilation algorithms, etc. The present invention does not limit the defect detection method.

Step S4, when the image to be tested has at least one defect, obtain the defect coordinates corresponding to the at least one defect.

Step S5, according to the center coordinate of the connected domain and the flaw coordinate, determine that the corresponding connected domain is a flaw connected domain or a normal connected domain.

Wherein, when step S5 is executed, please refer to FIG. 3 , step S5 includes the following sub-steps: step S51 : calculating the center coordinates of the complex connected domain to obtain complex center coordinates; step S52 : calculating each of the flaw coordinates and the complex number Euclidean distance of the central coordinates to obtain complex Euclidean distance; Step S53: Select the defect connected domain according to the default rule, and mark other connected domains on the image mask as normal connected domain.

In one embodiment, the center coordinates are barycentric coordinates. In this way, in step S51, the complex barycentric coordinates can be obtained according to the following barycentric coordinate calculation formula (1).

<maths><img file="TWI771908B_D0001.tif" /></maths>

In formula (1), X and Y are the row and column coordinates of the barycentric coordinates, S represents the connected domain, (i, j) represents the coordinates of each pixel on the connected domain, and N represents the pixel of the connected domain. number.

It can be understood that the Euclidean distance refers to the true distance between two points in a multi-dimensional space. In this embodiment, since the size of the image mask and the image to be tested are the same, the position of the flaw coordinates on the image to be tested is the same as that on the image mask. Thus, in step S52, the Euclidean distance between the flaw coordinate and the center coordinate is determined on the image mask, and the distance between the flaw coordinate and the center coordinate. In this way, the Euclidean distance between the flaw coordinates and the center coordinates of the connected domains can be directly calculated by the following Euclidean distance calculation formula (2).

<maths><img file="TWI771908B_D0002.tif" /></maths>

In formula (2), (<i>i</i>₁，<i>j</i>₁) is the center coordinate of any of the connected domains, (<i>i</i>₂，<i>j</i>₂) are the flaw coordinates.<i>ρ</i>is the Euclidean distance between the flaw coordinates and the center coordinates.

In step S53, the default rule is: respectively select the minimum value in the Euclidean distance of the complex group to obtain the complex minimum value; select the complex connected domain corresponding to the minimum value of the complex number, and record it as the defect connectivity area.

Step S6, respectively generating a corresponding first image mask and a second image mask for the defect connected region and the normal connected region. 4, Step S6 includes the following sub-steps: Step S61: Set the pixel value of the pixel in the normal connected region in the image mask to 0 to obtain the first image mask; Step S62 : Repeat the above steps to set the pixel value of the pixel of the defect connected region in the image mask to 0, so as to obtain the second image mask.

In step S7, the first image mask and the second image mask are respectively processed with the image to be tested, so as to obtain the image of the defective component corresponding to the connected region of the defect and the image corresponding to the normal The normal component image corresponding to the connected domain. 5 , step S7 includes the following sub-steps: Step S71 : multiplying the first image mask by the image to be tested to obtain the image of the defective component, and adding the image to the defective component The flaw coordinates are marked on the image; step S72 : multiply the second image mask by the image to be tested to obtain the normal component image.

It can be understood that, in other embodiments, when the detection result of step S3 is that the image to be tested is flawless, that is, when the image mask is the second image mask, step S7 is directly executed to Obtain normal component images.

Referring to FIG. 6 , in the embodiment of the present application, the image classification and labeling method of the present application will be described in detail by taking as an example that the image to be tested is provided with two electronic components.

First, an image to be tested and a corresponding reference image are acquired, wherein, as described above, the image to be tested includes two electronic components. The image to be tested and the reference image are compared to generate an image mask corresponding to the image to be tested, and the image mask includes a connected domain 1 and a connected domain 2 .

Next, flaw detection is performed on the image to be tested, and the detection result is that the image to be tested has a flaw (ie, a triangle in the figure). Obtain the flaw coordinates corresponding to the flaw.

Then, the barycentric coordinates of the connected domain 1 and the connected domain 2 are respectively calculated according to the barycentric coordinate formula, which are denoted as the central coordinate a and the central coordinate b, respectively. Then calculate the Euclidean distance between the center coordinate a and the center coordinate b and the flaw coordinate respectively, which are denoted as Euclidean distance (1) and Euclidean distance (2). Comparing the Euclidean distance (1) and the Euclidean distance (2), it is obtained that the Euclidean distance (1) is greater than the Euclidean distance (2), then the connected domain 1 is normally connected Domain, Connected Domain 2 is the flaw connected domain corresponding to the flaw.

Next, set the pixel value of the pixel value of the normal connected region in the image mask to 0 to obtain a first image mask; repeat this step to set the pixel value of the defect connected region in the image mask to 0 The pixel value of the pixel is set to 0 to obtain a second image mask.

Then, the first image mask is multiplied by the image to be tested to obtain an image of a defective element, and the defect is marked on the image of the defective element. The second image mask is multiplied by the image to be tested to obtain a normal component image.

The present invention obtains an image mask including a complex connected domain by comparing the reference image and the corresponding image to be tested. When there is a defect in the image to be tested, find the defect connected region corresponding to the defect, and mark the connected region on the image mask except the defect connected region as a normal connected region. Set the pixel value of the pixel of the normal connected domain in the image mask to 0 to obtain a first image mask; set the pixel value of the pixel of the defective connected domain in the image mask to 0, A second image mask is obtained. The first image mask is multiplied by the image to be tested to obtain an image of a defective component; the second image mask is multiplied by the image to be tested to obtain an image of a normal component. When the image to be tested is flawless, the image mask is directly multiplied by the image to be tested to obtain a normal component image. The image classification and labeling method provided by the present invention effectively reduces the labor cost during data labeling.

Referring to FIG. 7 , another embodiment of the present invention further provides an image classification and labeling apparatus 100 . The image classification and labeling device 100 includes an acquisition module 101 , a comparison module 102 , a defect detection module 103 , a coordinate acquisition module 104 , a determination module 105 , a mask generation module 106 and a processing module 107 .

Wherein, the acquisition module 101 is used for acquiring an image to be measured.

The comparison module 102 is used for comparing the image to be tested with a reference image to generate a corresponding image mask, and the image mask includes a complex connected domain.

The defect detection module 103 is used for defect detection on the image to be tested.

The coordinate obtaining module 104 is used for obtaining a defect coordinate corresponding to the at least one defect when the detection result of the defect detection module 103 is that the image to be tested has at least one defect.

The determining module 105 is used for determining whether the corresponding connected domain is a defective connected domain or a normal connected domain according to the center coordinates of the connected domain and the defect coordinates.

The mask generating module 106 is used for generating a corresponding first image mask and a second image mask for the defective connected region and the normal connected region, respectively.

The processing module 107 is used for processing the first image mask and the second image mask with the image to be tested respectively, so as to obtain the image of the defective component corresponding to the connected region of the defect. and a normal component image corresponding to the normal connected domain.

It can be understood that when the detection result of the defect detection module 103 is that the image to be tested is flawless, that is, when the mask is the second image mask, the processing module 107 is directly used to The mask and the image to be tested are processed to obtain the normal component image.

It can be understood that the acquisition module 101, the comparison module 102, the defect detection module 103, the coordinate acquisition module 104, the determination module 105, the mask generation module 106 and the processing module 107 are used to jointly realize the above image For the steps S1 to S7 in the embodiment of the classification and labeling method, the specific implementation process of each of the functional modules will not be repeated here. For details, please refer to the above steps S1 to S7.

It can be understood that referring to FIG. 8 , another embodiment of the present invention further provides an electronic device 200 . The electronic device 200 includes a memory 201 , a processor 202 , and a computer program 203 stored in the memory 201 and running on the processor 202 .

The electronic device 200 may be any one of a smart phone, a tablet computer, a laptop computer, an embedded computer, a desktop computer, or a server or the like. Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 200, and does not constitute a limitation to the electronic device 200. It may include more or less components than the one shown, or combine some components, or different parts.

When the processor 202 executes the computer program 203, the steps in the above-mentioned embodiment of the image classification and labeling method are implemented, for example, the steps S1-S7 shown in the first embodiment. Or, when the processor 202 executes the computer program 203, the functions of each module/unit in the above-mentioned embodiment of the image classification and labeling apparatus 100 are realized, for example, the acquisition module 101, the comparison module 102, A defect detection module 103 , a coordinate acquisition module 104 , a determination module 105 , a mask generation module 106 and a processing module 107 .

The computer program 203 can be divided into one or more modules/units, and the one or more modules/units are stored in the memory 201 and executed by the processor 202 to complete the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 203 in the electronic device 200 . For example, the computer program 203 can be divided into the acquisition module 101, the comparison module 102, the defect detection module 103, the coordinate acquisition module 104, the determination module 105, and the mask generation module 106 in the second embodiment and processing module 107 .

The processor 202 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), and application specific integrated circuits (ASICs). ), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General purpose processors can It is a microprocessor or the processor 202 can also be any conventional processor, etc. The processor 202 is the control center of the electronic device 200, and uses various interfaces and circuits to connect various parts of the entire electronic device 200.

The memory 201 can be used to store the computer program 203 and/or module/unit. The processor 202 runs or executes the computer program and/or module/unit stored in the memory 201, And call the data stored in the memory 201 to realize various functions of the electronic device 200 . The memory 201 may mainly include a program storage area and a data storage area. Wherein, the storage program area can store the operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.) and the like. The storage data area can store data (such as video data, audio data, phone book, etc.) created according to the use of the electronic device 200 and the like. In addition, the memory 201 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, Smart Media Card (SMC), Secure Digital (Secure Digital, SD) card, flash memory card (Flash Card), at least one disk memory device, flash memory device, or other volatile solid state memory device.

In an embodiment of the present invention, the electronic device 200 is an automated optical inspection (Automated Optical Inspection, AOI) instrument. It can be understood that automatic optical inspection equipment is a device based on optical principles to detect common defects encountered in welding production.

If the modules/units integrated in the electronic device 200 are implemented in the form of software functional modules and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on this understanding, the present invention realizes all or part of the processes in the methods of the above embodiments, and can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a storage medium, and the computer When the program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code may be in the form of original code, executable file or some intermediate form. The computer-readable medium may include: capable of carrying the computer Program code of any entity or device, recording medium, pen drive, removable hard disk, magnetic disk, CD-ROM, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access) Memory), electrical signals, and software distribution media. It should be noted that the content contained in the computer-readable media may be modified as appropriate in accordance with the requirements of legislation and patent practice in certain jurisdictions, for example, in some jurisdictions, according to legislation and patent practice, computer-readable media Electric carrier signals and telecommunication signals are not included.

In the several embodiments provided by the present invention, it should be understood that the disclosed electronic device and method can be implemented in other ways. For example, the electronic device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and other division methods may be used in actual implementation.

In addition, each functional module in each embodiment of the present invention may be integrated in the same processing module, or each module may exist physically alone, or two or more modules may be integrated in the same module. The above-mentioned integrated modules can be implemented either in the form of hardware or in the form of hardware plus software function modules.

It will be apparent to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, but that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary rather than restrictive, and the scope of the present invention is defined by the scope of the appended claims rather than the above description, and is therefore intended to fall within the scope of the claims. All changes within the meaning and scope of equivalents to the scope are encompassed within the invention. Any reference signs in the scope of the patent application should not be construed as limiting the scope of the patent application referred to. Furthermore, it is clear that the word "comprising" does not exclude other modules or steps, and the singular does not exclude the plural. A plurality of modules or electronic devices stated in the scope of the electronic device patent application can also be implemented by software or hardware by the same module or electronic device. The terms first, second, etc. are used to denote names and do not denote any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified. or equivalent replacement without departing from the spirit and scope of the technical solutions of the present invention.