Method and apparatus for visual detection and inspection of objects

[Technical field to which the invention belongs] The present invention relates to automatic detection and inspection of objects manufactured on a production line, and more specifically to industrial machine vision and automatic image analysis.

[Explanation of related technologies] Industrial production relies on the automatic detection of manufactured objects. One form of automatic detection that has been widely used for decades is usually based on infrared or visible light, photoelectric sensors, and electromagnetic energy-based optoelectronics that make some form of electronic judgment.

A known form of automatic optoelectronic testing utilizes the processing of photodetectors. A typical photodetector has a light source and a photoelectric sensor that respond to the intensity of light reflected by a point on the surface of the object or emitted along a path that the object may pass through. A user-adjustable sensitivity threshold defines the intensity of light such that the output signal of the photodetector is excited to be greater (or smaller).

Photodetectors, often referred to as gates, are used to detect the presence of an object to be inspected. Other photodetectors are placed relative to the gate to detect the light reflected at the appropriate point on the object. With proper adjustment of the sensitivity threshold, these other photodetectors can detect the presence of certain features of the object, such as labels and holes. The determination of the state of an object (eg, pass or fail) is made using the output signals of these other photodetectors when the object is detected by the gate. This decision is typically made by a programmable logic controller (PLC) or other suitable electronic device.

Automatic inspection using a photodetector has various advantages. Photodetectors are inexpensive, easy to set up, and operate at very high speeds (PLCs take longer to make decisions, but the output responds within hundreds of microseconds of object detection).

However, automatic inspection using a photodetector also has various drawbacks such as the following.

A simple perception of the intensity of light reflected from a point on an object is often insufficient for inspection. Instead, it may be necessary to perform a pattern analysis of the brightness of the light reflected from the vast area. For example, in order to detect edges, it may be necessary to analyze the pattern of light brightness to see if it corresponds to the transition from bright to dark.

Placing a photodetector can be difficult if many points on the object need to be inspected. Each such inspection point requires the use of another photodetector, which must be physically performed in such a way that it is not disturbed by the placement of other photodetectors. Interference may be due to location restrictions, light sources, or crosstalk from other factors.

Production lines can usually produce mixed products, but each has its own inspection requirements. The arrangement of photodetectors is very inflexible, as switching lines from one product to the next requires the photodetectors to be physically moved and readjusted. The cost of performing a line switch and the risk of associated human error often offset the low cost and simplicity of the photodetector.

To use an array of photodetectors, the object must be in a predetermined, known location so that the appropriate point on the object can be detected. This need can add additional cost and complexity that offset the low cost and simplicity of photodetectors.

Another known form of photoelectron automatic inspection is to use a device that can take a two-dimensional visual field digital image of the object to be inspected and then analyze the image to make a decision. .. Such devices are commonly referred to as machine vision systems, or simply vision systems. The image is obtained by exposing the photosensitive elements of the two-dimensional array to the light collected by the lens in the array for a short period of time called integration time or shutter time. Arrays are referred to as imaging devices, and individual elements are referred to as pixels. Each pixel measures the intensity of the light that hits it within the shutter time. The measured intensity values are then converted to digital numbers and stored in memory within the visual system to tie the image, which is known in this technique to determine the condition of the object under examination. It is analyzed by digital processing elements such as computers.

There are cases where the object is brought into the field of view and is stationary, and there are cases where the object is continuously moving in the field of view. Events outside the vision system, such as signals from photodetectors or communications from PLCs, computers, or other automated devices, are used to notify the vision system that an object has been identified in the field of view. Images are acquired and analyzed. Such an event is called a trigger.

Machine vision systems avoid the drawbacks associated with the use of photodetector arrays. They perform pattern analysis of the brightness of the light reflected from a vast area, easily respond to many prominent features of the object, address production line shifts by software systems and / or processing, and are unclear Corresponds to the position of various objects.

Machine vision systems have the following drawbacks compared to the arrangement of photodetectors. Often 10 times more expensive than photodetector arrays and relatively expensive. -Often requires people with specific technical training and is difficult to set up. It typically takes tens or hundreds of milliseconds to make a decision, so it is operated even more slowly than an array of photodetectors. In addition, the decision time can vary significantly and unexpectedly depending on the object.

Machine vision systems are limiting because they judge from a single image for each object identified in one place in vision (each object is identified in a different or unpredictable position). Sometimes, there is only one such position on which to base a decision on an object). This unique position provides information obtained from the only viewpoint, the only direction with respect to the lighting. Having only one point of view often leads to inaccurate judgment. Observations have been made over a long period of time in this regard, and in some cases, for example, a difference of only one pixel in the field of view can make an inaccurate judgment accurate. On the other hand, when a person inspects an object, the object is moved with respect to the eyes or light in order to improve the reliability of judgment.

Some traditional visual systems take multiple images of stationary objects in the field of view and average them to produce a single image for analysis. Averaging reduces measurement noise and, as a result, improves making decisions, but still has only one viewpoint and direction of illumination, requires considerable additional time, and requires the object to remain stationary.

Traditional visual systems designed to read alphanumeric, barcode, or 2D matrix codes capture multiple images until some accurate reading is obtained or all variations are performed. , Change the direction of lighting. This method is because such code contains enough verbose information that the visual system is certain if the reading is correct, and the object remains stationary in the field of view long enough to perform all variations. It is valid. That method is generally not suitable for inspecting an object, and even if the object is in continuous operation. Moreover, the decision is based on a single image only, as only one perspective is provided and no information is needed from images that did not produce results with correct reading.

Some traditional visual systems are used to guide robots in pick-and-place applications where objects are in continuous motion through the field of view. Such systems are designed to move objects at a speed at which the visual system has the opportunity to see each object at least twice. However, the purpose of this design is not to benefit from multiple perspectives, but rather to temporarily slow down the visual system, such as when the number of objects in the field of view is greater than average. Make sure you don't miss it all. These systems do not use potential additional information given by multiple perspectives.

Machine vision systems also have limitations in the use of trigger signals. Since the trigger signal is required, the setting becomes more complicated, for example, the installation and adjustment of a photodetector is required to transmit an appropriate signal, and software for a PLC or a computer must be created. Almost a photodetector is used when the object is moving continuously, in which case it may be necessary to physically move it to convert the production line, which is the vision system. It offsets some of the strengths. In addition, photodetectors respond only to changes in the intensity of light reflected from or transmitted along the path of an object. Such a condition may not be sufficient to provide reliable detection when an object enters the field of view.

Traditional visual systems designed to read alphanumeric codes, barcodes, or two-dimensional (2D) matrix codes can be operated without triggers by continuously capturing images and reading the codes. For the same reasons explained above, such a method is generally not suitable for inspecting an object, nor is it appropriate when the object is in continuous operation.

Conventional technology vision systems used for continuously moving objects can function without the use of triggers by using a method called self-triggering. These systems typically work by monitoring the difference in brightness or color that represents the state of an object in one or more captured image parts. Self-triggers are rarely used in practice due to some limitations, including:

The vision system's response to self-triggers is too slow to be used at normal production rates. -The method of detecting when an object appears is often not sufficient. A vision system is the location of an object on a particular repeatable production line, usually emitted by a photodetector that acts as a trigger and that the PLC or processor needs to function at the discretion of the vision system. Does not provide a useful output signal tuned to.

Many of the limitations of machine vision are also due to the inability to capture and analyze moving objects from multiple fields of view, or to react to events occurring in the field of view. .. Since most vision systems can acquire a new image while analyzing the current image, the maximum operating speed of the vision system is determined by the acquisition time and how long the analysis time takes. In general, the most important determinant of this speed is the number of pixels that make up the imaging device.

The time required to acquire an image is mainly determined by the number of pixels in the image pickup apparatus, and there are two reasons for this. First, the shutter time is determined by the amount of light and the sensitivity of each pixel. In general, increasing the number of pixels means that the size of the pixels decreases, so the sensitivity decreases, and as the number of pixels increases, the shutter time also increases. Second, conversion time and storage time are proportional to the number of pixels. Therefore, the larger the number of pixels, the longer the acquisition time.

For at least the last 25 years, traditional vision systems have generally used 300,000 pixels, but more recently, systems that can use 1 million pixels have emerged, and for many years as little as 75,000. Fewer systems have used pixels. Similar to digital cameras, the recent trend is to increase the number of pixels and get better image resolution. During this time, computer speeds have increased tremendously and imagers have moved from vacuum tubes to semiconductors, but machine vision image acquisition times have generally doubled from 1 / 30th to 1 / 60th of a second. I just showed it. Faster computers have enabled more sophisticated analysis, but the maximum speed at which the vision system operates remains largely unchanged.

Recently introduced CMOS imaging devices reduce conversion time and storage time to acquire small parts of photosensitive elements. In theory, such an imager can handle very short acquisition times, but in reality, the light sensitivity of the pixels is not as good as when using a full array, making a fast and useful imager. Needed, very difficult and / or expensive to achieve very short shutter times.

Due in part to the problem of image acquisition time, no image analysis method has been developed that is suitable for manipulating at significantly higher rates than 60 images per second. Similarly, the use of multiple viewpoints, untriggered operation, generation of properly synchronized output signals, and various other useful features are not fully considered in conventional techniques.

Recently, a pilot device called a focal plane array processor has been developed at a research facility. These devices combine analog signal processing elements and photosensitive elements on a single circuit board and can operate at rates of over 10,000 images per second. However, analog signal processing elements have significantly limited capabilities when compared to digital image analysis, and it is not yet clear whether such devices are suitable for automated industrial inspection.

Given the shortcomings of the array of optical detectors, and the shortcomings and limitations of current machine visual systems, systems and methods that use 2D imagers and digital image analysis to improve object detection and inspection in industrial production. Is urgently needed.

<p num="0030"> [Outline of Invention] The present invention is for automatic photoelectron detection and inspection of an object based on obtaining a digital image of a two-dimensional field of view where the object to be detected and inspected may be placed and analyzing and determining the image. Provide systems and methods. These systems and methods perform pattern analysis of the brightness of light reflected from vast areas, deal with many prominent features in objects, respond to production line transformations by software means, and are obscure and diverse objects. Handle the position of. They are cheaper, easier to set up, and operate much faster than traditional machine vision systems. These systems and methods also allow multiple perspectives on moving objects, operate without triggers, deliver properly tuned output signals, and provide other notable and useful capabilities that are clear to those skilled in the art. provide.</p><p num="0031"> The present invention is primarily intended for applications in which the object is moving continuously and provides certain important advantages in those examples, but it is also on conventional technical systems in applications where the object is stationary. It can be used to your advantage.</p><p num="0032"> One of the features of the present invention is a device called a visual detector, which can acquire and analyze continuous images at a higher speed than a vision system according to a conventional technique. Such a continuous image acquired and analyzed is called a frame. The speed at which a frame is acquired and analyzed is called the frame speed, which is fast enough to be shown in multiple consecutive frames as a moving object passes through the field of view (FOV). Since the object is moving somewhere in a continuous frame, it is placed at multiple positions within the FOV, so it can be viewed from multiple viewpoints and positions with respect to the illumination.</p><p num="0033"> Another feature of the present invention is a method called dynamic image analysis, in which the object inspects the object by acquiring and analyzing multiple frames in the field of view, and the evidence obtained from each of these frames is obtained. The result is derived comprehensively. This method offers significant benefits over traditional machine vision systems that make decisions based on a single frame.</p><p num="0034"> Furthermore, another feature of the present invention is a method of detecting an event that may occur in the visual field, which is referred to as visual event detection. When an event such as an object passing through the field of view occurs, visual event detection can be used to detect the object without the need for a trigger signal.</p><p num="0035"> Further features in the present invention will become apparent upon examination of the drawings and the contents detailed herein.</p><p num="0036"> One advantage of the methods and devices of the invention in which an object moves is that the inspecting person can move the object in various ways with respect to the eyes and light by considering the evidence obtained from multiple viewpoints and positions with respect to the illumination. Just as making more reliable decisions, visual detectors can make more reliable decisions than traditional visual systems.</p><p num="0037"> Another advantage is that objects can be reliably detected without a trigger signal, such as a photodetector. This reduces costs, is easy to install, and by changing the software of the visual detector, it can change the production line to another product without having to manually relocate the photodetector. it can.</p><p num="0038"> Another advantage is that the visual detector can track the position of the object as it passes through the field of view and determine the speed and time it will pass through a fixed reference point. The output signal is synchronized to this fixed reference point and other useful information about the object can be obtained as taught here.</p><p num="0039"> In order to obtain an image from multiple viewpoints, it is desirable that the movement of the object to be detected or inspected between consecutive frames in the field of view is negligible, usually no more than a few pixels. As taught here, it is generally desirable that the moving speed of the object does not exceed about a quarter of the FOV in each frame and is less than or equal to 5% of the FOV in normal embodiments. It is desirable that this be protected by maintaining a sufficiently fast frame time, not by slowing down the manufacturing process. In one example of the system, the frame speed is at least 200 frames per second, and in other examples the frame speed is at least 40 times the average speed at which the object passes in front of the visual detector.</p><p num="0040"> A typical system is taught to be able to acquire and analyze 500 frames per second. This system uses an ultrasensitive imager with much smaller pixels than a conventional vision system. With high sensitivity, the shutter time can be very short by using inexpensive LED lighting, and the image acquisition time is very short with a relatively small number of pixels. The imaging device works with a digital signal processor (DSP) that can receive and store pixel data at the same time as the analysis work. By using the method taught here and using appropriate software for the DSP, the analysis time of each frame can be kept within the time to read the next frame. The acquisition and analysis methods and equipment combine to provide the desired fast frame time. By carefully combining the capabilities of the imaging device, DSP, and lighting with the objectives of the present invention, a typical system can be significantly cheaper than a machine vision system using conventional technology.</p><p num="0041"> Methods of detecting visual events include finding evidence of whether an event is occurring or not by acquiring consecutive frames and analyzing each frame. When the detection of a visual event is used to detect an object without the need for a trigger signal, the analysis finds evidence that the object is in the field of view.</p><p num="0042"> In a typical method, the evidence is represented by a value called the object detection weight, which indicates the confidence that an object is in the field of view. The value may simply be a yes or no choice to indicate high or low confidence, a number indicating the range of confidence, or any information item indicating evidence. Examples of such numbers are so-called fuzzy logic values, which are described in more detail here. It should be noted here that no machine can make a perfect decision from a single image, and therefore make a decision based on incomplete evidence.</p><p num="0043"> When making an object detection, each frame is tested to determine if the evidence is sufficient to show that the object is in the field of view. If the value is simple yes or no, then if the value is "yes" then the evidence is considered sufficient. If it is a numerical value, whether the evidence is sufficient is judged by comparing the numerical value with the threshold value. A frame with sufficient evidence is called a valid frame. It should be noted here that what is sufficient evidence is ultimately defined by the person who sets up the visual detector based on an understanding of the particular application at hand. The visual detector automatically uses its definition to make a decision.</p><p num="0044"> When detecting an object, the high frame time of the visual detector produces a plurality of valid frames for each object passing through the field of view. These frames may not be strictly contiguous, but there may be viewpoints or other conditions where there is insufficient evidence that the object is in the field of view as it passes through the field of view. There is also sex. Therefore, it is desirable that the detection of an object starts when valid frames are confirmed and does not end until a number of invalid frames are confirmed. The number can be appropriately selected by the user.</p><p num="0045"> Once the effective frame has been identified in response to an object passing through the field of view, further analysis should be performed to determine if the object was indeed detected. In this additional analysis, the number of valid frames, the total of the object detection weights, the average of the object detection weights, and other valid frame statistics should be taken into account.</p><p num="0046"> The above examples of visual event detection are intended to be exemplary and not intended to be all-encompassing. Obviously, there are many ways to achieve the objectives of visual event detection within the ideas of the invention that will occur by the parties.</p><p num="0047"> The method of dynamic image analysis includes the acquisition and analysis of a plurality of frames for inspecting an object, and "inspection" means confirming certain information about the state of the object. In one embodiment of this method, the condition of an object includes whether it meets the inspection criteria appropriately selected by the user.</p><p num="0048"> A feature of the present invention is that dynamic image analysis is used in combination with visual event detection, so the effective frame selected by the visual event detection method was used when inspecting an object by the dynamic image analysis method. It is a thing. A feature of the present invention is that the frame used for dynamic image analysis can be acquired in response to a trigger signal.</p><p num="0049"> Each of these frames is analyzed to determine evidence that the object meets inspection criteria. In one practice, the evidence is in the form of a value, referred to as the passing score of the object, which indicates the level of confidence that the object meets the inspection criteria. Similar to object detection weighting, the value is indicated by a simple, yes / no choice, which represents a high or low confidence level, or a number such as a fuzzy logic value, which represents a range of confidence levels. , Or any information item that indicates evidence.</p><p num="0050"> The state of an object is determined, for example, by statistics of the passing score of the object, such as average or percentile values. The state is also determined by weighted statistics such as a weighted average or weighted percentile value using object detection weights, for example. Weighted statistics effectively put more weight on the evidence from a reliable frame that the object is actually within the field of view of that frame.</p><p num="0051"> Evidence for the detection and inspection of an object is obtained by examining information about the visible properties of one or more objects. Visible properties are a part of an object, and the amount, pattern, or other property of the emitted light indicates information that represents the existence, individuality, or state of the object. Light is emitted by any process or combination of processes, including, but not limited to, reflection, transmission, or direct refraction from sources inside or outside the object, or from sources inside the object.</p><p num="0052"> One aspect of the invention is a method of obtaining evidence, which includes object detection weights and passing scores for objects, and evidence is required for image analysis work in one or more regions of interest within each frame. To. In an embodiment of this method, the image analysis task calculates a measurement based on the pixel values in the region of interest, and the measurement responds to some of the appropriate features in the visibility characteristics of the object. The measured value is converted into a logical value by a threshold operation, and the logical value obtained from the region of interest is also used to show evidence of the frame. Logical values are binary or fuzzy logical values, and thresholds and logical joins are appropriately binary or fuzzy.</p><p num="0053"> In visual event detection, evidence of an object's position in the field of view is effectively defined by regions of interest, measurements, thresholds, logical connections, and other parameters detailed here, but they are collectively visual. Referred to as detector configurations, and they are selected by the user depending on the application of the present invention. Similarly, the configuration of the visual detector defines what constitutes sufficient evidence.</p><p num="0054"> In dynamic image analysis, evidence that an object meets inspection criteria is also effectively defined by the configuration of the visual detector.</p><p num="0055"> One aspect of the invention involves having information about the detection or inspection of an object and determining the outcome. The results can be reported to automated devices for a variety of purposes, including devices such as rejection mechanisms that act on the basis of reports. In one example, the result is an output pulse that is always transmitted each time an object is detected. In another example, the result is an output pulse that is transmitted only to objects that meet the detection criteria. In yet another example, the output pulse, which is useful for controlling the rejection actuator, is transmitted only to objects that do not meet the detection criteria.</p><p num="0056"> Another aspect of the invention is a method of transmitting an output signal that is synchronized with a time, a rotation coder count, or other event recorder that indicates when an object has passed a fixed reference point on the production line. Synchronized signals provide information about the position of objects in the manufacturing process and can be used as a useful tool for automated devices such as downstream rejection activators.</p><p num="0057"> The present invention is further fully understood from the following detailed description associated with the accompanying figures.</p>

<figref num="1">Shown is a conventional mechanical visual system used to inspect an object on a production line.</figref><figref num="2">Shows a schedule that describes the typical operating cycle of a traditional mechanical vision system.</figref><figref num="3">A conventional inspection of an object on a production line using a photodetector is shown.</figref><figref num="4">An embodiment of a visual detector for inspecting an object on a production line according to the present invention is shown.</figref><figref num="5">Use the Visual Event Detector Shows a schedule that describes a typical operating cycle for a visual detector.</figref><figref num="6">Here is a schedule that describes a typical operating cycle for a visual detector that uses a trigger signal.</figref><figref num="7">A schedule is provided that describes a typical operating cycle of a visual detector for continuous analysis of the manufacturing process.</figref><figref num="8">A high-level block diagram for a visual detector in a manufacturing environment is shown.</figref><figref num="9">A high-level block diagram of an embodiment of a visual detector is shown.</figref><figref num="10">Show the proper lighting arrangement for the visual detector.</figref><figref num="11">Shows the fuzzy logic elements used in embodiments to weigh and determine evidence, including determining whether an object is present and passing inspection.</figref><figref num="12">How the evidence deserves dynamic image analysis in embodiments will be described.</figref><figref num="13">In another embodiment, how dynamic image evidence is considered will be described.</figref><figref num="14">Demonstrates the organization of a set of software elements (eg, program commands on a computer-readable medium) used by embodiments to analyze frames, make decisions, sense inputs, and control output signals.</figref><figref num="15">Here are some of the typical configurations of visual detectors that can be used to detect typical objects.</figref><figref num="16">Here is another part of the configuration that corresponds to the typical setting in Figure 15.</figref><figref num="17">A method of analyzing a region of interest for measuring the brightness and contrast of variable characteristics will be described.</figref><figref num="18">A method of analyzing a region of interest for detecting a step edge will be described.</figref><figref num="19">A method of analyzing the region of interest for detecting the step edge will be further described.</figref><figref num="20">A method of analyzing a region of interest for detecting a ridge edge will be described.</figref><figref num="21">A method of analyzing the region of interest for detecting the ridge edge will be further described.</figref><figref num="22">Shows the graphical controls displayed on the human-machine interface (HMI) for the user to see and manipulate to set parameters for edge detection.</figref><figref num="23">A method for analyzing a region of interest for detecting a spot will be described.</figref><figref num="24">A method for analyzing a region of interest for detecting a spot will be further described.</figref><figref num="25">We will use HMI to construct the analysis and describe how to analyze the region of interest to track the position of an object in the field of view.</figref><figref num="26">HMI is used to construct the analysis, and a method of analyzing the region of interest for tracking the position of an object in the field of view is further described.</figref><figref num="27">HMI is used to construct the analysis, and a method of analyzing the region of interest for tracking the position of an object in the field of view is further described.</figref><figref num="28">In an example where the placement of the region of interest must be performed fairly along the boundaries of interest, how to use the HMI to construct the analysis and analyze the region of interest to track the position of objects in the field of view. explain.</figref><figref num="29">In the example of rotating and resizing an object, we will use HMI to construct the analysis and describe how to analyze the region of interest to track the position of the object in the field of view.</figref><figref num="30">Based on a ladder diagram, a widely used industrial programming language, we provide alternative representations for some of the structures of visual detectors.</figref><figref num="31">The timing chart used to explain how the visual detection output signal was synchronized is shown.</figref><figref num="32">An example is shown in which the time it takes for an object to pass a fixed reference point is measured, and the velocity of the object, pixel size calibration, and the distance and orientation of a distant object are also measured.</figref><figref num="33">Shows the output buffer used to generate synchronized pulses for downstream control of the actuator.</figref><figref num="34">Show the user a part of the HMI with the configuration of object detection parameters.</figref><figref num="35">Show the user a part of the HMI with the configuration of the object inspection parameters.</figref><figref num="36">Show the user part of the HMI the configuration of the output signal.</figref><figref num="37">One method for constructing an invention that performs a visual event detector when connected to a PLC will be described.</figref><figref num="38">One method of the invention for performing visual event detection for direct control of a rejection activator will be described.</figref><figref num="39">One method for constructing the invention for using a trigger signal when connected to a PLC will be described.</figref><figref num="40">One method of the invention for using a trigger signal for direct control of a rejection activator will be described.</figref><figref num="41">One method for constructing an invention for detecting and recording a defect image on the continuation web will be described.</figref><figref num="42">One method for constructing an invention for detecting an image of a defect on a continuous web for signal sorting will be described.</figref><figref num="43">One method for constructing an invention for detecting an image of a defect on a continuation web that provides signal synchronization will be described.</figref><figref num="44">One method for constructing an invention for detecting visual events without tracking the position of an object in the field of view will be described.</figref><figref num="45">Based on the example given by the user, the role used by the visual detector embodiment to learn the appropriate parameters will be described.</figref><figref num="46">Based on the example given by the user, the role used by the visual detector embodiment to learn the appropriate parameters will be further described.</figref><figref num="47">Explain the use of a phase-locked loop (PLL) to measure the presentation rate of an object and detect missing or extra objects on a production line that is present on the object at near slow speed.</figref><figref num="48">The operation of the latest software PLL used in the embodiment will be described.</figref>

[Detailed description of the invention] [Explanation of prior art] Figure 1 shows a conventional mechanical visual system used to inspect objects on a production line. Objects 110, 112, 114, 116 and 118 move from left to right on conveyor 100. Each object is expected to contain certain features, for example Level 120 and Hall 124. Incorrectly manufactured objects may impair one or more features, for example object 116 may have unintentional features, such as no holes. In many production lines, the operation of the conveyor is detected by a rotary encoder 180, which sends a signal 168 to a programmable logic controller (PLC) 140.

The object passes through the photodetector 130, which emits a ray of light 135 to detect the presence of the object. Trigger signals 162 and 166 are sent from the photodetector to the PLC 140 and the mechanical vision system 150. At the rising edge of the trigger signal 166, the visual system 150 takes an image of the object, inspects the image to determine if it has the expected features, and reports the inspection result to the PLC through signal 160.

At the rising edge of the trigger signal 162, the PLC records the time and / or the encoder count. When a PLC receives a test result from a visual system, it may do various things with the result as needed. For example, the PLC may control the rejection actuator 170 through signal 164 to remove the detection object 116 from the conveyor. Since the rejection actuator is generally downstream from the inspection point defined by the photodetector beam 135, the PLC must delay the signal 164 to the rejection actuator until the detection part is in front of the rejection actuator. Must be. This delay should be related to the trigger signal 162, ie the time recorded by the PLC and / or the count, as the time required by the visual system to complete the examination is usually variable. Is. The delay is appropriate if the conveyor is moving at a constant speed, otherwise an encoder is preferred.

Lighting is not shown in FIG. 1, but lighting is provided as needed by various methods of known technology.

In the example of FIG. 1, the object is moving continuously. There are also many applications such as making a manufacturing device object stationary in front of a visual system.

Figure 2 shows a schedule that illustrates the typical operating cycle of a traditional mechanical vision system. Illustrated are the operating steps of two typical objects 200 and 210. The operating cycle includes four steps: trigger 220, image acquisition 230, analysis 240 and report 250. During cycle 260, the visual system is on standby. The schedule is not shown in reality and the amount of time it takes for the shown steps will vary significantly from application to application.

The trigger 220 is some external event to the visual system, such as a signal from the photodetector 130 or a message from a PLC, computer, or other part of an automated device.

Image acquisition step 230 is acquired by exposing the photosensitive elements of a two-dimensional array, called pixels, to the light collected in the array by the lens for a short period of time, called integration time or shutter time. Each pixel measures the intensity of the light that hits it within the shutter time. The measured intensity value is converted to a digital number and stored in the memory of the visual system.

During analysis step 240, the visual system manipulates stored pixel values using known techniques to determine when an object is inspected. During reporting step 250, the visual system conveys information about the state of the object to a suitable automated device, such as a PLC.

Figure 3 shows a conventional inspection of an object on a production line using a photodetector. Conveyor 100, objects 110, 112, 114, 116, 118 and labels 120, hall 124, encoder 180, rejection activator 170 and signals 164 and 168 are illustrated in FIG. The first photodetector 320 with the beam 325 is used to detect the presence of an object. The second photodetector 300 with the beam 305 is located relative to the first photodetector 320 so that the presence of the label 120 can be detected. A third photodetector using the beam 315 is located relative to the first photodetector 320 so that the presence of the hall 124 can be detected.

The PLC340 samples signals 330 and 333 from photodetectors 300 and 310 on the rising edge of photodetector 330 to signal 336 to determine the presence of characteristics 120 and 124. If one or two properties are missing, signal 164 is sent to rejection actuator 170 to delay appropriately based on encoder 180 and remove the detected object from the conveyor.

[Basic operation of the present invention] FIG. 4 shows an embodiment of a visual detector according to the invention for inspecting an object on a production line. Conveyor 100 transports a relative motion object between the object and the field of view of the visual detector 400. Objects 110, 112, 114, 116, 118 and label 120, hall 124, encoder 180, rejection actuator 170 are illustrated in FIG. The first photodetector 320 with the beam 325 is used to detect the presence of an object. The visual detector 400 detects the presence of an object from its appearance and inspects it based on an appropriate inspection standard. If an object is detected, the visual detector sends a signal 420 to reject the actuator 170 to remove the object from the conveyor stream. The encoder 180 sends the signal 410 to the visual detector, and the object passes through some fixed fictitious reference point 430, called the recording point, to take steps to prevent a normal delay of the signal 420 from the encoder count. used. If the encoder is not used, the delay is based on time instead.

In an alternative embodiment, the visual detector sends a signal to the PLC for a variety of purposes, including controlling a rejection activator.

In other embodiments, it is suitable for ultrafast applications, or where visual detectors cannot reliably detect the presence of an object, and photodetectors are used to detect the presence of an object and are used for that purpose. Send a signal to the vessel.

In yet another embodiment, there are no discrete objects, but the physical distribution continuously passes through the visual detector, for example the web. In this case, the object is continuously inspected and the signal is timely sent by a visual detector to a device such as a PLC.

When a visual detector detects the presence of discrete objects by appearance, it is said to operate in visual event detection mode. When a visual detector uses an external signal, such as a photodetector, to detect the presence of discrete objects, it is said to operate in external trigger mode. When the visual detector continuously inspects an object, it is said to operate in continuous analysis mode.

Figure 5 shows a schedule that illustrates the typical operating cycle of a visual detector operating in visual event detection mode. A box labeled "c", such as box 520, indicates image acquisition. A box labeled "a", such as box 530, indicates image acquisition. Since the acquisition "c" of the next image preferably overlaps with the analysis "a" of the current image, the analysis step 530 analyzes the image captured in the acquisition step 520. This schedule shows that the analysis takes less time than the acquisition, but in general, the analysis will be shorter or longer than the acquisition because it depends on the details of the application.

When acquisition and analysis overlap, the rate at which the visual detector can capture and analyze the image is determined by the acquisition time and the length of the analysis time. This is the "frame speed".

The present invention ensures that an object is detected without a trigger signal given by the photodetector 130. Referring to FIG. 4, there is no trigger signal indicating the presence of an object, and in FIG. 5, there is no corresponding trigger step as in step 220 of FIG.

Also referring to FIG. 5, part 500 of the schedule corresponds to the inspection of the first object and includes the acquisition and analysis of seven frames. The second part 510 corresponds to the inspection of the second object and includes five frames.

Each analysis step initially considers evidence that the object is present. Frames with sufficient evidence are called valid. The valid frame analysis steps are indicated by thick boundaries, such as analysis step 540. In an embodiment, if a valid frame is found, the inspection of the object begins and if a continuous invalid frame is found, it ends. In the example of FIG. 5, the inspection of the first object begins with the first valid frame corresponding to analysis step 540 and ends with two consecutive invalid frames corresponding to analysis steps 546 and 548. It should be noted here that in the first object, a single invalid frame corresponding to analysis step 542 is not enough to finish the inspection.

When the inspection of the object is completed, for example at the end of analysis step 548, the state of the object is determined based on the evidence obtained from the valid frame. In an embodiment, if an inadequate number of valid frames is found and is considered to be inadequate evidence that the object actually existed, the operation continues as if no valid frames were found. Otherwise, the object is determined to have been detected and the evidence from the valid frame is determined to determine its situation, for example, pass or fail. Within the scope of the present invention, some of the various methods of detecting an object and determining the situation are described below, and many others can be considered by one of ordinary skill in the art.

Once an object is detected and determined, it may be reported to a device such as a PLC using signals familiar to those skilled in the art. In such cases, a reporting step similar to step 250 in Figure 2 will appear in the schedule. The example of FIG. 5 corresponds to the setting shown in FIG. 4 that the visual detector is used to control the downstream rejection activator 170 through signal 420. By considering the position of the object in the effective frame, if it passes through the field of view, the field of view detector estimates the recording times 550 and 552 that the object passes through the reference point 430. It should be noted here that when the encoder 180 is used, the recording time is actually the encoder count, and the reader understands that the time and the count are used alternately. The report 560, with one pulse of appropriate duration to the rejection actuator 170, is published after an exact delay 570 in time or an encoder count from a recording time of 550.

It should be noted here that the report 560 can often be delayed significantly past the subsequent inspection of the object, such as the 510. Visual detectors use the well-known first-in first-out (FIFO) buffering technique to hold reports in a timely manner.

When the inspection of the object is completed, the visual detector may enter idle step 580. Such steps are optional, but may be preferable for several reasons. If the maximum object velocity is known, there is no need to look for the object just before the new object arrives. The idle step will extend the life of the lighting system by eliminating the opportunity to detect defective objects and keeping signals away during the idle step if the object cannot arrive.

Figure 6 shows a schedule that illustrates the typical operating cycle of a visual detector operating in external trigger mode. Trigger step 620, which is similar in function to the conventional trigger step 220, initiates inspection of the first object 600. A series of image acquisition steps 630, 632, 634 and corresponding analysis steps 640, 642, 644 are used for dynamic image analysis. In visual event detection mode, it is preferable that a frame rate object is high enough to move a small portion of the field of view, often with only a few pixels per frame, between consecutive frames. After a fixed number of frames, the number selected based on the details of the application is used by the evidence obtained from the analysis of the frame to make the final decision on the situation of the object, and it is in one embodiment. Is being supplied to the automated device in reporting step 650. Following the reporting step, idle step 660 is input until the next trigger step 670, which begins the inspection of the second object 610.

In another embodiment, the reporting step is delayed in a manner similar to that shown in FIG. In this embodiment, the recording time 680 is the time (or encoder count) corresponding to trigger step 620.

Figure 7 shows a schedule that illustrates the typical operating cycle of a visual detector operating in continuous analysis mode. Frames are continuously acquired, analyzed and reported. One such cycle includes acquisition step 700, analysis step 710, and reporting step 720.

[Example device] FIG. 8 shows a high level block diagram for a visual detector in a manufacturing environment. The visual detector 800 is connected to a suitable automatic device 810, which may include a PLC, a rejection activator, and / or a photodetector with a signal 820. The visual detector may also use the signal 840 to connect to a human-machine interface (HMI) 830 such as a PC or mobile terminal. During normal production use, HMI is used for configuration and monitoring and may be removed. The signal is executed in a format and / or protocol that meets any conditions and is transferred in a wired or wireless format.

FIG. 9 shows a block diagram of an embodiment of the visual detector. The Digital Signal Processor (DSP) 900 launches software to control acquisition, analysis, reporting, HMI communication, and other appropriate functions required for visual detectors. The DSP900 connects to memory 910, which includes fast random access memory for programs and data, and non-volatile memory for holding program and configuration information when power is removed. The DSP is also connected to the I / O module 920, which supplies the HMI interface 930, the lighting module 940 and the imaging device 960, which are the signals of the automatic device. The lens 950 images on the photosensitive member of the image pickup apparatus 960.

The DSP900 is a device for digital computing, information storage, and connection to other digital elements, including but not limited to general purpose computers, PLCs, or microprocessors. The DSP900 is inexpensive, but preferably fast enough to handle high frame rates. It is more preferable that pixel data can be simultaneously received from the image pickup apparatus and stored in the image analysis.

In the embodiment of FIG. 9, the DSP900 is an ADSP-BF531 produced by Analog Devices in Norwood, Massachusetts. The parallel peripheral interface (PPI) 970 of the ADSP-BF531DSP900 receives pixel data from the imager 960 and sends the data to memory controller 974 through direct memory access (DMA) channel 972 for storage of memory 910. Under proper software control, the use of PPI970 and DMA972 causes simultaneous image acquisition in other analyzes performed by DSP900. The software instructions for controlling with PPI970 and DMA972 are ADSP-BF533 Blackfin Processor Hardware Reference (Part No. 82-002005-01) and Blackfin Processor Instruction. Performed by one of ordinary skill in the art according to the program instructions contained in SetReference (Part No. 82-000410-14) and incorporated by reference to both here. It should be noted here that the devices of ADSP-BF531, competing ADSP-BF532 and ADSP-BF533 have the same program instructions and can be exchanged in this embodiment to obtain an appropriate price / execution transaction. To be used.

The preferred high frame rate by the visual detector recommends the use of an imaging device rather than the imaging device used in conventional technical visual systems. The imaging device preferably uses inexpensive lighting and is unusually light sensitive so that it can be operated with a very short shutter time. It is even more preferable to be able to digitize and transfer pixel data with a DSP that is even faster than conventional technical systems. It is still preferable to have a global shutter, which is not expensive.

These objects may be addressed by choosing an imager with higher photosensitivity and lower resolution than the imager used in conventional technology systems. In the embodiment of FIG. 9, the imaging device 960 is an LM9630 manufactured by National Semiconductor, Santa Clara, California. The LM9630 has 128x100 pixels, a total of 12800 arrays, about 24 times less than a typical traditional technology system. Each pixel is 20 square microns in size with high photosensitivity. The LM9630 can deliver 500 frames per second if a set with a shutter time of 300 microseconds is sensitive enough (in most cases) to make it use LED lighting and make it 300 microseconds. This resolution is considered to be much too low for a visual system, but is sufficient for the feature detection task of the present invention. The electrical interface and software control of the LM9630 will be carried out by one of ordinary skill in the art in accordance with the instructions contained in the LM9630 data sheet for Rev1.0 of January 2004, which will be incorporated by reference to both here.

The illumination 940 is not expensive, but preferably bright enough for a short shutter time. In an embodiment, a row of bright red LEDs operating at 630 nanometers uses, for example, the HLMP-ED25 manufactured by Agilent Technologies. In another embodiment, a bright white LED is used to perform the preferred lighting.

In the embodiment of FIG. 9, the I / O module 920 supplies output signals 922 and 924 and input signals 926. Such an output signal is used to supply signal 420 (FIG. 4) for control of the rejection activator 170. The input signal 926 is used to supply an external trigger.

An image acquisition device in the following terms means acquiring and storing a digital image. In the embodiment of FIG. 9, the image acquisition device 980 comprises a DSP900, an image pickup device 960, a memory 910, and associated electrical interfaces and software instructions.

An analyzer of the following terms provides a means of analyzing digital data, including but not limited to digital images. In the embodiment of FIG. 9, the analyzer 982 comprises a DSP900, a memory 910, and associated electrical interfaces and software instructions.

The output signal of the following terms provides a means for producing an output signal that corresponds to the analyzer. In the embodiment of FIG. 9, the output signal 984 comprises an I / O module 920 and an output signal 922.

It will be appreciated by the parties that there are many alternating sequences, devices, software instructions used within the scope of the invention to implement the image acquisition device 980, analyzer 982, output signal device 984. ..

Various trade-offs may be made in order to efficiently operate the apparatus according to the present invention for a specific purpose. Consider the following definition.

b is the percentage of FOV used by some of the objects that contain visible features that are inspected and determined by selecting the visual magnification of the lens 950 to take full advantage of the available resolution of the imager 960. e is the percentage of FOV used as the error vote difference, n is the preferred minimum number of frames in which each object is typically seen, s is generally the space between objects as a product of FOVs, depending on the manufacturing conditions, p is the presentation rate of the object, which is generally determined by the manufacturing conditions. m is the maximum ratio of FOVs that an object will move between consecutive frames, chosen based on the above values. r is the minimum frame rate chosen based on the above values. From these definitions, it is shown as follows.

<maths num="1"><img id="000002" he="12" wi="156" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

<maths num="2"><img id="000003" he="10" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

To make good use of the available resolution of the imager, b should be at least 50%. For dynamic image analysis, n must be at least 2. Therefore, it is more preferred that the object move less than about 1/4 of the field of view between continuous frames.

In embodiments, reasonable values may be b = 75%, e = 5%, n = 4. This implies that the frame rate is chosen, i.e. m 5%, as the object moves less than or equal to about 5% in FOV between frames. When the manufacturing condition is s = 2, the frame rate r needs to multiply the object presentation rate p by at least about 40 times. To handle a 5Hz object presentation rate, it is a fair and typical industrial manufacturing, with a preferred frame rate of at least about 200Hz. This rate can be leveraged to use the LM9630 with a maximum shutter time of 3.3 ms, as long as the image analysis is arranged to adapt within a 5 ms frame period. It is possible to achieve this rate using imaging devices, including up to about 40,000 pixels, using available techniques.

With the same embodiment and a higher object presentation rate of 12.5 Hz, the preferred frame rate is at least about 500 Hz. The LM9630 can handle this rate by using shutters up to 300 microseconds.

In another embodiment, m 2%, so the fair value should be b = 75%, e = 15%, n = 5. At s = 2 and p = 5Hz, the preferred frame rate is also at least about 500Hz.

Figure 10 shows the proper lighting arrangement for the visual detector. In an embodiment, 18 LED rings 1000, including the conventional LED 1010, are surrounded by a lens 1030. The ring is divided into 6 rows of 3 LEDs, including row 1020, which can be controlled independently. Independent control of columns adapts the lighting direction according to the needs of a given application.

In an alternative embodiment, a rectangular array of 16 LEDs, including typical LEDs, 1050 surrounds the lens with looseness. The array is divided into four columns as shown, contained in column 1070 of the example.

According to the present invention, the ability of dynamic image analysis can be increased by controlling the LEDs, as continuous frames are acquired while changing the illuminated columns. By considering the evidence obtained from frames illuminated from different directions, it enables reliable detection characteristics that are difficult to detect in fixed lighting directions. Correspondingly, the present invention makes use of changing the directional illumination in a moving object for analysis.

[Fuzzy logic decision making] FIG. 11 shows fuzzy logic elements used in embodiments to weigh and determine evidence, including determining whether an object is present and passing inspection.

A fuzzy logic value is a number between 0 and 1 that indicates an estimate of the confidence that a particular state is true. A number of 1 indicates a high degree of confidence that the state is true, a value of 0 indicates a high degree of confidence that the state is false, and an intermediate value indicates an intermediate level of trust.

A more familiar binary logic is a subset of fuzzy logic whose confidence values are limited by 0 to 1. Therefore, the embodiment described here using fuzzy logic can use these values replaced by an equivalent binary logic method or device and use them as another binary logic value in a fuzzy logic method or device. it can.

A fuzzy logic value is obtained by using a fuzzy threshold value, just as a dual value logic value is obtained from an original measurement value by using a threshold value. Referring to FIG. 11, Graph 1100 shows the fuzzy threshold. The X-axis 1110 indicates the original measurement, the f-axis 1114 indicates the fuzzy logic value, the range of x is all possible original measurements, and the range of f is 0 f 1. ..

In an embodiment, the fuzzy threshold comprises two numbers, represented by the x-axis, the low threshold t01120, and the high threshold t11122, corresponding to points on functions 1124 and 1126. The fuzzy threshold can be defined as follows.

<maths num="3"><img id="000004" he="16" wi="160" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

It should be noted here that this function works well when t1 &lt;t0. Other features can also be used as fuzzy thresholds, such as sigmoids.

<maths num="4"><img id="000005" he="14" wi="160" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

t and σ are threshold parameters. In embodiments where simplicity is the goal, conventional binary thresholds can use binary logical values as a result.

Fuzzy decision making is based on fuzzy versions of AND1140, OR1150, NOT1160. A fuzzy AND of two or more fuzzy logic values is the minimum value, and a fuzzy OR is the maximum value. The fuzzy NOT of f is 1-f. Fuzzy logic is identical in binary notation if the fuzzy logic value is limited by 0 or 1.

In embodiments, whenever a truth determination is required, a fuzzy logic value is considered true if it is at least 0.5 and false if it is 0.5 or less.

It will be clear to the parties that there is no criticism of the values 0 and 1 in relation to the fuzzy logic here. Use as an intermediate value to indicate an intermediate level of trust, any number to indicate high confidence that the state is true, and a different number to indicate high confidence that the state is false. Can be done.

[Dynamic image analysis] FIG. 12 shows how evidence deserves dynamic image analysis in embodiments. In this embodiment, the following two decisions, called basic decisions, must be made. 1. Is the object or the visible function of the object located within the visible range? 2. In that case, what is the state of the object?

The information that makes up the evidence that an object is in a visible range is called object detection weighting. The information that constitutes evidence of the state of an object is called the object passing score. In various embodiments, the condition of the object indicates whether the object meets the inspection criteria appropriately determined by the user. In the following, an object that meets the inspection criteria may be regarded as "passed inspection".

FIG. 12 shows two plots, the object detection plot 1200 and the object pass plot 1202. The horizontal axis of the two plots represents the frame sequence number i, and each frame is represented by a vertical line such as model line 1204.

In the embodiment of FIG. 12, the object detection weight is a fuzzy logic value d._iIs evidence that the object is located in the FOV of frame i, and is calculated by the vision detector at each frame using the method described below. Object passing score is fuzzy logic value p_iIs evidence that the object meets the appropriate inspection criteria in frame i, and is calculated by the vision detector at the selected frame using the method described below. The vertical axis of the object detection plot 1200 is d_iThe vertical axis of the object pass plot 1202 represents pi.

In the example of FIG. 12, d_iA frame with 0.5 is considered to be valid (see the description in Figure 5 above for a description of the valid frame). For reference, d_iLine 1230 is drawn with = 0.5. Object detection weights and passing scores for valid frames are drawn with black circles, for example points 1210 and 1212, and invalid frames are drawn with white circles, for example points 1214 and 1216. In the embodiment of FIG. 12, the object pass weight is calculated for all frames regardless of whether it is considered "valid", but the object pass weight is calculated only for the effective frame. There are also examples.

In the example of FIG. 12, all valid frames correspond to the inspection of one object, and as described in the description of FIG. 5 above, the isolation invalid frame 1220 does not interrupt the inspection.

In one embodiment, it is determined that the object is detected based on whether or not the number of effective frames exceeds a certain threshold value. In another embodiment, the object is determined to have been detected based on whether the sum of the object detection weights for the entire effective frame exceeds a certain threshold. These thresholds are set accordingly for a given application (see Figure 34).

In the embodiment of FIG. 12, it is determined that the objects have passed the inspection of whether or not the weighted average of the object passing points is at least 0.5, which is weighted by the corresponding object detection weights. More precisely, the object passes the inspection if:

<maths num="5"><img id="000006" he="20" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

Here the total exceeds all valid frames. The effect of this formula is not to average the passing scores of the object, but to weight each score based on the reliability that the object really existed in the corresponding frame.

In an alternative embodiment, if the object passing score average is at least 0.5, the object is judged to pass the inspection. This is equal to the weighted average where all weights are uniform.

In the example of FIG. 12, the weighted average of the object passing points is about 0.86, which is shown on the coordinates by line 1240. The number of valid frames is 11, and the total object detection weight is about 9.5. In this example, the object is detected and passes the inspection.

FIG. 13 shows how dynamic image evidence is considered in another embodiment. In this example, the characteristics of the object being inspected are confusing, and when the object moves through the field of view, mainly when the perspective of the display and lighting is just right, a few effective frames such as frames 1300 and 1310 Appears with the reliability of. In these few frames, the object should pass the inspection as long as there is sufficient evidence and its features are present. In this scenario, it is not possible to know in advance which valid frames will have this evidence. Therefore, the weighted average of passing scores does not fit in this case. An alternative is to pass the object if the passing score exceeds the threshold in any of these valid frames, but this alternative may allow the object to pass based on too little evidence. In the embodiment of FIG. 13, the weighted percentile method is used.

The weighted percentile method is based on the fraction Q (p) if the passing score is at least p.

<maths num="6"><img id="000007" he="22" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

The object is judged to pass if Q (p) is at least some threshold t. In the embodiment of FIG. 13, p = 0.5 and plotted as line 1320. In this case, a suitable threshold t would be 10%.

Useful behavior is obtained using different values of t. For example, if t = 50%, the object is determined to have a weighted median of at least p in order to pass the inspection. The weighted median is similar to the weighted average, but has better properties in some cases.

For example, for high values such as t = 90%, the object will only be judged to pass the inspection if the passing score is at least p and the overwhelming majority of weights correspond. If t = 100%, the object will be judged to pass the inspection only if the passing score of all valid frames is at least p. Any valid frame means having a frame with a passing score of at least p, and if Q (p) is greater than zero, the object will also be judged to pass the inspection.

In other useful variations, if the passing score is at least p, not a fraction of the total weight, the object is judged to pass the test based on the total weight.

In the alternative embodiment, the percentile method is used based on the number of frames with a passing score of at least p. This is comparable to the weighted percentile method when all weights are equal. The above method of examining evidence to determine whether an object has been detected and whether the object has passed the inspection is intended as a useful embodiment, but methods that can be used in the area of the present invention. It does not limit. For example, the typical constant 0.5 used above may be replaced by any suitable value. One of ordinary skill in the art will come up with many more methods for dynamic image analysis.

[Software element of the present invention] FIG. 14 shows the organization of a set of software elements (eg, program instructions on a computer-readable medium) used by embodiments for frame analysis, judgment, input sensing, and output signal conditioning. Elements may be executed using the class hierarchy of traditional object-oriented programming languages such as C ++ so that each element corresponds to a class. However, any programming technique and / or language that meets the criteria can be used to perform the processes described herein.

As illustrated, a class with a dotted border, like gadget class 1400, does not exist independently and is an abstract base class used to build concrete derived classes like locator class 1420. Is. The solid-lined classes represent dynamic objects that can be created and destroyed using the HMI 830 when the user needs to configure the application. Dashed-lined classes, such as input 1450, represent static objects associated with a particular hardware or software resource. Static objects are always present and cannot be created or destroyed by the user.

Since all classes are derived from gadget class 1400, all objects that are instances of the class shown in Figure 14 are a type of gadget. In an embodiment, each gadget is 1. Have a name that the user can choose 2. It has a logical output (fuzzy theoretical value) that can be used as a logical input for other gadgets to control the judgment and output signals. 3. Has a set of parameters that the user can configure to identify the operation 4. Has one such parameter that can be used to invert the logical output (ie fuzzy NOT) 5. It is executable and its parameters, logical inputs if applicable, logical outputs are updated based on the contents of the current frame for certain gadgets, and may cause side effects such as setting output signals. ..

Frame analysis work involves running each gadget once, in a fixed order, to ensure that all logical inputs to the gadget have been updated before running the gadget. In some embodiments, the gadget does not run during frames when its logical output is not needed.

Photo class 1410 is the base class for all gadgets whose logical output depends on the content of the current frame. These classes are the classes that actually perform image analysis. Each photo measures certain features of the current frame's region of interest (ROI). ROI corresponds to the visibility function of the object to be inspected. This measurement is called the analog output of the photograph. The logical output of a photo is calculated from an analog output using a fuzzy threshold that exists in a set of parameters that can be configured by the user, called the sensitivity threshold. The logical output of a photograph can be used to provide evidence used in making decisions.

Detector class 1430 is a basic class of photography whose primary purpose is to provide evidence in making measurements and decisions in ROI. In embodiments, all detector ROIs are circular. A circular ROI simplifies execution because it doesn't have to handle rotations, and it simplifies what the user has to learn because there is only one form of ROI. Detector parameters include position and ROI diameter.

The brightness detector 1440 measures the weighted average and percentile brightness in ROI. The contrast detector 1442 measures the contrast at the ROI. Edge detector 1444 measures the area where the ROI looks like an edge in a particular direction. Spot detector 1446 measures the area where the ROI looks like a hole-like round feature. Template detector 1448 measures the area in which the ROI appears to be a user-selected, pre-trained pattern. The operation of the detector is further described below.

The locator class 1420 represents a photo with two main purposes. The first is to generate a logical output that can provide evidence in making a decision, which can be used like any other detector. Second, by locating an object in the field of view of the visual detector, the ROI position of other photographs can be moved to track the position of the object. Any locator can be used for one or both purposes.

In an alternative embodiment, the locator searches for edges in a one-dimensional region within the frame. The search direction is perpendicular to the edge and is within the parameters set by the user. The analog output of the locator is similar to the analog output of the edge detector. The locator is further described below.

Input class 1450 represents an input signal sent to a visual detector, such as an external trigger. Output class 1452 represents an output signal from a visual detector that may be used to control a rejection actuator. For each physical input, such as the typical input signal 926 (Figure 9), there is one static instance of the input class, and for each physical output, such as the typical output signals 922 and 924. And there is one static instance of the output class.

Gate-based class 1460 makes fuzzy logic decisions. Each gate has one or more logical inputs that can be connected to the logical outputs of other gadgets. Each logical input can be inverted (fuzzy NOT) using user-configurable parameters. AND gate 1462 performs a fuzzy AND operation and OR gate 1464 performs a fuzzy OR operation.

Judgment class 1470 is the basic class of two static objects, object detection judgment 1472 and object acceptance judgment 1474. Judgment performs dynamic image analysis by making the main judgment and examining the evidence for continuous frames. Each decision has a logical input for the user to connect the logical output of the photo, or more typical gate, which gives a logical coupling of the gadget, which is usually the photo and another gate.

The object detection determination 1472 determines whether an object has been detected, and the object acceptance determination 1474 determines whether the object passes the inspection. The logical input to the object detection determination provides the weight of the object detection for each frame, and the logical input to the object acceptance determination provides the object passing score for each frame.

The logical output of the object detection decision provides a pulse indicating when the decision was made. In a mode of operation referred to as "output during process", a pulse rise occurs at the start of inspection of the object, for example at the end of analysis step 540 in FIG. 5, and a fall occurs at the end of analysis step 548, for example. Occurs when the inspection of an object is completed, such as when. In another mode, "output at end", a pulse rise occurs when the inspection of the object is completed, for example at the end of analysis step 548 in FIG. Occurs shortly after, like at the end.

The logical output of the object pass judgment provides a level indicating whether the recently inspected object has passed. The state of the level changes when the inspection of the object is completed, for example at the end of analysis step 548.

Figure 15 shows an example of how photographs can be used to inspect objects. FIG. 15 represents an image of object 110 (from FIG. 1), which, along with label mechanism 120 and hole mechanism 124, has superimposed graphics representing photographs, which are displayed on the HMI 830 for user viewing and manipulation. .. The display of images and superimposed graphics on the HMI is called image display.

FIG. 15 represents an image display showing an object 1500 containing labels 1510 and holes 1512. The object in this example contains 6 visible features to be inspected and corresponds to 2 locators and 4 detectors as described below.

The locator 1520 is used to detect and locate the top edge of an object, and another locator 1522 is used to detect and locate the right edge.

Luminance detector 1530 is used to help detect the presence of an object. In this example, the background is brighter than the object, the sensitivity threshold is set to distinguish between the two brightness levels, and the logical output is inverted to detect a darker object rather than a lighter background. ..

Further, as explained below, the locators 1520 and 1522, as well as the luminance detector 1530, both provide the evidence necessary to determine that an object has been detected.

The contrast detector 1540 is used to detect the presence of holes 1512. In the absence of holes, the contrast is very low, and in the presence of holes, the contrast is very high. Spot detectors are also used.

Edge detector 1560 is used to detect the presence and location of label 1510. If the label is not present, is in the wrong position horizontally, or is rotating significantly, the analog output of the edge detector will be very low.

The brightness detector 1550 is used to check if the label is correct. In this example, the correct label is white and the incorrect label is dark.

Luminance detector 1530, contrast detector 1540, luminance detector so that as the object moves from left to right through the field of view of the visual detector, the locator 1522 tracks the right edge of the object and is in the correct position with respect to the object. Relocate the 1550 and edge detector 1560. The locator 1520 repositions the detector based on the position of the top edge of the object and corrects any vertical variation of the object in the field of view. In general, the locator can be placed in any position.

Users can manipulate the photo within the image display using known HMI technology. The photo can be selected by clicking the mouse and its ROI can be moved, resized, and rotated by dragging. Additional operations related to the locator are described below.

FIG. 16 shows a logical representation, including a wiring diagram corresponding to the configuration example of FIG. The wiring diagram shows all the gadgets used to detect events and interfaces to the automated equipment, and the connections between the logical inputs and outputs of the gadgets. The wiring diagram is displayed on the HMI830 for the user to see and operate. The display of gadgets and their logical interconnection in the HMI is called the logical display.

Further referring to the wiring diagram of FIG. 16, the locator 1620 named "top" is connected to the AND gate 1610 by a wire 1624, corresponding to the locator 1522 in the image display of FIG. Similarly, the "side" locator 1622 corresponds to the locator 1530 and the "box" detector 1630 corresponds to the luminance detector 1530 and is also wired to the AND gate 1610. As shown by the small circle 1632, and as described above, the logical output of the "box" detector 1630 is inverted to detect darker objects against a bright background.

The logic output of AND Gate 1610 represents the level of reliability that the top edge of the object has been detected, the right edge of the object has been detected, and the background has not been detected. If the reliability is high and all three conditions are true, the object itself is detected because of the high reliability. The logic output of the AND gate 1610 is wired to the object detection determination 1600 because it is used as a weight for object detection for each frame.

Since the logical input to the object detection determination in this case depends on the current frame, the visual detector is operating in the visual event detection mode. To operate in external trigger mode, the input gadget is wired to object detection. Nothing is wired to the object detection to operate in continuous analysis mode.

The choice of gadget to wire to object detection is made by the user based on the knowledge of the application. In the examples of FIGS. 15 and 16, the user may decide that detecting only the top and right edges is not sufficient to confirm the presence of an object. It should be noted here that the locator 1522 can react to the left edge of the label as well as to the right edge of the object, and perhaps at this point, in the production cycle, the locator 1520 is the background. It is possible that he found another end in. By adding the detector 1530 and using the AND gate 1610 to meet all three conditions, object detection will be reliable.

In the wiring diagram, the contrast detector "Hall" 1640 corresponds to the contrast detector 1540, the luminance detector "label" 1650 corresponds to the luminance detector 1550, and the edge detector "label edge" 1660 corresponds to edge detection. Wired to AND gate 1612, corresponding to vessel 1560. The logical output of AND gate 1612 represents the level of reliability that all three image characteristics have been detected and is wired to the object pass verdict 1602 to provide said object pass points for each frame.

The logic output of the logic detection decision 1600 is wired to the AND gate 1670. The logic output of the object pass judgment 1602 is inverted and wired to the AND gate 1670. Since the object detection judgment is set to the "output at end" mode, there is a pulse in the logical output of the object detection judgment 1600 after the object is detected and the inspection is completed. Since the logical output of the object pass 1602 is inverted, it exists on the logical output of AND gate 1670 only if the object does not pass the inspection. The logical output of AND Gate 1670 is wired to an output gadget 1680 named "Reject" that controls the output signal from a visual detector that can be directly connected to the rejection actuator 170. The output gadget 1680 is set by the user to provide the appropriate delay 570 required for the downstream rejection activator.

Users can operate the gadget within the logical display by using known HMI technology. Gadgets can be selected by clicking the mouse, their positions can be moved by dragging, and wires can be created by dragging and dropping.

To make it easier for users to understand the visual detector, gadgets and / or wires can change their visual appearance to indicate fuzzy logic values. For example, gadgets and / or wires can be displayed in red if the logical value is less than 0.5, and green otherwise. In FIG. 16, wires 1604 and 1672 are drawn with broken lines to indicate a logical value of less than 0.5, indicating that other wires, such as wire 1624, have a logical value equal to or greater than 0.5. It is drawn with a solid line to show it.

Those skilled in the art will recognize that a variety of objects can be detected and inspected with proper gadget selection, configuration, and wiring. Those skilled in the art will also recognize that the gadget class hierarchy is only one of many software technologies used to implement the present invention.

[Detector image analysis method] FIG. 17 illustrates the method used in the luminance detector and the contrast detector. In one embodiment of the luminance detector, the analog output is an average intermediate level within the ROI. In an embodiment, a positive weighted kernel 1700 is created for the size and shape of the ROI, and the analog output A is a weighted average intermediate level.

<maths num="7"><img id="000008" he="20" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

wi is the i-th weight and zi is the corresponding pixel intermediate level. In the embodiment of FIG. 17, the weights estimate a Gaussian function of the distance r from the center of the kernel to the center of each weight.

<maths num="8"><img id="000009" he="14" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

Pixels near the center are weighted higher than pixels near the edges. One of the advantages of a centrally weighted luminance detector is that if the luminance characteristics are near the edge of the detector's ROI, a small change in its position will not have a large change in the analog output. Is. In Figure 17, a = 99, but any appropriate value is used. The value σ is set based on the kernel diameter d.

<maths num="9"><img id="000010" he="13" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

In the embodiment of FIG. 17, b = 1.0.

In another embodiment, the analog output is defined by the function C (q), whose intermediate levels are as follows:

<maths num="10"><img id="000011" he="20" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

q is the percentile selected by the user. C is an intermediate level inverse cumulative weighted generalized function. The various useful values for q are listed in the table below.

<tables num="1"><img id="000012" he="42" wi="160" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></tables>

In some embodiments of the contrast detector, the analog output is an intermediate level standard deviation within the ROI. In an exemplary embodiment, a column of positive weights 1700 can be used to calculate the weighted standard deviation as follows.

<maths num="11"><img id="000013" he="26" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

In yet another embodiment, the analog output is calculated by the following equation.

<maths num="12"><img id="000014" he="8" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

However, the q value is user-selectable. Useful values are q_hi = 0.95, q_lo= 0.05.

FIG. 18 shows a method for performing an edge detector that detects step edges. Step kernel 1800 is created for the size and shape of the ROI and the given orientation of the edges. In step kernel 1800, the ROI is a circle with a diameter of 12 pixels and the edge orientation is 15 degrees from the horizontal. Step kernel 1800 is an approximation of the first derivative of a Gaussian function with a distance t from the end to the center of each weight.

<maths num="13"><img id="000015" he="17" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

In FIG. 18, a = 99, but any suitable value can be used. As one embodiment, b = 0.5 is used in equation (9).

k_iStep kernel 1800 with values is the desired step edge template e_iAnd positive weight w_iIs considered to be the product of.

<maths num="14"><img id="000016" he="24" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

It should be noted here that the desired step edge template value e_iIs k_iWhen&gt; 0, it is +1 and corresponds to the black area on the white background of step kernel 1800, and k_iWhen &lt;0, it is -1, which corresponds to the black area of step kernel 1800 and the white area.

Contrast C and step kernel weighted normalization correlation R²And pixel value z_iA similar form of ROI with is defined as:

<maths num="15"><img id="000017" he="51" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

Contrast C uses the standard formula for weighted standard deviation, R²Uses standard expressions in weighted normalization correlation, but is simplified by step kernel 1800 To.

<maths num="16"><img id="000018" he="23" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

Value k_iThe direct step kernel 1810 with is also the same as step kernel 1800, but with a 90 degree rotation. ratio:

<maths num="17"><img id="000019" he="21" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

Is a reasonable estimate of the tangent between the actual and predicted direction of the edge, especially if D is also a good measure of the angle itself. It should be noted here that the orthogonal step template 1810 does not need to be created and the values from step template 1800 can be used, but they must correspond to the pixel values of the ROI in different orders.

FIG. 19 shows how the values R2, C, and D are used to determine the analog output in an embodiment of an edge detector. It is certain that an edge will be detected if all three conditions are met. 1. ROI seems to be the desired step edge and the desired step edge template weighted normalization correlation R²Means that the ROI is high. 2. Contrast C is significantly above the noise threshold. 3. Angle D is small.

Weighted normalization correlation operation 1900 using ROI 1910 and step kernel 1920 is R²To calculate. Contrast operation 1930 using ROI 1910 and step kernel 1920 calculates C, which is converted by fuzzy threshold operation 1940 to a fuzzy logic value of 1942, which indicates the confidence that contrast exceeds the noise level. The weighted correlation operations 1950 and 1952 using the absolute values of the arctangents of ROI1910, step kernel 1920 and orthogonal step kernel 1922, ratio operation 1960 are between the edge orientation expected by the fuzzy threshold operation 1970 and the actual edge orientation. Compute D, which is converted to the fuzzy logic value 1972, which indicates the reliability that the angle of is small.

Fuzzy AND element 1980 R to generate analog output 1990 for edge detector²And operate fuzzy logic values 1942 and 1972. It should be noted here that R2, which ranges from 0 to 1, can be used directly as a fuzzy logic value. The analog output 1990 has a range of 0 to 1, but if one different range is desired, it can be multiplied by a constant, for example 100. It should be noted here that the logical output of the edge detector is generated from the analog output using the sensitivity thresholds of all photographs.

FIG. 20 illustrates a method for performing an edge detector that detects ridge edges. The ridge kernel 2000 is created according to the size and shape of the ROI and the predetermined direction θ of the edge. In Ridge Kernel 2000, the ROI is a circle with a diameter of 12 pixels and its direction θ is 15 degrees from the horizontal. The ridge kernel 2000 is an approximation of the first derivative of the Gaussian function at the distance r from the end to the center of each weight.

<maths num="18"><img id="000020" he="19" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

In FIG. 20, a = 99, but any suitable value can be used. In an embodiment, formula 9 uses b = 0.33.

Using Ridge Kernel 2000 is similar to using Step Kernel 1800. Contrast C is calculated using the same formula, but R because the sum of kernel values is not 0²Uses a different formula:

<maths num="19"><img id="000021" he="33" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

It should be noted here that this formula is reduced from the formula used for the step edge if the sum of the kernel values is 0.

Different methods are used to determine the angle D between the actual and predicted directions. Value K_i⁺A positive rotating ridge kernel 2020 with is formed in the edge direction θ + a and has a value of K._i^-Negative rotating ridge kernel 2010 with is formed in the edge direction θ-a. The parabola adapts to the point.

<maths num="20"><img id="000022" he="43" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

The X coordinate of the parabolic minimum is a good estimate of the angle D between the actual and predicted directions of the edge.

FIG. 21 shows how the ridge kernel is used to determine the analog output in an embodiment of an edge detector capable of detecting whether it is a step edge or a ridge edge. For ridge edge detection, the weighted normalization correlation 2100 is R²Use ROI 2110 and Ridge Kernel 2120 to calculate. The contrast operation 2130 uses the ROI 2110 and the ridge kernel 2120 to calculate the C whose contrast is converted to fuzzy logic by the fuzzy threshold 2140. Correlation elements 2150, 2152, and 2154 use the ROI 2110 and ridge kernel 2120, the positive rotating ridge kernel 2124 and the negative rotating ridge kernel 2122 to calculate the angle D at which contrast is converted to fuzzy logic by the fuzzy threshold 2170. To do.

R²And fuzzy logic values are used by the fuzzy AND element 2180 to generate a ridge analog output 2192 for edge detectors that can detect ridge edges. For edge detectors capable of detecting step or ridge edges, the ridge analog output 2192 and analog output 1990 from the step edge detector 2188 can be used by the fuzzy OR element 2182 to produce a synthetic analog output 2190.

Figure 22 shows the graphical controls displayed on the HMI for the user to see and manipulate to set parameters for edge detection. The set of graphical controls displayed on the HMI830 for setting gadget parameters is referred to as the parameter view. The parameter view of the other photographs is sufficiently similar to FIG. 22, which is apparent to those skilled in the art as a method of construction.

The name text box 2200 lets the user see and enter the gadget name. The time display 2202 shows the time taken for the latest launch of the gadget. Logical output label 2204 indicates the current logical output value of the gadget and may be changed to a color, shape, or other indicator to distinguish between true ( 0.5) and false (&lt;0.5). The inversion check box 2206 inverts the gadget logic output.

The agree button 2210 and the disagree button 2212 are used as teachings, as further explained below (Fig. 45).

Position control 2220 is used to position the photo in the field of view. The diameter spinner 2222 is used to change the diameter of the detector. Directional control 2224 is used to place the edge detector in the expected edge direction. The position, diameter, and arrangement can also be set by image manipulation of the image display, for example, as in the image display of FIG.

The edge type checkbox 2230 is used to select the type of edge detected and the edge polarity. You can select the step to shade, the step to shade, dark ridge, and light ridge. Any synthesis of choices is possible, except that no choice is made.

The sway spinner 2240 uses an arrangement with the highest analog output, specifying the parameter j, such that the edge detector activates an arrangement ± j pixels set around a specific position.

Sensitivity threshold control 2250 allows the user to set a sensitivity fuzzy threshold for a photo. Zero point label 2251 can be set by the zero point slider 2252, t₀The values to 1120 (Fig. 11) are shown. The 1-point label 2253 can be set by the 1-point slider 2254, the value t₁Shows 1122. Analog output label 2255 indicates the current analog output of the photo. The analog output is also shown graphically by filling within the area of the analog output label 2255, which shrinks and grows like a mercury thermometer on its side. The filled area is t₀First zone under 2256, t₀And t₁Second zone between 2257, t₁It can be displayed in three different colors or patterns corresponding to the third zone 2258 above.

Contrast threshold control 2260 shows the user Contrast C and allows the user to set contrast fuzzy thresholds 1940 and 2140. These controls are operated in the same manner as the sensitivity threshold control 2250.

Directional error control 2270 allows the user to show the angle between the actual direction and the predicted direction D and set the directional fuzzy thresholds 1970 and 2170. Since these controls correspond to a low value of D increasing the fuzzy logic value, the threshold display is the sensitivity threshold control 2250, except that it fills from right to left instead of left to right. It is operated in the same way.

FIG. 23 illustrates how to perform a spot detector. The Spot Kernel 2300 is created for the size and shape of the ROI. For the Spot Kernel 2300, the ROI is a circle with a diameter of 15 pixels. The spot kernel 2300 uses equations 18 and 9 and is an approximation of the quadratic derivative of the Gaussian function at the distance r from the center of the kernel to the center of each weight. In an embodiment, b = 0.6.

Using Spot Kernel 2300 is similar to using Ridge Kernel 2000. Weighted normalization correlation R²And contrast C are calculated using the same formula used in the ridge kernel.

FIG. 24 shows how the spot kernel is used to determine the analog output in an embodiment of the spot detector. The operation of the spot detector is the same as the edge detector embodiment shown in FIG. 19, except that the angle D is not calculated or used. Weighted normalization correlation 2400 is R²Use ROI 2410 and Spot Kernel 2420 to calculate. Contrast 2430 uses ROI 2410 and Ridge Kernel 2420 to calculate the C converted to fuzzy logical values by the fuzzy threshold 2440. R²And fuzzy logic values are used by the fuzzy AND element 2480 to produce the spot analog output 2490.

[Methods for detectors and human-machine interfaces] FIG. 25 is a set of image displays used to illustrate the operation of the detector according to embodiments. The first image display 2500 and the second image display 2502 have one detector 2510 and one detector 2512. The reader understands that the following description of detector 2510 and detector 2512 generally applies to any detector and detector. Further, the reader understands that within the scope of the present invention, a number of alternative methods for constructing the detector are derived.

In an embodiment, the detector uses a variety of all well-known techniques to search for tangents in a one-dimensional range. The search direction is the normal of the tangent, and the detector has a width parameter used to specify the correction along the tangent and is used in a well-known manner. The analog output of the detector depends on the particular method used to find the tangent.

In an embodiment, the detector searches a one-dimensional range of the tangent and creates a one-dimensional shape along the search range using a well-known method of calculating the prediction of ROI parallel to the tangent. The one-dimensional shape is convolved and integrated with the one-dimensional tangent kernel, and the position of the peak response corresponds to the position of the tangent. Interpolation, such as the well-known parabolic interpolation, can be used when it is desirable to improve the accuracy of the tangent position. In another embodiment, the tangents are located by searching for the peak analog output using the tangent detector of FIG. 19 or 21, again by interpolating if desired to improve accuracy. Can be searched for.

In another embodiment, the detector searches a multidimensional range using well-known methods such as transformation, rotation, and sizing degrees of freedom. Since it will be obvious to those skilled in the art how to employ a multidimensional detector to place a photo in practice of the present invention, the following studies will be preferred for one-dimensional detectors for convenience. limit.

The detector 2510 and detector 2512 can move the FOV by clicking and dragging anywhere on the boundary. The detector 2510 has a resizing handle 2520 for changing the diameter, and the detector 2512 has a resizing handle 2522 for changing the width and range and a rotating handle 2524 for changing the direction. All photos can be moved by dragging the border and have similar handles suitable for operation.

In the embodiment of FIG. 25, the detector is depicted in the image display as a rectangle with a straight line segment inside, called a plunger 2530. The width of the detector is parallel to the plunger and the range is the normal of the plunger. The detector is oriented by the user so that it is approximately parallel to the tangent where the plunger is found. The rectangle indicates the search range, and the plunger indicates the position of the tangent, if any. If no tangent is detected, the plunger is drawn in the center of the range.

The detector has rail 2532, shown as a dashed line in the figure, which coincides with the plunger but extends in both directions with respect to the tangent to the image display.

Each photo can be linked to zero or more detectors, up to a maximum number determined by a particular embodiment of the invention. The number of links determines the number of degrees of freedom that the detector can control. The degrees of freedom include rotation, size, and two-conversion degrees of freedom. In the embodiment, the maximum number of links is 2, and only the degree of freedom of conversion is controlled.

The link defines how the photo moves as the detector plunger moves, following the tangents of the image. Movement is defined to lock the photo at a fixed distance to the rail of the linked detector. In an embodiment, the link can actually be made from structural elements and bearings and is drawn using mechanical analogy so that the photo moves in the same way when a force is applied to the plunger.

In Figure 25, the link from the detector 2510 to the detector 2512 has a rod 2540, and the post 2542 allows the rod to move freely along the detector 2510 and the rail 2532, but at the correct angle with respect to the rail. It is firmly attached to the fixing slider 2544. If extended, the rod will be drawn on the border of the photo so that it passes through the center of the photo and at two possible points near the rail. The detector rails are only visible if there is a link.

Each photo has an emitter depicted as a square handle on a portion of the border. For example, the detector 2510 has an emitter 2550 and the detector 2512 has an emitter 2552. The link is created by dragging and dropping the photo emitter to any point on the detector. If the link already exists, drag and drop may delete the link. Alternatively, another mechanism may be used for deletion. Users may not create more links from photo or circular dependencies than the maximum number of links allowed. A tool tip is provided to tell the user if the link is created, deleted, or rejected (and why) to assist the user while dragging the emitter onto the detector.

Dragging the detector does not change the behavior of the plunger. If there is a plunger, it is fixed on the tangent line, and if there is no plunger, it returns to the center. Thus, dragging the detector while the normal is being detected only changes the search range. The plunger does not move relative to the field of view. More generally, dragging the detector does not change the position of the linked photos. Dragging the detector adjusts the rod length as needed to prevent other photos from moving relative to the field of view.

Any plunger, whether normal or not, can be manually dragged within the range of the detector, and any linked photo will move accordingly. This allows the user to see the effects of linkage. As soon as the mouse button is released, the plunger quickly returns to the correct position (returns the linked photo to the correct position).

In FIG. 25, the detector 2510 is linked to one detector 2512, so one conversion degree of freedom is controlled. The degrees of freedom are normal to the tangential direction, that is, the direction of the rod 2540. Comparing the second image display 2502 with the first image display 2500, the plunger 2530 has moved to the right to follow the tangent (hidden) of the image. Note that the position of the detector 2512 in the field of view has not changed, but the detector 2510 follows the plunger to the right and follows the tangent of the object, thus following the motion of the object itself. By mechanical analogy, the detector 2510 moves so that it is firmly attached to the rail 2532 by the rod 2540, and the rail moves with the plunger.

FIG. 26 shows a set of image displays used to illustrate the behavior of detectors linked to two detectors. In the first image display 2600 and the second image display 2602, the detector 2610 is linked to the first detector 2620 and the second detector 2630, so that the two degrees of freedom of conversion are controlled. The degrees of freedom are in the direction of the first rod 2622 and the second rod 2632. Note that the two degrees of freedom are not orthogonal because they are not orthogonal. The handle and emitter are not shown in Figure 26.

Comparing the second image display 2602 to the first image display 2600, the first plunger 2624 has moved downward to follow the first tangent (hidden) of the image, and the second plunger 2634 , It is moving to the left and slightly downward so as to follow the second tangent (hidden). Note that the positions of the detectors 2620 and 2630 in the field of view have not changed, but the detector 2610 moves to the lower left to follow the plunger and follows the tangent of the object, thus following the motion of the object itself.

By mechanical analogy, the detector 2610 moves because it is firmly attached to the first rail by the first rod 2622 and to the second rail by the second rod 2632. Note that the first slider 2628 slides to the left along the first rail 2626 and the second slider 2638 slides down along the second rail 2636. The slider slides along the rail when two non-orthogonal detectors are linked to the photo.

If the photo is linked to two nearly parallel detectors, the movement will be unstable. It is useful to limit the angle between the detectors and set the linked photos so that they do not move below that. This condition can be indicated in some way in the image display, such as by displaying two rods using a special color such as red.

It provides significant flexibility because it allows the detector to be mounted either in a fixed position or linked to another detector. In Figure 26, neither detector is linked, so it remains in a fixed position in the field of view and is in a fixed position relative to the illumination, which is often desirable.

FIG. 27 shows a set of image displays used to illustrate the behavior of detectors linked to two detectors, one of which is linked to each other. In the first image display 2700 and the second image display 2702, the detector 2710 is linked to the first detector 2720 and the second detector 2730. The second detector 2730 is also linked from the rod 2740, post 2742 and slider 2744 to the first detector 2720. Slider 2744 slides along rail 2722 of first detector 2720.

Note that there is no need to limit the number of photos that can be linked to the detector. The limit of freedom is the number of links a photo can have for the detector. In the example of Figure 27, the detector 2710 is linked to two detectors, controlling the two degrees of freedom of conversion. Since the second detector 2730 is linked to one detector, one conversion degree of freedom is controlled. The first detector 2720 is not linked to the detector and remains fixed in the field of view.

The detector is configured to follow the tangents above and to the right of the arc. Comparing the second image display 2702 to the first image display 2700, the arc 2750 is moving downwards, which causes the rails 2722 to move downwards. This causes both the detector 2710 and the second detector 2730 to move downwards. The detector 2710 is in the same position relative to the object, as is the second detector 2730. This is desirable in this case because if the second detector 2730 is fixed in the field of view, you will lose sight of the tangent to the right of the arc 2750 that moves up and down. Note that this is not a problem if the tangents of the objects in the image are straight lines.

The first detector 2720 does not have a detector that moves left or right to find the tangent above the arc 2750. You can't link to the second detector 2730 because you're creating a link on a circle, but one detector has to do the first and can't link to anything, so it's not allowed. Instead, the movement of the object from the field of view ensures that the first detector 2720 finds the upper tangent. In the example of FIG. 27, the motion is from left to right, and due to the high frame speed of the vision detector, the object moves only slightly in each frame. Eventually, the first detector 2720 finds the upper tangent and finds the top of the arc 2750 near the center of the detector on a number of frames that vary depending on the velocity of the object. On these frames, the second detector 2730 is properly positioned to find the right tangent and moves the detector 2710 left and right as needed to put it in the correct position.

FIG. 28 shows a method for handling the case where the tangent found by the detector does not extend the straight line. Therefore, the placement of the detector must be fairly accurate along the boundaries of the object. This method can be used in applications where the first detector 2720 in Figure 27 is moving at high speed, so there may be an opportunity to miss the entire upper tangent as the object moves through the field of view. Absent.

To address such cases, the detector has parameters that can be used to specify the number of parallel bending motions created in the tangent search. The curving motions are spaced along the tangents in an amount that provides sufficient overlap so that the tangents do not hit the gaps in the curving motion.

Figure 28 shows the detector 2800 with four curved motions, with a tangent found on the second curved motion from the left. Triangular curved motion markers, such as 2810 and 2812 in the example curved motion markers, are shown outside the dashed curved motion square 2820 so as not to collide with the detector diagram. If no tangent is found anywhere in the curvature, the detector returns to the center of the curvature square (because of the even number of curvatures, it is not a curvature marker).

FIG. 29 shows how a detector can be used to handle the rotation and resizing of an object, even in an embodiment where only two degrees of freedom of transformation are controlled. The conversion-only limitation provides users with considerable convenience and transparency, but rotating and resizing small objects allows photos in different parts of the field of view to be converted differently depending on different detectors. Therefore, it can be processed as before. Small rotations and resizing are well estimated by transformations within a small area of the field of view, so as long as the photo is linked to at least one nearby detector, object rotations and resizing are like transformations. appear.

Also referring to FIG. 29, the first image display 2900 and the second image display 2902 are the first detector 2910, the second detector 2912, the first detector 2920, the second detector 2922, And includes a third detector 2924.

The first detector 2910 is linked to the nearby first detector 2920 and second detector 2922, and will properly rotate or resize the object (unless the change is too large). Be placed. However, the second detector 2912 is far away, and due to rotation, the second detector 2912 is likely to be in the wrong position in the vertical direction with respect to the second detector 2922, and due to resizing, the second detector 2912 is likely to be misaligned. It is likely to be misaligned horizontally with respect to one detector 2920. The third detector 2924 is used to obtain the vertical position of the second detector 2912 instead of the second detector 2922, which allows it to handle the rotation of the entire object. The remote first detector 2920 is used to obtain the horizontal direction for the second detector 2912, so the size of the object does not change much. If it is necessary to handle resizing in addition to rotation, add a fourth detector horizontally near the second detector 2912.

Comparing the second image display 2902 to the first image display 2900, the object (hidden) moves to the right and rotates counterclockwise, which is the detector when the detector follows the object tangent. It can be seen by moving. The second detector 2922 and the third detector 2924 are linked to the first detector and stay close to the detector.

[Another logical representation using a ladder diagram] FIG. 30 shows another representation of a logical representation based on a ladder diagram, a widely used industrial programming language. The ladder diagram of FIG. 30 is substantially the same as the wiring diagram shown in FIG. 16 and represents the same set of equipment and logical interconnections in the diagram. In the embodiment, the user can optionally switch the logical display between the wiring diagram and the ladder diagram, and can operate and edit any of the diagrams.

In an embodiment, to draw a ladder diagram for the configuration of the equipment, create one level for each equipment with logical inputs for gates, judgments, and outputs. The order of the levels is the execution order automatically determined for the device. Each level consists of one contact for each logical input, followed by an icon for the device. Contacts usually open with a non-inverting connection and usually close with an inverted connection. At the AND gate, the contacts are continuous, and at the OR gate, the contacts are parallel. The label associated with each contact indicates the name of the device connected to the logical input.

To simplify the ladder diagram, the user can choose to hide any gate that has a logical output connected to only one logical input of another device via a non-inverting connection. .. If a gate is hidden, the level of that gate is not displayed. Instead, the contacts at that gate usually open the contacts that appear at the level at which the gate is used.

Referring to FIG. 30, level 3000 indicates the input to the object detection determination 3002, level 3010 indicates the input to the object pass determination 3012, and level 3022 indicates the input to the output device named "reject". .. The top 3030 and side 3032 of normally open contacts and the normally closed contact box 3034 are displayed at level 3000 because the AND gate 1610 is connected to only one logical output via a non-inverting connection. Represents the input to AND Gate 1610 (Figure 16), replacing the contact.

Similarly, level 3010 is connected to the normally open contact hole 3040, label 3042, and AND gate (hidden AND gate 1612), and the output is connected to object pass verdict 3012, label edge 3044. Shown.

Level 3020 has a normally open contact 3050 and a normally closed contact 3052 that is connected to an AND gate (hidden AND gate 1670) and whose output is connected to an output device 3022 named "Reject". Shown.

It is noted that the ladder diagrams used in the above embodiments of the present invention are a limited subset of the ladder diagrams widely used in the industry. It is provided primarily to assist users with knowledge of ladder diagrams. The subset is chosen to match the functionality of the routing diagram, simplifying implementation and also allowing the user to choose between the routing and the ladder diagram as needed.

In another embodiment, a more complex implementation of the ladder diagram is provided and the wiring diagram is not used. The result is a powerful industrial inspection machine, but with increased complexity, by combining the functionality of the vision detector with the functionality of the PLC.

[Marking, synchronized output, and related measurements] FIG. 31 shows a timing diagram used to illustrate how the output signal of the vision detector can be synchronized with the recording time. Signal synchronization is desirable for a variety of industrial inspection purposes, such as downstream rejection actuators. The requirements for these signals depend on how the object is presented and how it is detected.

As mentioned above, objects are detected by visual event detection or external triggers (continuous analysis mode is used in the absence of individual objects). In addition, the object can be presented by indexing or by continuous motion when it comes to a stop in the field of view. When using external triggers, the analysis is typically the same regardless of how the object is presented. However, in the case of visual event detection, the analysis may depend on whether the object comes to a stop (indexing) or has almost the same motion (continuous). For example, vision detectors may not be able to measure or use the velocity of an object in indexed applications.

Visual event detection is a new feature, suggesting new output signals, especially when continuous object presentation is used. It is desirable that the vision detector can control some external actuators either directly or by acting as an input to a PLC. This suggests that, at least in the case of continuous presentation, the timing of the output signal is associated with constant accuracy to the point in time when the object passes a certain fixed point in the production flow. In the example of FIG. 4, the fixed point is the recording point 430, and in the timelines of FIGS. 5 and 6, the times are the recording times 550, 552, and 680. In FIG. 31, the time is a recording time of 3100. Note that the encoder count can be used instead of time.

In prior art vision systems, this goal is a photodetector that is explored by an external trigger and typically reacts within microseconds of recording time. This signal that triggers the vision system (eg, signal 166 in Figure 1) is also used in the PLC (signal 162) to synchronize the output of the vision system with the downstream actuator. The output of the vision system is in milliseconds, and this time is too variable to be used out of sync.

The present invention can provide a recording time-synchronized output with considerable accuracy when used in visual event detection mode, regardless of whether the operating device is directly controlled or used by a PLC. However, one problem is that, like vision systems, and unlike photodetectors, vision detectors make decisions about an object after a few milliseconds of recording time. Moreover, this delay varies considerably depending on how many frames have been analyzed, and at least when the recording time occurs in the acquisition / processing cycle.

FIG. 31 shows the object detection logic output 3140 and the object pass logic output 3150. Note that the object detection determination is in the "output when executed" mode, which is the mode used for output signal synchronization. When the determination is made at decision point 3110, the object pass logic output 3150 changes state and the detection pulse 3170 is displayed at the object detection logic output 3140. Note that the decision delay 3130 from the recording time 3100 to the determination point 3110 is variable.

If the object detection logic output 3140 and the object pass logic output 3150 are wired to an AND gate, a pulse (hidden) is created only when an object that passes the inspection is detected. If the logic output of the object pass is inverted, the pulse (hidden) is created only when the AND gate detects an object that fails the inspection.

The detection pulse 3170 and the pulse indicating that a passing object was detected and a failing object was detected are all useful. In indexed applications, it may be used directly by the actuator. The PLC can use an external trigger to synchronize these pulses with the actuator. However, if the object is in continuous motion and no external trigger is used, the variable determination delay 3130 may prevent the direct pulse from being used to control the actuator.

The present invention solves this problem by measuring the recording time 3100 and then synchronizing the output pulse 3180 of the output signal 3160. The output pulse 3180 occurs with a constant output delay of 3120 from the recording time of 3100. Also referring to the timing diagram of FIG. 5, the output pulse 3180 is an example of report step 560, and the output delay 3120 corresponds to the delay 570.

The operation of measuring the recording time is called "marking". Recording time is measured in milliseconds by linear interpolation, least squares matching, or other well-known method, using the well-known time (count) at which the image was acquired and the well-known position of the object determined by an appropriate detector. It can be determined with an accuracy of less than a second. Accuracy depends on shutter time, overall acquisition / processing cycle time, and object speed.

In an embodiment, the user selects one detector whose search range is substantially along the direction of motion used for marking. The recording point is arbitrarily selected to be the center point of the detector range. As mentioned above, the recording point is a fictitious reference point where the exact position does not matter as long as it is fixed. The user can achieve the desired synchronization of the output signal by adjusting the delay from this arbitrary time. If it is detected that the object has not crossed the recording point during the active frame, the recording time will be based on estimation and will be less accurate.

The user may instead specify the recording time that occurs when the object is first detected. This option may be selected in applications where no detector is used, for example, when visual event detection relies on a detector located in a fixed position in the field of view (see Figure 44). Marking at the first detection point may be possible in some applications to improve accuracy by interpolating fuzzy logic values into the object detection decision to find the time when the values intersect 0.5. Despite this, it may be less accurate than when the detector is used.

Note that the output signal can only be synchronized to the recording time if the output delay 3120 is longer than the longest predictive delay 3130. As such, the actuator needs to be sufficiently downstream from the recording point, which is expected for almost all applications.

When an external trigger is used, the recording time is relative to the time the trigger occurs (eg, recording time 680 in Figure 6). The output pulse 3180 can be synchronized at this time for direct control of the downstream actuator. Marking can be used, but generally not required, when using external triggers and PLCs, as was the case with prior art vision systems.

FIG. 32 is an example of measuring the recording time in the visual event detection mode, and also an example of measuring the object velocity, the calibration of the pixel size, and the distance and the posture of the object. Each column in the figure corresponds to one frame acquired and analyzed by the present invention, except for the first column, which has the function of labeling the columns. Column 3210 of the frame is a frame number that functions to identify the frame. The time column 3220 indicates the time in milliseconds that the frame was acquired, measured from any reference time. Encoder column 3225 shows the encoder count at the time the frame was acquired. The object detection column 3220 shows the logical input to the object detection determination. From this we can see that the active frames are 59 to 46.

The location column 3240 indicates the position of the tangent, measured by the detector, in a substantially direction of motion, oriented so that it has a search range selected by the user. The location is measured relative to the center of the detector's search range and is shown only in the active frame. It can be seen that the position is between zero, record points 61 and 62, and crosses record points between hours 44.2 and 46.2, counts 569 and 617.

In this example, the dynamic image analysis ends at frame 66 after two consecutive inactive frames are found. The location-based recording time, shown in the sixth column 3250, is calculated at the end of frame 66. The value of 45.0 shown is the linear interpolation time between frames 61 and 62 where positions intersect zero. Alternatively, the straight line can touch the points for the active frame from the time sequence 3220 and the position sequence 3240, and the straight line can be used to calculate the time value corresponding to position 0.

The recording time based on the time when the object was first detected is shown in the 7th column 3260.

A lot of additional useful information can be obtained from the measurement data summarized in the following table.

<tables num="2"><img id="000023" he="41" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></tables>

Thus, the information is either using two points (linear interpolation using zero intersection frames 61 and 62 of the recording, and slope using frames 59 and 64 for velocity and size), or least squares. It can be calculated by conformance or other well-known method of technology. The results are similar for the two methods shown, but the least squares fitting method is generally more accurate, but more complex.

The recording time and count may be used for output signal synchronization, as described above. Object velocity may be transmitted to automated equipment for a variety of purposes. The pixel size calculation provides a pixel size calibration of the encoder count and is proportional to the physical distance of the production line. Such calibration may have slightly different optical magnifications by presenting the distance of the field of view with a physical unit for the user and adjusting the size and position of the photo based on the calibration. It can be used for a variety of well-known purposes, such as transferring setups between instruments.

The pixel size can be calculated for each object, so the value can be used to determine the distance between the objects. A small pixel size corresponds to a distant object. On a constant velocity production line, object velocity can be used to make the same decision, just as an object in the distance moves slower than an object in the vicinity when viewed from the vehicle window.

The data in Figure 32 may be obtained from a single detector nominated by the user for this purpose. In another embodiment, the user can nominate two other detectors, either parallel or normal to the direction of motion in the field of view. All such detectors are oriented so that they have a search range parallel to the direction of motion. The difference in pixel size calibration obtained from these two detectors is due to the detectors being separated parallel to the direction of motion and around the axis that is normal to the direction of motion. Represents the rotation of an object in space in the plane of the object and around an axis parallel to the direction of motion of the separated detector, which is normal to the direction of motion.

FIG. 33 shows an output FIFO used to generate synchronous pulses for downstream control of the actuator. In general, a large number of objects may line up between the inspection point and the downstream actuator, and if there is an output pulse, each such object until each object reaches the actuator and needs a pulse. A FIFO or similar mechanism is needed to retain information about the corresponding output pulse.

Referring to FIGS. 5 and 31, report step 560 of the first object 500 corresponds to the output pulse 3180, and after the second object 510 is analyzed, the delay 570 corresponds to the output delay 3120. Occurs later. If the output delay 3120 is greater than the time between objects, FITO or a similar mechanism is needed.

Figure 33 shows a simple FIFO that can be used for this purpose. The FIFO holds a number that corresponds to the time the output signal changes state. At decision point 3110 (FIG. 31), the output pulse can be scheduled by placing a time on the FIFO input 3320, such as the first tangent time 3300 and the last tangent time 3310. The software timer or other mechanism runs at the scheduled time determined by removing the time from the FIFO output 3330. At the scheduled time, the state of the output signal changes and the new scheduled time is removed from the FIFO.

[Judgment, output, and usage example] FIG. 34 shows a user-set parameter display for object detection determination. The first parameter display 3490 is used for the visible event detection mode and the second parameter display 3492 is used for the external trigger mode.

The presentation control 3400 allows the choice of either index or continuous object presentation.

The frame filtering control 3410 allows the user to set limits on the active frame count or total object detection weight. The minimum frame spinner 3412 allows the user to select the minimum required for the count or weight threshold, as described above in FIG. 12, with reference to FIG. The maximum frame spinner 3414 allows the user to choose a maximum frame count or weight. The dynamic image analysis is then finished and a decision is made, regardless of how many active frames are found. This allows the user to limit the amount of time the invention spends on an object, which is especially useful for indexes or slow objects.

The idle time control 3420 allows the user to specify a minimum and maximum time for idle step 580 (Figure 5). If the minimum and maximum are equal, the idle time is fixed by the user. If the maximum is greater than the minimum, the vision detector can automatically select a time within a specified range based on the measured speed at which the object is presented. The phase-fixed loops shown in Figures 47 and 48 can be used to measure the object presentation velocity. In an embodiment, the automatically selected idle time is half the object presentation cycle and is shortened if it needs to be within a user-specified range.

The Lost Frame Spinner 3430 allows the user to specify the maximum number of consecutive inactive frames allowed without terminating dynamic image analysis. Such a frame is described by analysis step 542 in FIG.

The marking control 3440 allows the user to select a marking mode. If location marking is selected, the user must specify the detector using the detector list control 3442.

The output mode control 3450 allows the user to select a mode that defines when the pulse appears on the logical output.

The frame count spinner 3460 allows the user to select the number of frames to analyze in external trigger mode.

FIG. 35 shows a parameter display for user setting of object pass determination. The mode control 3500 allows the user to choose between the weighted averaging method described for FIG. 12 and the percentile method described for FIG. Once the percentile method is selected, the threshold t described can be selected using the percentile spinner 3510.

FIG. 36 shows a parameter display for user setting of the output device. The mode control 3600 allows the user to choose how the output signal is controlled. In "straight line" mode, the logical input is passed directly to the output signal without any delay or synchronization. In "delay" mode, on the rising tangent of the logical input, the output pulse is from the recently recorded recording time (or encoder count) for the amount specified by the delay control 3610 and for the period specified by the pulse control 3620. Scheduled to occur at a late time. Scheduled pulses may be placed in a FIFO associated with the output device.

FIG. 37 illustrates one way of setting the invention to operate in visible event detection mode when connected to a PLC. The logical display shows the object detection determination 3720 wired to the "detection out" output device 3730 and the object acceptance determination 3740 wired to the "pass out" output device 3750. The appropriate set of photos and gates are wired to the verdict but not displayed.

The object detection determination 3720 is set for continuous presentation, locational recording, and output at the end of execution, as shown in FIG.

The first parameter display 3700 indicates that the "detection out" output device 3730 is configured to generate a synchronized 10ms pulse 25ms after the recording time. This pulse is triggered by the rising tangent of the object detection determination 3720. The second parameter display 3710 indicates that the "pass out" output device 3750 is configured to transmit the pass / fail results from the object pass determination 3740 directly to the output signal.

The PLC can sense two output signals, but latches the "pass out" output on the tangent, noting the time of the rising tangent of the signal from the "detection out". The PLC then understands that the object has been detected and knows the result of the inspection when the object crosses the recording point (25 ms before the rising tangent of the "detection out").

FIG. 38 illustrates one way of configuring the invention to operate in visual event detection mode for direct control of the rejection activator. The logic display shows an AND gate 3810 wired for object detection and object pass determination to generate a pulse when a failing object is detected. The AND gate 3810 is wired to the "reject out" output device 3820 to receive this pulse. Parameter display 3800 indicates that the reject out output device 3820 is configured to generate a synchronized 100ms pulse 650 encoder count after the recording time. This pulse is triggered by the detection of a rejected object. The object detection determination is configured for continuous presentation, locational recording, and run-time output.

FIG. 39 illustrates one way of configuring the invention to operate in external trigger mode when connected to a PLC. As shown in FIG. 37, the logical display shows an object detection determination wired to the detection out output device and an object acceptance determination wired to the pass out output device. However, instead of wiring a series of photos and gates to the logical input of the object detection determination, the "TrigIn" input device 3920 is used. The fact that the logical input of object detection does not depend on the photo, but on at least one input device, commands the verdict to operate in external trigger mode. If nothing is wired to the object detection, operate in continuous analysis mode.

The first parameter display 3900 and the second parameter display 3910 indicate that both output devices pass the logical input directly to the output signal. The PLC senses the two output signals and latches the "pass out" output on the rising tangent of the "detection out". The PLC should then be warned that the object has been detected and the result of the inspection and, if necessary, obtain the recording time from an external trigger.

Another way to configure the invention to operate in external trigger mode when connecting to a PLC is to use the output device settings in Figure 37 to create synchronous output pulses. In this way, the PLC can get the recording time from the vision detector, so it does not waste a valid input on the connection to the external trigger.

FIG. 40 illustrates one way of configuring the invention to operate in external trigger mode for direct control of the rejection activator. The operation is the same as the settings shown and described in FIG. 38, except that the "TrigIn" input device 4000 is wired to object detection rather than a series of photos and gates. Note that the recording time is only the time of the external trigger.

[Continuous analysis and example] FIG. 41 illustrates one way of configuring the invention to operate in continuous analysis mode for flow detection on a continuous web. Image display 4110 shows a portion of a continuous web 4100 moving through a vision detector.

The detector 4120 and tangent detector 4122 are configured to inspect the web. If the web breaks, folds, or becomes significantly frayed at either tangent, the detector 4120 and / or tangent detector 4122 will produce a failing output (logical value &lt;0.5). As the web moves up and down, the detector 4120 tracks the upper tangent and keeps the tangent detector 4122 in the right relative position to detect the lower tangent. However, if the width of the web changes significantly, the tangent detector 4142 will produce a failing output.

In logical representation, the "upper" detector 4140 represents the detector 4120 and the "lower" detector 4150 represents the tangent detector 4122. These are wired to the AND gate 4160, which is wired to the "detection out" output device 4170, with the logic outputs inverted. As can be seen in the parameter display 4130, the inverted output of the AND gate 4160 is passed directly to the output signal.

Therefore, the output signal is always asserted when the logical output of the photo fails. The signal is updated at the high frame speed of the vision detector to indicate that the web state is continuous.

FIG. 42 illustrates one way of constructing the present invention for the detection of continuous web flows using object detection determination for signal filtering. The settings are similar to the settings in Figure 41, and image display 4110 and parameter display 4130 also apply to this figure. The logic display shows the same settings as in FIG. 41, except that the object detection determination 4260 is placed between the AND gate 4160 and the detection out output device 4170.

The parameter display 4200 shows how the object detection determination is set. In this application, there are no individual objects, but objects are defined as bad web stretches. Since the output mode is set to "output to process", an output pulse is created for each stretch of the defective web, and the period is the period of the defective product.

Setting the minimum frame count to 3 removes the slight stretch of bad web and is not detected. Allowing up to three lost frames also eliminates the slightest stretch of the good web that is buried in the longer defective parts. Thus, the output signal is similar to the signal in FIG. 41, but with a small amount removed. Note that these values are just examples and the user chooses the appropriate values for a particular application.

Since no maximum frame count is specified, bad web growth can be continuous forever and can be considered as one bad. The idle time may be set to zero so that the web is constantly inspected.

FIG. 43 illustrates one way of constructing the present invention for the detection of flows on a continuous web using output devices for object detection determination and signal synchronization. The settings are the same as in Figure 42, except that the Detect Out output device 4170 is configured as shown in Parameter Display 4300.

Note that in the parameter display 4200, the object detection determination is configured to set the recording time to the "object", here the time when bad web growth is first detected. As can be seen in the parameter display 4300, the 50ms output pulse is the 650 encoder count generated after the recording time.

[Detection of objects without a detector] FIG. 44 illustrates one way of constructing the Japanese invention to operate in a visual event detection mode without a detector as an input to an object detection determination to determine an active frame. Object 4400, including feature 4410, moves the field of view of the vision detector from left to right. The contrast detector ROI4420 is arranged to detect the presence of feature 4410. The brightness detector ROI4430 on the left and the brightness detector ROI4432 on the right are arranged to detect an object within the range of the field of view defined by the distance.

The logical display shows the "right" luminance detector 4440 corresponding to the right luminance detector ROI4432 and the "left" luminance detector 4442 corresponding to the left luminance detector ROI4430. The "right" luminance detector 4440 produces a true logical output when the object does not cover the right luminance detector ROI4432 because the background of this example is brighter than the object 4400. The "left" luminance detector 4442 produces a true logical output when the object 4400 does not cover the left luminance detector ROI4430 because the output is inverted. Therefore, the AND gate 4460 creates a true logic output when the right tangent of the object 4400 is between the left luminance detector ROI4430 and the right luminance detector ROI4432.

Note that the logic output of AND gate 4460 is actually a fuzzy logic level between 0 and 1 when the right tangent of object 4400 partially covers one of the ROIs. The contrast detector ROI4420 does not move because no detector is used, so it detects feature 4410 in a range of positions defined by the distance between the left luminance detector ROI4430 and the right luminance detector ROI4432. Must be large enough.

The AND gate 4460 is wired to the object detection determination, and the "hole" contrast detector 4470 corresponding to the contrast detector ROI4420 is wired to the object acceptance determination. The determination in this example is configured for visible event detection and direct control of the rejection activator, as shown in FIG.

In the example of FIG. 44, the right tangent of object 4400 is placed fairly straight so that a detector can easily be used instead of a set of luminance detectors. However, there are applications where it may be difficult to use the detector due to the lack of a clear battle, and there are applications where the use of a combination of detectors is beneficial.

[Learning] FIG. 45 illustrates the rules that may be used in some embodiments of the vision detector to learn proper parameter settings based on the example presented by the user. This figure is used in conjunction with FIG. 22 to illustrate an example learning method.

In embodiments, the photo can learn the appropriate setting of fuzzy thresholds for detection sensitivity. The learning process may also provide suggestions on which detectors to use and, where appropriate, which settings are likely to work best. As usual, learning presents an object that the user has determined whether it is good or bad, and shows it by interaction with HMI830.

Learning is optional as to whether default or manual settings are available, but highly recommended for brightness and contrast detectors as analog output has no absolute meaning is a physical unit (eg gray level). Will be done. Learning is less important for edge, spot, and template detectors, as these outputs are primarily based on dimensionless and absolutely meaningful normalized correlation values.

For example, if the edge or spot detector has an analog output of 80, then the image ROI of the ideal edge or spot template has a correlation coefficient of at least 0.8, so the edge or spot is definitely detected. Is pretty confident. If the output is 25, you can be pretty sure that no edges or spots have been detected. But for a brightness detector, for example, is 80 bright enough? Is 25 dark enough? This is most often best learned by example.

There are two parts to the learning process. The first question is how users interact with the HMI 830 to teach specific examples. The second question is what the photo does when such an example is presented.

Referring to FIG. 22, the photo parameter display includes an up arrow button 2210 and a down arrow button 2212. The button is next to the logical output label 2204 and may change color or other features to distinguish between true (greater than or equal to 0.5) and false (less than or equal to 0.5) as described above. In the following discussion, green is used for true and red is used for mistake. Although not shown in FIG. 22, the up arrow button 2210 is green and the down arrow button 2212 is red. The proximity of the logical output label 2204 to the matching color allows the user to understand the function of the button.

Clicking the green up arrow button 2210 means "learning that the logical output of this photo should be green (true)". This operation is called "upward learning". Clicking on the red down arrow button 2212 means "learning that the logical output of this photo should now be red (wrong)". This operation is called "downward learning". These meanings are intended to be particularly clear and unique when the ability to invert the output is required. The words "good" and "bad" are often used to describe examples of objects used for learning, whether the output is inverted and how the connected gates are used. The meaning changes according to.

Suppose an object uses three detectors, all of which must have true output in order to pass the inspection. Each detector can be taught individually, but this would be unnecessarily time-consuming. If a good object is presented, all three detectors must be true, so it is useful to say "this is a good object" with a single click anywhere. On the other hand, if a bad object is presented, it may not be possible to know which detector is wrong, so it is generally taught individually.

Similarly, if the three detectors are OR, then the object will pass if any of them have a true output, and a single click can teach a bad object, but a good object. In, detectors are generally trained individually. However, again, these rules change when the inputs and outputs are inverted.

Teaching multiple detectors with a single click can be managed without confusion by adding up and down arrow buttons to the gate, following the rules shown in Figure 45. The buttons are displayed and clicked in the parameter display of each bait, but in Figure 45 the buttons are shown in the logical display for illustration. Again, by clicking the green (red) up arrow (down arrow) button, we tell the gate to learn up (down), that is, "The logical output of this gate must now be green (red). It means "learning what you have to do".

The AND gate of the non-inverting output learns upwards by instructing the equipment wired to the non-inverting input to learn upwards and to the equipment wired to the inverting input to learn downwards. .. For example, the non-inverted AND gate 4500 learns upwards by instructing photo 4502 to learn upwards, AND gate 4504 to learn upwards, and photo 4506 to learn downwards. .. The non-inverting output AND gate cannot learn downwards, so you can disable the button and ignore the learning request.

The AND gate of the inverting output learns the downward direction by instructing the device wired to the non-inverting input to learn upwards and the device wired to the non-inverting input to learn downwards. For example, an inverted AND gate 4510 learns downwards by instructing photo 4512 to learn upwards, photo 4514 to learn upwards, and OR gate 4516 to learn downwards. To do. The inverted output AND gate cannot learn upwards, so you can disable the button and ignore the learning request.

The non-inverting output OR gate learns downwards by instructing the equipment wired to the non-inverting input to learn downwards and to the equipment wired to the inverting inputs to learn upwards. For example, a non-inverted OR gate 4520 learns downwards by instructing OR gate 4522 to learn downwards, photo 4524 to learn downwards, and photo 4526 to learn upwards. .. Since the non-inverting output OR gate cannot learn upwards, the button can be disabled and the learning request can be ignored.

The inverting output OR gate learns upwards by instructing the equipment wired to the non-inverting output to learn upwards so that the equipment wired to the non-inverting output learns upwards. For example, the inverted OR gate 4530 learns upwards by instructing photo 4532 to learn downwards, photo 4534 to learn downwards, and AND gate 4536 to learn upwards. Since the inverted output OR gate cannot learn downwards, the button can be disabled and the learning request can be ignored.

A photo instructed by a gate to learn upwards or downwards behaves as if a similar button was clicked. The gate returns the learning command to the input, as described. All other devices ignore the learning command in this example.

One exception to the above rule is that all gates with only one input wired directly or indirectly to the photo from the other gates can learn up and down, so both buttons are enabled. is there. The learning command for such a gate is returned to the input and is inverted if either the output of the gate or the input is inverted, rather than both.

The user does not need to remember or understand these understandings. So the only rule to remember is to click on the color you want to output. Whenever the mouse is over the up and down buttons, tool tips are provided to explain which photos are being trained or why the buttons are disabled. When the up and down buttons are clicked, clear but non-commanding feedback is provided to ensure that the training has certainly occurred.

FIG. 46 illustrates how a photo learns upwards or downwards. In an embodiment, each photo holds two datasets called "high output data" and "low output data". Each such set contains a count, a sum of analog outputs, and a sum of squared analog outputs, so the average output is low m._lo, Average output height m_hi, Standard deviation of low output σ_lo, And the standard deviation of the output height σ_hiCan be calculated. For each learning command, Photo adds an analog output to the dataset as follows:

<tables num="3"><img id="000024" he="26" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></tables>

In the example of FIG. 46, the x-axis 4640 corresponds to the analog output of the photo. The distribution of high power value 4600 and the distribution of low power value 4610 are learned from appropriate object examples. The average low power 4630, average high power 4620, low power standard deviation (hidden), and high power standard deviation (hidden) have been calculated.

The data is low threshold t₀1120 and high threshold t₁Used to calculate the photo sensing fuzzy threshold of 4650, as defined by 1122 (see also Figure 11).

<maths num="21"><img id="000025" he="12" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

The parameter k may be chosen appropriately. In an embodiment, k = 0.

The method for learning upwards or downwards is summarized in the next step. 1. Calculate the analog output and add the low or high output data as upward or downward, and as corresponding to the inversion or non-inversion of the logical output. 2. Calculate the mean and standard deviation of the high and low output data. 3. Use formula 21 to update the perceived fuzzy threshold based on mean and standard deviation.

Photos can also save recent manual threshold settings, if any, so users can restore them if they so desire. All data and settings can be saved for all photos on the HMI 830, so learning can continue in multiple sessions. The user can clear and verify the data using the appropriate HMI command.

If your dataset has no examples, you can use the average default value. In embodiments, the defaults are:

<maths num="22"><img id="000026" he="13" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

Where a_loIs the lowest possible analog flagship, a_hiIs the best. If the set of data contains less than one example, the standard deviation can be estimated to be zero. That is, although not generally recommended, it is possible to learn from a single example.

In another embodiment, the learning command for the detector calculates the analog output for which results are obtained from each type of detector operating on the ROI of the taught detector. Data for each detector type is calculated and stored and used to suggest better detector selection by searching for those with large distances between low power and high power examples.

[Use phase-locked loop for lost and extra object detection] FIG. 47 illustrates the use of a phase-locked loop (PLL) to measure the presentation speed of an object and detect lost and extra objects in a production line that presents the object at approximately constant speed. The implementation of a PLL that synchronizes with the pulse train and detects lost and extra pulses is clear to those of skill in the art.

In one embodiment, a signal containing an output pulse synchronized with the recording time, for example, an output signal 3160 containing an output pulse 3180 (FIG. 31) is connected to the input of a conventional PLL. Traditional PLL methods for lost and extra pulse detection are used to detect lost and extra objects. The frequency of the voltage-controlled oscillator used by the conventional PLL gives the object presentation speed.

In the embodiment, the software PLL inside the vision detector DSP900 (Fig. 9) is used. With reference to FIG. 47, the recording time column 4700 is used by the software PLL to calculate the corresponding time window column 4710 while the recording is expected to occur. Lost objects are detected at time 4730, corresponding to the end of time window 4720, because time window 4720 does not include recording time. Since the recording time 4740 is detected outside of which time window, the extra object is detected at the time 4750 corresponding to the recording time 4740. Information about object presentation speed and lost and extra objects can be reported to the automated device.

FIG. 48 illustrates the operation of the new software PLL used in the embodiment. The recording time is drawn on the vertical axis 4800, and the object count is shown on the horizontal axis 4810. Each drawn point, including the example point 4820, corresponds to one detected object. To ensure numerical accuracy in the calculations, object count 0 may be used for the most recent object, and negative counts may be used for the previous object. The next expected object corresponds to a count of 1.

The optimal straight line 4830 is further calculated using the weighted least squares method described below. Weighting is chosen to weight more recent points more strongly than points farther away. The slope of the optimal straight line 4830 gives the presentation period of the object, and the time corresponding to count = 1 gives the expected time of the next object.

In one embodiment, the fixed number of recent points is given an equal weight, and the old points are given a weight of zero, so only the most recent points are used. The set of recent points used is stored in the FIFO buffer, and well-known methods are used to update the least squares data as new points, which are added and the old points removed.

In embodiments, a weighted 4840 is used that corresponds to the impulse response of the individual Butterworth filters. The individual Butterworth filter is a two-pole, highly attenuated, eternal impulse response digital low frequency filter that is easily implemented in software. It has excellent low frequency and step response characteristics, examines the entire recording time history, has adjustable parameters to control the frequency response, and does not require a FIFO buffer.

The output Yi at the count i of the Butterworth filter by the input Xi is

<maths num="23"><img id="000027" he="12" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

Where f is the filter parameter

<maths num="24"><img id="000028" he="9" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

And v_iIs an intermediate term called the velocity term.

When the input is i is non-zero, the unit impulse x₀= 1, x_iIf = 0, the output is an impulse response and the symbol w_iReferenced by. The effect of the filter is to convolve an input with an impulse response, creating a weighted average of all previous inputs, in which case the weight is x._-iIs. Weighted 4840 is the value w when f = 0.12_-iIs.

In the case of a Butterworth PLL, three Butterworth filters are used, corresponding to the data needed to calculate the least squares optimal line. Recording time is t_iWhen i = 0, it corresponds to the latest object, when i &lt;0, it corresponds to the previous object, and when i = l, it corresponds to the next object. In addition, all the time is t₀That is, t₀Relative to = 0. The following polymerizer is required.

<maths num="25"><img id="000029" he="33" wi="160" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

Note that the sum exceeds the range - i 0. These values are taken as the output of the three Butterworth filters and the specified input t_i, It_iAnd t_i²Is.

The following additional values are required and are derived from the filter parameter f.

<maths num="26"><img id="000030" he="40" wi="160" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

Optimal straight line 4830 is t_i= a_iGiven by + b, therefore

<maths num="27"><img id="000031" he="14" wi="160" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

Will be. The value a is a very accurate measurement of the current object presentation period. Straight line averaging can be used to calculate the expected recording time at any point in the near future. As a fraction of the object presentation period, the optimal straight line weighted RMS error is

<maths num="28"><img id="000032" he="15" wi="160" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

It is a variable index of the object presentation speed.

The new recording times entered in the three filters are new objects and t₀Time t, which is the elapsed time between recent times at = 0_iOccurs in. For each of the three Butterworth filters, the output needs to be adjusted and the velocities are i = 0 and t for the new object.₀Need to reset to = 0. This is necessary for Equation 26 to be modified, and also for the accuracy of the numbers. Without this modification, C would change at each point. The following are the formulas for the three filters and show the modifications. A dash symbol is used to indicate new values for output and velocity, where u is a temporary value.

<maths num="29"><img id="000033" he="65" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

The Butterworth PLL output and velocity terms can be initialized to correspond to a set of values (a, b, E) as follows:

<maths num="30"><img id="000034" he="53" wi="159" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

This allows the PLL to be reset to start at a particular point p using Equation 30 with a = p, b = 0, and E = 0. You can also change the filter coefficients while running the PLL by initializing the output and velocity terms with the current values of a, b, E and the new f.

In an embodiment, the Butterworth PLL determines if it is locked on the input recording time by considering the number of objects since the PLL was reset and the current value of E. The PLL is considered to be unlocked between the reset and the next n objects. If at least n objects are observed, the PLL will have an E with a lock threshold of E._lLocks below and E unlocks threshold E_uIf it exceeds, it will be unlocked. Further improvement is the unlocked filter parameter f_uIs used when the PLL is unlocked and the locked filter parameter f_lIs used when the filter is locked. Switching filter parameters is performed using Equation 30 to initialize the output and velocity items using the current values of a, b, and E.

In embodiments, n = 8, E_l= 0.1, E_u= 0.2, f_l= 0.05, and f_u=0.25。

Without m continuous objects, the next object arrives at approximately the next timing.

<maths num="31"><img id="000035" he="8" wi="160" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

Recording time t of this equation and a new object_nTherefore, m can be determined by the following equation.

<maths num="32"><img id="000036" he="13" wi="160" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

To retain the formula for the Butterwar non-PLL correction, the recording time for all missing objects must be inserted. This may be achieved by using the following equation:

<maths num="33"><img id="000037" he="11" wi="160" file="JP5784397B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

The time of m + 1 is entered in Equation 29.

It is understood that any individual low frequency filter can be used in place of the Butterworth filter to implement the PLL according to the optimal linear method of the present invention. Based on the above method, the equations 23-33 are replaced appropriately for the selected filter.

The details of various examples of the present invention have been described above. It is explicitly considered that a wide range of changes and additions can be made without departing from the spirit and scope of the present invention. For example, the processors and computing units herein are examples, and various processors and computers, both stand-alone and distributed, can be employed to perform the calculations herein. Similarly, the imaging apparatus and other vision components described herein are examples, and improved or different components can be employed within the disclosure of the present invention. Therefore, this detailed description is merely an example and does not limit the scope of the present invention.

110, 112, 114, 116, 118 objects 120 labels 124 holes 180 encoder 320 photodetector 400 visual detector