Autonomous welding robots

CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on U.S. Provisional Patent Application No. 63/153,109, filed February 24, 2021, entitled "SYSTEMS AND METHODS FOR OPERATING AND CONTROLLING A WELDING ROBOT," and "SYSTEMS AND METHODS claims the benefit of U.S. Provisional Patent Application No. 63/282,827, filed on November 24, 2021, entitled "For OPERATING AND CONTROLLING A WELDING ROBOT," the entire contents of each application being incorporated herein by reference for all purposes. Incorporated into the specification.

Background Robotic manufacturing involves the use of robots to perform one or more aspects of a manufacturing process. Robotic welding is one application in the field of robotic manufacturing. In robotic welding, a robot welds two or more components together along one or more seams. Such robots automate processes that would normally be performed by humans or by machines directly controlled by humans, thus offering significant advantages in production time, reliability, efficiency, and cost.

SUMMARY A computer-implemented method of generating instructions for a welding robot, in various examples. A computer-implemented method includes the process of identifying the expected location and expected orientation of candidate seams on a part to be welded based on a computer-aided design (CAD) model of the part, in order to generate a representation of the part. scanning a workspace containing the part; identifying candidate seams on the part based on a representation of the part and expected positions and orientations of the candidate seams; actual positions and orientations of the candidate seams; and generating welding instructions for a welding robot based at least in part on the actual location and orientation of the candidate seam.

In an example, a computer-implemented method of generating welding instructions for a welding robot. The method includes the steps of: obtaining image data of a workspace including a part to be welded via a sensor; identifying a plurality of points on the part to be welded based on the image data; from the plurality of points; The method includes identifying candidate seams on the parts to be welded and generating welding instructions for a welding robot based at least in part on the identification of the candidate seams.

<figref num="1">1 is a block diagram of an autonomous robotic welding system, according to various examples. FIG.</figref><figref num="2">1 is a schematic diagram of an autonomous robotic welding system, according to various examples; FIG.</figref><figref num="3">1 is a schematic diagram of an autonomous robotic welding system, according to various examples; FIG.</figref><figref num="4">3 is an exemplary point cloud of a part with a weldable seam, according to various examples; FIG.</figref><figref num="5">3 is an exemplary point cloud of a part with a weldable seam, according to various examples; FIG.</figref><figref num="6">FIG. 2 is a block diagram illustrating an alignment process flow, according to various examples.</figref><figref num="7">1 is a schematic diagram of a graph search technique that may determine a path plan for a robot, according to various examples; FIG.</figref><figref num="8">1 is a mathematical model of a robotic arm according to various examples;</figref><figref num="9">1 is a flowchart of a method for performing autonomous welding, according to various examples.</figref><figref num="10">1 is a flowchart of a method for performing autonomous welding, according to various examples.</figref><figref num="11">1 is a flowchart of a method for performing autonomous welding, according to various examples.</figref>

DETAILED DESCRIPTION Traditional welding techniques are tedious, labor intensive, and inefficient. Traditional welding techniques also are not flexible enough to accommodate irregularities commonly encountered during the manufacturing process, leading to undesirable downtime and inefficiencies. For example, traditional welding techniques require a skilled programmer to generate instructions for a welding robot to perform a welding operation. These instructions instruct the welding robot regarding the movements, paths, trajectories, and welding parameters that must be used to perform a particular welding task. The instructions are written with the assumption that a large volume of work will be performed, with the same welding operations being repeated over and over again. Thus, any deviations encountered during the welding process (eg, different parts) can result in a misplaced weld. Misplaced welds increase inefficiency, cost, and other negative aspects of mass production.

In some cases, a computer-aided design (CAD) model of a part may be useful to a welding robot to facilitate the welding operation. For example, a CAD model of a part to be welded may be provided to a welding robot, and the welding robot may use the CAD model to guide its movements, such as the location of seams to be welded. The seam(s) to be welded are annotated in the CAD model and the welding robot (e.g., an annotation containing user-selected edges, each edge representing a seam), and sensors are used to identify the seam. After locating, perform welding according to the notes. Although such approaches may reduce or eliminate the need for skilled programmers or manufacturing engineers, they have limitations. For example, welding work is a very precise work. Generally, to create an acceptable weld, it is desirable that the weld tip be located within 1 mm of the target location associated with the seam. When guiding a welding robot based on a CAD model, even if the part fits perfectly in the CAD model, the actual seam may be more than 1mm from the modeled position, which may cause the welding tip to It may be difficult or impossible to create a well-positioned and acceptable weld. If the CAD model is a simplification of the actual part (which is common in low volume production), the seams may be moved more than a centimeter from the modeled position. Therefore, it can be difficult to precisely locate seams based on CAD models using known techniques. Therefore, welding robots may create unacceptable welds, thereby resulting in defective parts.

In addition, prior solutions for controlling welding robots allow the welding robot (more specifically, the robot arm) to move within the manufacturing workspace along a path from a first point to a second point, such as a seam. It requires a skilled operator to provide specific commands to the welding robot to avoid collisions with other components (e.g. parts, sensors, clamps, etc.) when moving the welding robot. Identifying a path that is free of obstacles and collisions (eg, a path that a robot can follow to weld a seam) is referred to herein as path planning. Requiring a skilled operator to path plan dozens or even hundreds of potential paths for a robotic arm is inefficient, cumbersome, and expensive. Additionally, traditional welding robots are often programmed to repeatedly follow the same path, the same movements, and the same trajectory. While this repetitive process may be acceptable in high-volume manufacturing situations where the manufacturing process is highly mature, in low- or medium-volume situations the components may be positioned in unexpected positions relative to the welding robot. This can lead to collisions, misaligned parts, poor tolerances, and other problems. Therefore, in certain situations a skilled operator may be required to facilitate welding.

The welding technology described herein is capable of automatically and dynamically generating useful instructions for a welding robot to accurately and precisely identify and weld seams, thus making it possible to automatically and dynamically generate useful instructions for the welding robot to accurately and precisely identify and weld seams. better than. Unlike prior systems and techniques, the welding techniques described herein do not necessarily require a CAD model of the parts to be welded (although in some instances, as discussed below, a CAD model may be useful). other prior information about the part or manufacturing workspace are also not necessarily required. Rather, the welding techniques described herein use moveable sensors to map manufacturing workspaces (particularly parts and seams) in three-dimensional (3D) space and use such maps to Locate and weld seams with accuracy and precision. The welding techniques described herein provide, among other things, the ability to identify multiple candidate seams for welding, the ability to interact with a user to select candidate seams for welding, and the ability to dynamically change welding parameters. , including various additional features that further distinguish this technology from previous inferior solutions, such as the ability to provide feedback on the welding operation. Additionally, the welding technique described herein uses data acquired by sensors to automatically and dynamically plan the path, i.e., the robot arm moves without collision with other components. is configured to automatically and dynamically identify one or more paths within a possible manufacturing workspace without prior information. The welding techniques described herein are also configured to dynamically perform path planning and welding using a combination of data acquired by sensors and a priori information (e.g., an annotated CAD model). ing. These and other examples will now be described below with reference to the drawings.

FIG. 1 is a block diagram of an autonomous robotic welding system 100, according to various examples. System 100 includes a manufacturing workspace 101, a user interface 106, a controller 108, and a storage device 109 storing a database 112. System 100 may include other components or subsystems not explicitly described herein. A manufacturing workspace 101 in which a robotic arm(s) is positioned on or coupled to a platform or positioning device, assisted by information received by one or more sensors. area or enclosure in which one or more parts are worked on or otherwise supported by. In examples, workspace 101 may be any suitable welding area designed with appropriate safety precautions for welding. For example, workspace 101 may be a welding area located in a workshop, job shop, manufacturing plant, assembly shop, etc. In the example, a manufacturing workspace 101 (or more generally, a workspace 101) includes a sensor 102 and a robot 110 configured to perform a welding-type process such as welding, brazing, joining, etc. It may include a component 114 to be welded (eg, a component with a seam) and a fixture 116. The fixture 116 may hold, position, and/or manipulate the component 114 and may be, for example, a clamp, a platform, a positioning device, or other type of fixture. Fixture 116 may be configured to securely hold component 114. In examples, fixture 116 is adjustable manually by a user or automatically by a motor. For example, fixture 116 may dynamically adjust its position, orientation, or other physical configuration prior to or during the welding process. In some examples, robot 110 may include one or more sensors 102. For example, one or more sensors 102 may be positioned on an arm of robot 110 (eg, on a welding head attached to the arm). In another example, one or more sensors 102 may be positioned on a movable, non-welded robot arm (which may be different from robot 110). In yet another example, one of the one or more sensors 102 may be positioned on the arm of the robot 110 and another one of the one or more sensors 102 may be It may be positioned on mobile equipment within the space. In yet another example, one of the one or more sensors 102 may be positioned on the arm of the robot 110 and another one of the one or more sensors 102 may be movable. The robot arm may be positioned on a type of non-welding robot arm.

Sensor 102 is configured to capture information regarding workspace 101. In the example, sensor 102 is an image sensor configured to capture visual information (eg, a two-dimensional (2D) image) about workspace 101. For example, sensor 102 may include a camera (eg, a camera with a built-in laser), a scanner (eg, a laser scanner), and the like. Sensor 102 may include a sensor such as a light detection and ranging (LiDAR) sensor. Alternatively or additionally, sensor 102 may be an acoustic sensor configured to emit and/or capture sound, such as an acoustic navigation and ranging (SONAR) device. Alternatively or additionally, sensor 102 may be an electromagnetic sensor configured to emit and/or capture electromagnetic (EM) waves, such as a wireless detection and ranging (RADAR) device. Through visual, acoustic, electromagnetic, and/or other sensing techniques, sensors 102 may collect information about physical structures within workspace 101. In examples, sensors 102 collect static information (e.g., fixed structures within workspace 101), and in other examples, sensors 102 collect dynamic information (e.g., moving structures within workspace 101). However, in yet other examples, the sensor 102 collects a combination of static and dynamic information. Sensors 102 may collect any suitable combination of any and all such information regarding physical structures within workspace 101 to generate a 3D representation of the physical structures within workspace 101. Such information may be provided to other components (eg, controller 108). As noted above, sensor 102 may capture and communicate any of a variety of information types, but this description assumes that sensor 102 primarily captures visual information (e.g., 2D images) of workspace 101 and that it It is envisaged that they will then be used together to generate a 3D representation of the workspace 101 as described below.

To generate a 3D representation of workspace 101, sensor 102 captures 2D images of physical structures within workspace 101 from various angles. For example, a single 2D image of fixture 116 or part 114 is inadequate to generate a 3D representation of that component; similarly, a single 2D image of fixture 116 or part 114 from a single angle, view, or plane Although a set of multiple 2D images of part 114 may also be inappropriate for generating a 3D representation of that component, multiple 2D images captured from multiple angles at various locations within workspace 101 may , may be sufficient to generate a 3D representation of a component such as fixture 116 or part 114. This means that by capturing 2D images in multiple orientations, the planning drawings of that component, including front, side, and top views of the component, are all needed to generate a 3D representation of that component. This is because spatial information about the three dimensional components is provided in a conceptually similar manner. Thus, in the example, sensor 102 is configured to move around workspace 101 to capture sufficient information to generate a 3D representation of structures within workspace 101. In the example, the sensors are stationary, but present in sufficient numbers and in sufficiently diverse positions around the workspace 101 such that sufficient information is captured by the sensors 102 to generate the aforementioned 3D representation. . In instances where sensor 102 is mobile, any suitable structure may be useful to facilitate such movement around workspace 101. For example, one or more sensors 102 may be positioned on a motorized truck system. Although the sensor 102 is configured to move around the workspace 101 on the track system, the track system itself may be stationary. However, in some examples, the sensor 102 is movable on the track system, and the track system itself is movable around the workspace 101. In still other examples, the one or more mirrors capture a 2D image from an initial viewpoint when the sensor 102 is in the first configuration, and use the mirrors to capture the 2D image from the other perspective when the sensor 102 is in the second configuration. Arranged within the workspace 101 is a sensor 102 that can pivot, swivel, rotate, or translate about a point and/or along an axis to capture 2D images from a viewpoint. In yet other examples, the sensor 102 may be mounted on an arm that may be configured to pivot, swivel, rotate, or translate about a point and/or along an axis; The arms may be configured to capture 2D images from various viewpoints as they extend through their full range of motion.

Additionally or alternatively, one or more sensors 102 may be positioned on the robot 110 (e.g., on the welding head of the robot 110) as the robot 110 moves around the workspace 101. The image data may be collected at any time. Because the robot 110 is movable in multiple degrees of freedom and thus in multiple dimensions, the sensors 102 positioned on the robot 110 may capture 2D images from various viewpoints. In yet other examples, one or more sensors 102 may be stationary while the physical structure being imaged moves around or within the workspace 101. For example, the part 114 to be imaged may be positioned on a fixture 116, such as a positioning device, and the positioning device and/or the part 114 may be rotated, translated (e.g., fixed sensor 102 (e.g., either a sensor coupled to robot 110 or a sensor separate from robot 110) may be moved in Capture multiple 2D images of various facets of.

In some examples, some or all of the sensor 102 configurations described above are implemented. Other sensor 102 configurations are also contemplated and within the scope of this disclosure.

With further reference to FIG. 1, robot 110 (eg, a welding head of robot 110) is configured to move within workspace 101 according to a path plan received from controller 108, as described below. Robot 110 is further configured to perform one or more suitable manufacturing processes (eg, welding operations) on part 114 according to instructions received from controller 108. In some examples, robot 110 may be a six-axis robot with a welding arm. Robot 110 may be any suitable robotic welding equipment, such as a YASKAWA® robot arm, an ABB® IRB robot, a KUKA® robot, or the like. Robot 110 can perform arc welding, resistance welding, spot welding, tungsten inert gas (TIG) welding, metal active gas (MAG) welding, metal inert gas (MIG) welding, laser welding, plasma welding, and combinations thereof. can be configured to do so.

With further reference to FIG. 1, workspace 101, and specifically sensor(s) 102 and robot 110 within workspace 101, are coupled to controller 108. Controller 108 is any suitable machine specifically and specifically configured (e.g., programmed) to perform the operations attributed here to controller 108, or system 100 more generally. In some examples, controller 108 is not a general purpose computer, but is instead specially programmed and/or hardware configured to perform the operations attributed here to controller 108, or more generally to system 100. ing. In some examples, controller 108 is or is an application specific integrated circuit (ASIC) configured to perform the operations attributed here to controller 108, or system 100 more generally. Contains ASICs. In some examples, controller 108 includes or is a processor, such as a central processing unit (CPU). In some examples, controller 108 is a field programmable gate array (FPGA). In the example, controller 108 stores executable code that, when executed by controller 108, causes controller 108 to perform one or more of the operations attributable here to controller 108, or more generally to system 100. Contains the memory that is being used. Controller 108 is not limited to the specific examples described herein.

Controller 108 controls sensor(s) 102 and robot 110 within workspace 101. In some examples, controller 108 controls fixture(s) 116 within workspace 101. For example, the controller 108 may cause the sensor(s) 102 to move within the workspace 101 as described above and/or capture 2D images, acoustic data, and/or EM data as described above. It may also be controlled to import the data. For example, controller 108 may control robot 110, as described herein, to perform welding operations and move within work space 101 according to path planning techniques as described below. For example, controller 108 may control fixture(s) 116, such as a positioning device (e.g., platform, clamp, etc.), to rotate, translate, or otherwise move one or more parts within workspace 101. You can also operate it to move. Controller 108 may also control other aspects of system 100. For example, the controller 108 may further interact with the user interface (UI) 106 by providing a graphical interface on the user interface (UI) 106 that allows a user to interact with and use the system 100. Inputs may be provided and the graphical interface allows the controller 108 to, for example, provide various types of information (e.g., identified seams that are candidates for welding, possible paths during path planning, welding parameter options or selections). etc.) and/or receive such information from the user. UI 106 may be any type of interface, including touch screen interfaces, voice activated interfaces, keypad interfaces, combinations thereof, and the like.

Further, controller 108 may interact with database 112, for example, by storing data in database 112 and/or retrieving data from database 112. Database 112, more generally, may be stored on any suitable type of storage device 109 configured to store any and all types of information. In some examples, the database 112 may be a random access memory (RAM), a memory buffer, a hard drive, an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM). ), flash memory, or the like. In some examples, database 112 may be stored on a cloud-based platform. Database 112 may store any information useful to system 100 in performing welding operations. For example, database 112 may store CAD models of parts 114. As another example, database 112 may store an annotated version of a CAD model of part 114. Database 112 may also store point clouds of parts 114 generated using CAD models (also referred to herein as CAD model point clouds). Similarly, welds for part 114 generated based on a 3D representation of part 114 and/or user input provided with respect to part 114 (e.g., regarding which seams of part 114 to weld, welding parameters, etc.) Instructions may be stored in database 112. In the example, storage device 109 stores executable code 111 that, when executed, causes controller 108 to perform one or more operations originating here from controller 108, or system 100 more generally. In examples, executable code 111 is a single self-contained program; in other examples, executable code includes other executable code that may be stored on storage device 109 or elsewhere. is a program that has one or more function calls to . In some examples, one or more functions resulting from execution of executable code 111 may be implemented by hardware. For example, multiple processors may be useful in performing one or more distinct tasks of executable code 111.

FIG. 2 is a schematic diagram of an example autonomous robotic welding system 200, according to various examples. System 200 is an example of system 100 of FIG. 1, where like numbers refer to like components. For example, system 200 includes workspace 201. Workspace 201 further optionally includes a movable sensor 202, a robot 210 (to which one or more sensors 202 (other than movable sensor 202) may be attached), and fixtures 216. . Robot 210 includes multiple joints and members (eg, shoulders, arms, elbows, etc.) that allow robot 210 to move in any suitable number of degrees of freedom. Robot 210 includes a welding head 210A that performs welding operations on a component, such as a component that may be supported by a fixture 216 (eg, a clamp). System 200 further includes a UI 206 coupled to workspace 201. During operation, sensor 202 collects 2D images of workspace 201 and provides the 2D images to a controller (not explicitly shown in FIG. 2). The controller generates a 3D representation (eg, a point cloud) of the workspace 201, such as the fixtures 216, the parts supported by the fixtures 216, and/or other structures within the workspace 201. The controller uses the 3D representation to identify seams (e.g., parts supported by fixtures 216) and cause robot 210 to collide with structures within workspace 201, as described herein. the robot 210 to plan a path for welding the seam without any problems and to weld the seam.

FIG. 3 is a schematic diagram of an autonomous robotic welding system 300, according to various examples. System 300 is an example of system 100 of FIG. 1 and system 200 of FIG. 2, where like numbers refer to like components. For example, system 300 includes workspace 301. Workspace 301 further optionally includes a movable sensor 302, a robot 310 (to which one or more sensors 302 may be attached), and fixtures 316 (e.g., a platform or positioning device). . Robot 310 includes multiple joints and members (eg, shoulders, arms, elbows, etc.) that allow robot 310 to move in any suitable number of degrees of freedom. Robot 310 includes a welding head 310A that performs welding operations on a component, such as a component that may be supported by fixture 316. System 300 further includes a UI 306 coupled to workspace 301. During operation, sensor 302 collects 2D images of workspace 301 and provides the 2D images to a controller (not explicitly shown in FIG. 3). The controller generates a 3D representation (eg, a point cloud) of the workspace 301, such as the fixtures 316, the parts supported by the fixtures 316, and/or other structures within the workspace 301. The controller uses the 3D representation to identify seams (e.g., parts supported by fixtures 316) and cause robot 310 to collide with structures within workspace 301, as described herein. the robot 310 to plan a path for welding the seam without any process and control the robot 310 to weld the seam.

Referring again to FIG. 1, as discussed above, the controller 108 receives the 2D image (and possibly other data, such as acoustic or EM data) from the sensor 102 and determines the structure depicted in the 2D image. Configured to generate a 3D representation. A 3D representation is sometimes called a point cloud. The point cloud may be a set of points, each representing a position in 3D space of a point on the surface of part 114 and/or fixture 116. In some examples, one or more 2D images (e.g., image data captured by sensor(s) 102 in a particular orientation with respect to part 114) are 3D image data of workspace 101. may be superimposed and/or stitched together by controller 108 to reconfigure and generate. The 3D image data can be collated to generate a point cloud having associated image data for at least some of the points in the point cloud.

In an example, 3D image data can be collated by controller 108 such that the point cloud generated from the data can have six degrees of freedom. For example, each point in a point cloud may represent an infinitely small location in 3D space. As mentioned above, sensor(s) 102 can capture multiple 2D images of a point from various angles. These multiple 2D images may be matched by controller 108 to determine the average image pixel for each point. An averaged image pixel can be attached to a point. For example, if the sensor(s) 102 is a color camera with red, green, and blue channels, then the six degrees of freedom are {x position, y position, z position, red intensity, green intensity, and blue intensity}. For example, if sensor(s) 102 is a black and white camera with black and white channels, four degrees of freedom may be created.

FIG. 4 is an example point cloud 400 of a part with a weldable seam, according to various examples. More specifically, point cloud 400 represents part 402 and part 404 to be welded together along seam 406. FIG. 5 is an example point cloud 500 of a part with a weldable seam, according to various examples. More specifically, point cloud 500 represents parts 502 and 504 to be welded together along seam 506. Controller 108 (FIG. 1) is configured to generate 3D point clouds 400, 500 based on 2D images captured by sensor 102, as described above. The controller 108 then uses the point cloud 400, 500 (or, in some examples, image data useful to generate the point cloud 400, 500) to identify seams, such as seams 406, 506. a weld path along the seam 406, 506, and according to the path plan, the robot 110 (FIG. 1) may be used to apply the weld along the seam 406, 506. We now describe in detail how controller 108 executes executable code 111 (FIG. 1) to perform such operations, including seam identification and path planning.

Controller 108 , when executing executable code 111 , uses a neural network to perform pixel-by-pixel classification and /or perform point-by-point classification (eg, using one or more point clouds) to identify and classify structures within workspace 101; For example, controller 108 performs pixel-by-pixel and/or point-by-point classification to identify each imaged structure in workspace 101 as, for example, part 114 of an interface on part 114 or between parts 114. It may be identified as a seam (referred to herein as a candidate seam), as a fixture 116, or as a robot 110. Controller 108 may, in the example, identify and classify pixels and/or points based on a neural network (eg, a U-net model) trained using appropriate training data. Neural networks can be trained on image data, point cloud data, spatial information data, or a combination thereof. Since the point cloud and/or image data includes information captured from various viewpoints within the workspace 101, the neural network can generate a fixed image on the part(s) 114 from multiple angles and/or viewpoints. can be operative to classify the tool 116 or candidate seams. In some examples, a neural network can be trained to operate directly on a point set, such as a dynamic graph convolutional neural network, and a neural network can be trained to operate directly on a point set to analyze unorganized points on a point set. May be implemented. In some examples, a first neural network can be trained on point cloud data to perform point-wise classification, and a second neural network can be trained on image data to perform pixel-wise classification. can do. The first neural network and the second neural network can independently identify the candidate seam and locate the candidate seam. The outputs from the first neural network and the second neural network may be combined as a final output to determine the location and orientation of one or more candidate seams on the part 114.

In some examples, when pixel-by-pixel classification is performed, the results can be projected onto 3D point cloud data and/or a meshed version of the point cloud data, thereby Provide information about your location. If the input data is image data (eg, a color image), spatial information such as depth information may be included with the color data to perform pixel-by-pixel segmentation. In some examples, pixel-by-pixel classification may be performed to identify candidate seams and to locate candidate seams relative to part 114, as described further below.

As discussed above, controller 108 may identify and classify pixels and/or points as particular structures within workspace 101, such as fixtures 116, parts 114, candidate seams on parts 114, and the like. Portions of the image and/or point cloud data that are classified as non-parts and non-candidate seam structures, such as fixtures 116, are segmented and separated (e.g., deleted or otherwise removed) from the data. The process may also separate data identified and classified as corresponding to part 114 and/or candidate seam(s) on part 114. In some examples, after identifying candidate seams and segmenting the non-part 114 and non-candidate seam data as described above (or optionally prior to such segmentation), the neural network The seam can be configured to analyze the seam to determine the seam type. For example, the neural network can be configured to determine whether a candidate seam is a butt seam, a corner seam, an edge seam, a lap seam, a T-seam, etc. A model (eg, a U-net model) may classify seam types based on data captured from multiple viewpoints within workspace 101.

When pixel-by-pixel classification is performed using image data, the controller 108 selects pixels of interest (e.g., part 114 and 114) may be projected onto 3D space. Alternatively, if point-wise classification is performed using point cloud data, the points of interest may already exist in 3D space of the point cloud. In either case, to the controller 108, the 3D points are an unordered set of points, and at least some of the 3D points may be clustered together. To eliminate such noise and generate a continuous, unbroken subset of points to represent candidate seams, we use Manifold Blurring and Mean Shift. Shift (MBMS) technology or similar technology may be applied. Such techniques can crowd the points and eliminate noise. Controller 108 may then apply a clustering method to decompose the candidate seams into individual candidate seams. Stated another way, instead of having several subsets of points representing multiple seams, clustering can decompose each subset of points into individual seams. Following clustering, controller 108 may fit a spline to each individual subset of points. Therefore, each individual subset of points may be an individual candidate seam.

In summary, and without limitation, using the techniques described above, controller 108 receives image data captured by sensor 102 from various locations and viewpoints within workspace 101. The controller 108 performs pixel-by-pixel and/or point-by-point classification techniques using neural networks to identify each pixel and/or point as a candidate seam for the part 114, an interface on or between parts 114, Classify and identify as fixtures 116, etc. Structures identified as non-part 114 structures and non-candidate seam structures are segmented and separated, and controller 108 performs additional processing on the remaining points (e.g., to mitigate noise). It can be done. By performing these operations, controller 108 may generate a set of candidate seams on part 114 that indicate the location and orientation of these seams. Controller 108 may then determine whether the candidate seam is actually a seam, and optionally perform additional processing using prior information, such as a CAD model of the part and seam, as described next. You may do so. The resulting data is suitable for use by controller 108 to plan a path for applying a weld along the identified seam, also as described below.

In some cases, the identified candidate seam may not be a seam (ie, the identified candidate seam may be a false positive). To determine whether the identified candidate seam is actually a seam, controller 108 uses images captured by sensor 102 from various viewpoints within workspace 101 to determine a confidence value. The confidence value represents the likelihood of whether the candidate seam determined from the corresponding viewpoint is the actual seam. Controller 108 may then compare the confidence values for the different viewpoints and eliminate candidate seams that are unlikely to be actual seams. For example, controller 108 may determine a mean, median, maximum, or any other suitable summary statistic of candidate values associated with a particular candidate seam. In general, a candidate seam that corresponds to an actual seam has a confidence value that is consistently high (eg, above a threshold) across the various viewpoints used to capture the candidate seam. If the confidence value summary statistic for the candidate seam exceeds a threshold, controller 108 may designate the candidate seam as an actual seam. Conversely, if the confidence value summary statistic for a candidate seam is below a threshold, the candidate seam may be designated as a false positive that is not eligible for welding.

As described above, after identifying a candidate seam that is an actual seam, the controller 108 uses prior information, such as a CAD model (or a point cloud version of the CAD model), to perform an additional process, referred to herein as alignment. You may also perform the following processing. More specifically, in some cases, differences may exist between the seam dimensions on the part and the seam dimensions in the CAD model, and the CAD model does not support path planning as described herein. The CAD model needs to be transformed (e.g., updated) to account for any such differences, since it can later be used to perform Accordingly, controller 108 connects a first seam (e.g., a candidate seam on part 114 that has been verified as an actual seam) to a second seam (e.g., on a CAD model corresponding to the first seam (e.g., an operator / seams annotated by the user) to determine the difference between the first seam and the second seam. Seams on the CAD model may be annotated as described above. The first seam and the second seam can be at approximately the same location if the CAD model and/or controller 108 accurately predicts the location of the candidate seam. Alternatively, the first seam and the second seam can partially overlap if the CAD model and/or controller 108 are partially accurate. Controller 108 may perform a comparison between the first seam and the second seam. This comparison of the first seam and the second seam may be based in part on the shape and relative position in space of both seams. The second seam can be identified as the same as the first seam if the first seam and the second seam are relatively similar in shape and proximal to each other. In this way, controller 108 can take into account surface topography on the part that is not accurately represented in the CAD model. In this manner, the controller 108 can identify candidate seams and can subselect, refine, or update candidate seams for the part using the CAD model of the part. . Each candidate seam may be an updated point set representing the position and orientation of the candidate seam relative to the part.

FIG. 6 is a block diagram illustrating an alignment process flow 600, according to various examples. Some or all steps in alignment process flow 600 are performed by controller 108 (FIG. 1). Controller 108 may first perform a coarse alignment 602 using a point cloud 604 of the CAD model and a scanned point cloud 606 formed using images captured by sensor 102 . CAD model point cloud 604 and scan point cloud 606 may be sampled such that the points have a uniform or nearly uniform distribution and such that the points both have equal or nearly equal point densities. In the example, the controller 108 downsamples the point clouds 604, 606, randomly and uniformly selecting and retaining points within the group and discarding remaining unselected points. For example, in some examples a Poisson Disk Sampling (PDS) downsampling algorithm may be implemented. The controller 108 may provide as inputs to the PDS algorithm the boundaries of the point clouds 604, 606, the minimum distance between samples, and the limits of samples to select before they are rejected. In some examples, a delta network may be useful in transforming one model into another during coarse registration 602. A delta network may be a Siamese network that takes a source model and a target model and encodes them into latent vectors. Controller 108 may use the latent vectors to predict point-by-point transformations that transform or update one model to another. Delta networks may not require a training dataset. Given a CAD model and scanned point clouds 604, 606, the controller 108 in the context of a delta network spends one or more epochs learning the dissimilarity or similarity between the two point clouds. During these epochs, the delta network learns meaningful features that are later useful for alignment. A delta network may use skip connections to learn deformations, but in some examples skip connections may not be used. In some cases, the CAD model includes surfaces that are not present in the 3D point cloud generated using captured images (eg, scans). In such a case, the delta network moves all points corresponding to the missing surface from the CAD model point cloud 604 to some point in the scan point cloud 606 (and updates the scan point cloud 606). Thus, during alignment, the controller 108 (e.g., a delta network) may transform (or update) the original CAD model using the learned features, or the deformed CAD model using the learned features. You may convert the model. In an example, a delta network may include an encoder network such as a dynamic graph convolutional neural network (DGCNN). After the point cloud is encoded into features, a concatenated vector consisting of both CAD and scan embeddings can be formed. After performing a pooling operation (eg, max pooling), a decoder may be applied to the resulting vector. In some examples, the decoder may include five convolutional layers with specific filters (eg, 256, 256, 512, 1024, Nx3 filters). The resulting output can be concatenated with the CAD model and scan embedding, max-pooled, and then provided once again to the decoder. The final result may include a point-by-point transformation.

Extraneous data and noise in the data (eg, the output of coarse alignment 602) can affect the alignment of part 114. For at least this reason, it is desirable to remove as much extraneous data and noise as possible. Bounding box 608 is useful to remove this extraneous data and noise (eg, fixture 116) to limit the area in which alignment is performed. In other words, data within the bounding box is preserved, but all data from outside the bounding box, 3D or otherwise, is discarded. The aforementioned bounding box may be any shape that can surround or encapsulate (eg, partially or completely) the CAD model itself. For example, the bounding box may be a dilated or expanded version of the CAD model. Data outside the bounding box may be removed from the final alignment, or may still be included but weighted to reduce its impact.

With further reference to FIG. 6, during fine registration 610, controller 108 passes the output of bounding box 608 as a patch through a set of convolutional layers in a neural network trained as an autoencoder. More specifically, data may be passed through the encoder section of the autoencoder and the decoder section of the autoencoder may not be used. The input data may be the XYZ position of the point of the patch within the shape, for example (128, 3). The output may be, for example, a vector of length 1024, which is useful for point-by-point features.

The corresponding point set that best supports the rigid transformation between the CAD point cloud model and the scanned point cloud model should be determined during registration. The corresponding candidate is a matrix in which each element stores the confidence or probability of a match between two points:<img file="JP2024508564A_D0001.tif" />(eg, in database 112).

Various methods are useful for finding corresponding points from this matrix, such as hard correspondences, soft correspondences, product manifold filters, graph cliques, covariances, etc. After the fine alignment 610 is completed, the alignment process is then completed (612).

As discussed above, in some cases, the actual location of the seam on the part 114 is determined by the controller 108 using sensor imaging (e.g., using a scanned point cloud) and/or (e.g., using a scanning point cloud). , using a CAD model point cloud) may differ from the seam location determined by the CAD model. In such cases, a scanning procedure (sometimes referred to herein as pre-scanning) is useful in correcting the determined seam position to more closely match the actual seam position on the part 114. . In the scanning procedure, a sensor 102 positioned on the robot 110 (referred to herein as an onboard sensor) scans the seam. In some cases, this scanning may be performed using an initial motion and/or path plan generated by controller 108 using a CAD model, a scan, or a combination thereof. For example, sensor 102 may scan any or all areas of workspace 101. During this initial movement and/or path planning, the sensor 102 may capture observed images and/or data. The observed images and/or data may be processed by controller 108 to generate seam point cloud data. Controller 108 may use seam point cloud data to correct seam position when processing point cloud(s) 604 and/or 606. Controller 108 may also use seam point cloud data in correcting path and motion plans.

In some examples, the alignment techniques described above may be useful for comparing and matching seams determined using a sensor 102 other than on-board sensor 102 to seams identified by on-board sensor 102. . By verifying the seams in this manner, the robot 110 (more specifically, the head of the robot 110) is positioned relative to the actual seams as necessary.

In some examples, the pre-scan trajectory of robot 110 is the same as the trajectory planned for welding along the seam. In some such instances, movements taken against the robot 110 during pre-scanning may cause seams or sensors 102 mounted with critical geometry to limit the probability of collision or reduce instances of collision. may be generated separately for better visualization or to scan the geometry around the seam in question.

In some examples, prescan techniques may include scanning more than a particular seam or seams, but rather other geometries of part(s) 114. The scan data may be scanned using any of the techniques described herein to locate, locate, and detect the seam and ensure that the head of the robot 110 is positioned and moved along the seam as necessary. or may be useful for more precise application of all (e.g. alignment techniques).

In some instances, scanning techniques (e.g., using a sensor/camera mounted on a welding arm/welding head to scan the actual seam) provide gap variation information about the seam rather than location and orientation information about the seam. May be useful for identifying gender information. For example, scan images captured by sensor(s) 102 on robot 110 during a scanning procedure can identify gap variability and modify welding trajectories or path planning to account for such gaps. It may be useful to adjust. For example, in 3D points, 2D image pixels, or combinations thereof, may be useful for locating variable gaps between parts 114 to be welded. In some examples, variable gap finding is useful, where 3D points, 2D image pixels, or a combination thereof locate variable sizes of multiple gaps between parts 114 to be welded together. useful for determining, identifying, and measuring For tack weld discovery or general weld discovery, previous welds or material deposits within the gap between parts 114 to be welded may be identified using 3D points and/or 2D image pixels. Any or all such techniques may be useful for optimizing welds, including path planning. In some cases, gap variability may be identified within the 3D point cloud generated using images captured by sensor 102. In still other cases, the gap variability may be affected by the actual (scanning seams). In any one case, controller 108 may be configured to dynamically adapt welding instructions (eg, welding voltage) based on the determined location and size of the gap. For example, by dynamically adjusting the welding instructions for a welding robot, precision welding of seams at variable gaps can be obtained. Adjusting the welding instructions may include adjusting one or more of a welder voltage, a welder current, an electrical pulse duration, an electrical pulse shape, and a material feed rate.

In an example, user interface 106 may provide the user with an option to display candidate seams. For example, user interface 106 may provide a graphical representation of part 114 and/or candidate seams on part 114. Additionally or alternatively, user interface 106 may group candidate seams based on seam type. As mentioned above, controller 108 can identify the seam type. For example, candidate seams identified as lap splices can be grouped under the label "lap splice" and presented to the user via the user interface 106 under the label "lap splice". Similarly, candidate seams identified as edge seams may be grouped under the label "Edge Seam" and may be presented to the user via the user interface 106 under the label "Edge Seam".

User interface 106 may further provide the user with the option of selecting candidate seams to be welded by robot 110. For example, each candidate seam on part 114 may be presented as a push button on user interface 106. When the user presses a particular candidate seam, the selection can be sent to controller 108. Controller 108 can generate instructions for robot 110 to perform a welding operation on that particular candidate seam.

In some examples, a user may be provided with an option to update welding parameters. For example, user interface 106 may provide the user with a list of different welding parameters. The user can select specific parameters to be updated. Changes to the selected parameters can be made by text input using, for example, drop-down menus. This update can be sent to controller 108 so that controller 108 can update instructions for robot 110.

In examples where system 100 is not provided with prior information (eg, a CAD model) of part 114, sensor(s) 102 may scan part 114. A representation of component 114 may be presented to a user via user interface 106. This representation of the part 114 may be a point cloud and/or a mesh of point clouds that includes projected 3D data of scanned images of the part 114 obtained from the sensor(s) 102. A user can annotate seams to be welded within the representation via user interface 106. Alternatively, controller 108 can identify candidate seams in the representation of part 114. Candidate seams may be presented to the user via user interface 106. The user can select the seam to be welded from among the candidate seams. User interface 106 can annotate the representation based on the user's selections. In some examples, annotated representations may be stored in database 112.

After one or more seams on part(s) 114 have been identified and corrected to the extent possible using the techniques described above (or using other suitable techniques), Controller 108 plans the path of robot 110 during the subsequent welding process. In some examples, graph matching techniques and/or graph search techniques may be useful in planning the path of robot 110. A particular seam identified above may include multiple points, and the path planning technique involves determining a different state of the robot 110 for each such point along a given seam. The state of the robot 110 may include, for example, the position of the robot 110 within the workspace 101 and the particular configuration of the arms of the robot 110 in any applicable number of degrees of freedom. For example, for a robot 110 with an arm with six degrees of freedom, the state of the robot 110 simply includes the position of the robot 110 within the workspace 101 (e.g., the position of the welding head of the robot 110 in three-dimensional xyz space). It will also include specific sub-states for each of the six degrees of freedom of the robot arm. Furthermore, when robot 110 transitions from a first state to a second state, robot 110 may change its position within workspace 101, and in such a case, robot 110 necessarily (e.g., It will traverse a certain path in the workspace 101 (along the seam being welded). Therefore, specifying a series of states for robot 110 necessarily involves specifying a path for robot 110 to move within workspace 101. Controller 108 may perform the pre-scan technique or a variation thereof after path planning is completed, and controller 108 uses information captured during the pre-scan technique to make various appropriate adjustments (e.g., to seams). Adjustment of either the XYZ axes or the coordinate system used to perform the actual welding may be performed along.

FIG. 7 is a schematic diagram 700 of a graph search technique for determining a path plan for robot 110 (eg, by controller 108). Each circle in diagram 700 represents a different state of robot 110 (e.g., a particular position of robot 110 within workspace 101 (e.g., a position of a welding head of robot 110 in 3D space), and a different configuration of an arm of robot 110, and the position and/or configuration of fixtures supporting components such as positioning devices, clamps, etc.). Each row 702, 706, and 710 represents a different point along the seam to be welded. Therefore, for the seam point corresponding to column 702, robot 110 can be in any one of states 704A-704D. Similarly, for the seam point corresponding to column 706, robot 110 may be in any one of states 708A-708D. Similarly, for the seam point corresponding to column 710, robot 110 may be in any of states 712A-712D. For example, if robot 110 is in state 704A when at the seam point corresponding to column 702, robot 110 then transitions to any of states 708A-708D for the next seam point corresponding to column 706. Good too. Similarly, upon entering states 708A-708D, robot 110 may subsequently transition to any of states 712A-712D for the next seam point corresponding to column 710, and so on. In some examples, entering a particular state may preclude entering other states. For example, entering state 704A may allow the possibility of subsequently entering states 708A through 708C, but not 708D, whereas entering state 704B may subsequently allow states 708C and 708D. However, the possibility of entering states 708A to 708B is not allowed. The scope of this disclosure is not limited to any particular number of seam points or to any particular number of robot 110 states.

In some examples, to determine a path plan for robot 110 using graph search techniques (e.g., in accordance with the technique shown in FIG. 700), controller 108 selects a state corresponding to seam point N from states 704A-704D. (eg, states 712A-712D). An objective function can be designed by controller 108 by assigning a cost to each state and each transition between states. Controller 108 finds the path that results in the minimum cost value of the objective function. For the freedom of having multiple starting and ending points to choose from, graph search methods such as Dijkstra's algorithm or A* can be implemented. In some examples, a brute force method may be useful in determining an appropriate path plan. The brute force technique involves controller 108 calculating all possible paths (eg, through diagram 700) and selecting the shortest path.

Controller 108 may determine whether a state at each seam point is feasible, at least in part because controller 108 enforces the chaining of states along a series of seam points in the seam. This means that it can be determined whether this will cause a collision between the robot 110 and a structure within the workspace 101, or even with parts of the robot 110 itself. To this end, the concept of achieving different states at different points of a seam may alternatively be expressed in the context of a seam with multiple waypoints. First, controller 108 may discretize the identified seams into a series of waypoints. Waypoints may constrain the orientation of a welding head connected to robot 120 in three (spatial/translational) degrees of freedom. Typically, constraints on the orientation of the robot's 120 welding head are provided in one or two rotational degrees of freedom about each waypoint for the purpose of producing some desired weld of some quality; Specifically, for the surface normal vectors emanating from the waypoint and the path of the weld seam. For example, the position of the welding head is located in the x, y, and z axes, as well as about one or two rotational axes perpendicular to the axis of the welding wire or tip of the welding machine, all with waypoints and waypoints. can be constrained to some nominal coordinate system attached to . These constraints in some examples may be angular limits or tolerances. Those skilled in the art will recognize that the ideal or desired weld angle may vary based on the geometry of the part or seam, the direction of gravity on the seam, and other factors. In some examples, controller 108 ensures that the seam is perpendicular to gravity for one or more reasons (e.g., to find a balance between welding and path planning for optimization purposes). The welding position can be restricted to 1F or 2F. Therefore, the position of the weld head can be held (constrained) by each waypoint in any suitable orientation relative to the seam. Typically, the weld head is not constrained about an axis of rotation (θ) that is coaxial with the axis of the weld head. For example, each waypoint may define the position of the welding head of welding robot 120 such that at each waypoint the welding head is in a fixed position and orientation relative to the weld seam. In some implementations, the waypoints are sufficiently finely discretized to make movement of the welding head substantially continuous.

Controller 108 may divide each waypoint into multiple nodes. Each node can represent a possible orientation of the weld head at that waypoint. As a non-limiting example, the welding head can be rotated about an axis of rotation coaxial with the axis of the welding head such that the welding head can rotate (e.g., 360 degrees) along the axis of rotation θ at each waypoint. It can be done without restrictions. Each waypoint can be divided into 20 nodes such that each node in each waypoint represents a weld head in 18 degree rotation increments. For example, a first pair of waypoint nodes may represent a weld head rotation of 0 degrees, a second pair of waypoint nodes may represent a weld head rotation of 18 degrees, and a third A waypoint node pair can represent a 36 degree weld head rotation, and so on. Each waypoint can be divided into 2, 10, 20, 60, 120, 360, or any suitable number of nodes. A subdivision of a node can represent a division in more than one degree of freedom. For example, the orientation of a welder tip around a waypoint can be defined by three angles. A weld path can be defined by connecting each pair of waypoint nodes. Thus, the distance between waypoints and the offset between adjacent waypoint nodes can represent the amount of translation and rotation of the welding head as it moves between the node-waypoint pair.

Controller 108 may evaluate each waypoint node pair for weld feasibility. For example, consider a non-limiting example of dividing a waypoint into 20 nodes. Controller 108 may evaluate whether a first waypoint node pair representing a weld head held at 0 degrees is feasible. In other words, controller 108 may evaluate whether robot 110 will collide with or interfere with a part, fixture, or the welding robot itself when placed in the position and orientation defined by its waypoint node pair. I can do it. Similarly, controller 108 can evaluate whether a second waypoint node pair, a third waypoint node pair, etc. are feasible. Controller 108 may similarly evaluate each waypoint. In this way, all possible nodes of all waypoints can be determined.

In some examples, a collision analysis as described herein compares a 3D model of workspace 101 and a 3D model of robot 110 to determine whether the two models overlap and, optionally, to determine whether the two models overlap. This can be done by determining whether some or all overlap. If the two models overlap, controller 108 may determine that a collision is possible. If the two models do not overlap, controller 108 may determine that a collision is unlikely. More specifically, in some examples, the controller 108 compares models for each of the sets of waypoint node pairs (such as the waypoint node pairs described above) and compares models for each of the sets of waypoint node pairs (such as the waypoint node pairs described above) and compares models for each of the sets of waypoint node pairs or cases. It may be determined that the two models overlap for everything. For the subset of waypoint node pairs for which model intersections are identified, the controller 108 may omit the waypoint node pairs within that subset from the planned path and select alternative pairs of those waypoint node pairs. can be identified. Controller 108 may repeat this process as necessary until a collision-free path is planned. Controller 108 may use a flexible collision library (FCL), which includes various techniques for efficient collision detection and proximity calculations as a tool for collision avoidance analysis. FCL is useful for performing multiple proximity queries on different model representations and can be used to perform probabilistic collision identification between point clouds. Additional or alternative resources may be used with or in place of the FCL.

Controller 108 may generate one or more feasible simulated (or evaluated, both terms are used interchangeably herein) weld paths, if physically feasible. A welding path may be a path taken by a welding robot to weld a candidate seam. In some examples, the weld path may include all waypoints of the seam. In some examples, the weld path may include some, but not all, waypoints of the candidate seam. The welding path may include movement of the robot and the welding head as the welding head moves between each pair of waypoint nodes. Once a feasible path between node-waypoint pairs is identified, the next feasible node-waypoint pair of consecutive waypoints, if any, can be identified. Those skilled in the art will recognize that many search trees or other strategies may be employed to evaluate the space of possible node-waypoint pairs. As discussed in further detail herein, cost parameters may be assigned or calculated for movement from each node-waypoint pair to a subsequent node-waypoint pair. The cost parameters can be associated with travel time, amount of travel (eg, including rotation) between node-waypoint pairs, and/or simulated/expected weld quality produced by the weld head during travel.

The node is not realizable for welding one or more waypoints and/or is realizable for moving between a previous waypoint node pair and either of the waypoint node pairs for a particular waypoint. If no path exists, controller 108 may determine alternative welding parameters such that at least some additional waypoint node pairs are feasible for welding. For example, if the controller 108 determines that neither of the waypoint node pairs for the first waypoint is feasible, thereby rendering the first waypoint unweldable, the controller 108 Alternative welding parameters, such as alternative welding angles, may be determined such that at least some waypoint node pairs for the waypoints are weldable. For example, controller 108 may remove or relax constraints on rotation about the x-axis and/or y-axis. Similarly, controller 108 may allow the weld angle to be varied by one or two additional rotational (angular) dimensions. For example, controller 108 can split non-weldable waypoints into two-dimensional nodes or three-dimensional nodes. Each node can then be evaluated for welding feasibility of the welding robot and weldment held at various welding angles and rotations. Additional rotation or other degrees of freedom about the x-axis and/or y-axis may make the waypoints accessible to the welding head such that the welding head does not encounter any collisions. In some implementations, the node is not feasible for welding one or more waypoints and/or between a previous waypoint node pair and one of the waypoint node pairs for a particular waypoint. positioning in determining a feasible path between the previous waypoint node pair and any of the waypoint node pairs for a particular waypoint if there is no feasible path to traverse the Controller 108 can use the degrees of freedom provided by the equipment system.

Based on the generated weld path, controller 108 can optimize the weld path for welding. (Optimal and optimize, as used herein, do not refer to determining the absolute best welding path, but generally to reducing welding time for less efficient welding paths.) , and/or techniques that can improve weld quality.) For example, controller 108 can determine a cost function that determines local minima and/or minima of robot 110 movement. Optimal welding paths minimize weld head rotation, as typically weld head rotation can increase the time to weld a seam and/or reduce weld quality. Accordingly, optimizing the weld path may include determining a weld path that passes through the maximum number of waypoints with the minimum amount of rotation.

In evaluating the feasibility of welding at each split node or node-waypoint pair, controller 108 may perform multiple calculations. In some examples, each of the multiple calculations may be mutually exclusive. In some examples, the first calculation may include a kinematic feasibility calculation, where the robot 110 arm of the employed welding robot is defined by a node or node-waypoint pair. calculate whether it is possible to mechanically reach (or exist) the state In some examples, in addition to the first calculation, a second calculation may also be performed by the controller 108, which may be mutually exclusive with the first calculation. The second calculation is based on whether the arm of robot 110 encounters a collision when accessing a joint (e.g., the node or node-waypoint pair in question) (e.g., workspace 101 or a structure within workspace 101). may include determining whether there is a conflict with

Controller 108 may perform the first calculation before performing the second calculation. In some examples, the second calculation is performed if the result of the first calculation is positive (e.g., the arm of robot 110 mechanically reaches the state defined by the node or node-waypoint pair). (or exists)). In some examples, if the result of the first calculation is negative (e.g., the arm of robot 110 mechanically reaches (or exists) the state defined by the node or node-waypoint pair) the second calculation may not be performed.

Kinematic feasibility may be correlated with the type of robotic arm employed. For purposes of this description, it will be assumed that welding robot 110 includes a six-axis robotic welding arm with a spherical wrist. A six-axis robot arm may have six degrees of freedom: three degrees of freedom in X, Y, Z Cartesian coordinates, and an additional three degrees of freedom due to the wrist-like nature of robot 110. For example, the wrist-like nature of the robot 110 may include a fourth degree of freedom in the vertical direction of the wrist (e.g., the wrist moves in +y and -y directions), a fifth degree of freedom in the lateral direction of the wrist (e.g., movement of the wrist in the -x and +x directions), resulting in a sixth degree of freedom of rotation. In some examples, the welding torch is attached to the wrist portion of robot 110.

In order to determine whether the arm of the robot 110 employed can mechanically reach (or exist) the state defined by the node or node-waypoint pair, i.e., a first calculation To do so, robot 110 may be mathematically modeled as shown in example model 800 of FIG. In some examples, controller 108 may solve the first three joint variables based on wrist position and the other three joint variables based on wrist orientation. Note that the torch is firmly attached to the wrist. Therefore, the translation between the torch tip and the wrist is assumed to be fixed. Geometric techniques (eg, the law of cosines) may be employed to find the first three joint variables (eg, variables S, L, U at 802, 804, 806, respectively).

After the first three joint variables (i.e., S, L, U) are successfully calculated, the controller 108 then calculates the last three joint variables (i.e., R, B, and T) at 808, 810, and 812, respectively, can be solved. Controller 108 may take into account several offsets within robot 110. These offsets may need to be considered and accounted for due to inconsistencies in the Uniform Robot Description Format (URDF) file. For example, in some instances, the position values (eg, the joint's X-axis) of a joint (eg, an actual joint of robot 110) may not match the values listed in its URDF file. Such offset values may be provided to controller 108 in a table. Controller 108 may, in some examples, take these offset values into account while mathematically modeling robot 110. In some examples, after the robot 110 is mathematically modeled, the controller 108 causes the arm of the robot 110 to mechanically reach (or exist) the state defined by the node or node-waypoint pair. You can decide whether or not you can.

As described above, the controller 108 allows the robot 110 to move within the workspace 101 containing a part 114, a fixture 116, or the robot 110 itself when placed in a position and orientation defined by a pair of waypoint nodes. You can evaluate whether it will collide with or interfere with others. Once the controller 108 determines the states in which the robot arm can exist, the controller 108 may use a second calculation to perform the aforementioned evaluation (e.g., regarding whether the robot will collide with anything in its environment). good.

FIG. 9 is a flowchart of a method 900 for performing autonomous welding, according to various examples. More specifically, FIG. 9 is a flowchart of a method 900 for operating and controlling a welding robot (eg, robot 110 of FIG. 1), according to some examples. At step 902, method 900 captures image data of a workspace (e.g., workspace 101 of FIG. 1) using one or more sensors (e.g., sensor(s) 102 of FIG. 1). Including obtaining. The image data may include 2D and/or 3D images of the workspace. As mentioned above, one or more parts to be welded, fixtures and/or clamps capable of safely holding the parts can be located in the workspace. In some examples, point clouds can be generated from image data. For example, three-dimensional image data can be reconstructed and generated by superimposing images on each other. Point clouds can be generated by comparing 3D image data with each other.

At step 904, method 900 includes identifying a set of points on the part to be welded based on the sensor data, which may be an image. The point set can represent possible seams to be welded. In some examples, a neural network can perform pixel-by-pixel segmentation on image data to identify point sets. Fixtures and clamps in the image data can be classified by a neural network based on image classification. Portions of the image data associated with fixtures and/or clamps can be segmented and separated such that those portions of the image data are not used to identify point sets, thereby reducing the search space. By doing so, the computational resources required to identify the point set to be welded can be reduced. In such instances, the point set can be distinguished from other portions of the image data (eg, portions of the image data that are not segmented apart).

At step 906, method 900 includes identifying candidate seams from the point set. For example, a subset of points within a point set can be identified as candidate seams. Neural networks can perform image classification and/or depth classification to identify candidate seams. In some examples, candidate seams may be located relative to the part. For example, to locate a candidate seam, the position and orientation of the candidate seam can be determined for the part.

Additionally, method 900 further includes verifying whether the candidate seam is an actual seam. As mentioned above, the sensor(s) can collect image data from multiple angles. For each image captured from a different angle, a confidence value can be determined representing whether the candidate seam determined from that angle is an actual seam. If the confidence value exceeds a threshold based on views taken from multiple angles, the candidate seam can be verified as an actual seam. In some aspects, method 900 also includes classifying the candidate seam as a seam type. For example, a neural network can determine whether a candidate seam is a butt seam, corner seam, edge seam, lap seam, T-seam, etc.

In some examples, after a candidate seam has been identified and verified, a subset of points can be clustered together to form an unbroken continuous seam. At step 908, method 900 includes generating welding instructions for the welding robot based on the candidate seams. For example, welding instructions may be generated by tracing a path from one end of the subset of points to the other end of the subset of points. This allows a path to be created for the seam. In other words, welding can be performed by tracing this path with the welding head. Additionally, path planning can be performed based on the identified and located candidate seams. For example, path planning can be based on seam paths that can be generated by clustering a subset of points.

In some examples, welding instructions may be based on seam type (eg, butt joint, corner joint, edge joint, lap joint, T-joint, etc.). In some examples, welding instructions can be updated based on input from a user via a user interface (eg, user interface 106 of FIG. 1). A user can select a candidate seam to be welded from all available candidate seams via the user interface. Path planning can be performed for the selected candidate seam and welding instructions can be generated for the selected candidate seam. In some examples, a user may update welding parameters via a user interface. Welding instructions can be updated based on updated welding parameters.

In this manner, a welding robot can be operated and controlled by performing method 900 without prior information (eg, a CAD model) of the parts to be welded. As the part is scanned to generate welding instructions, a scanned image representation of the part can be annotated (eg, via a user interface) with one or more candidate seams. Annotated representations can be used to define 3D models of parts. The 3D model of the part can be saved in a database for subsequent welding of additional instances of the part.

FIG. 10 is a flowchart of a method 1000 for performing autonomous welding, according to various examples. More specifically, FIG. 10 is a flowchart of a method 1000 for operating and controlling a welding robot (eg, robot 110 of FIG. 1), according to some examples. At step 1002, method 1000 includes identifying an expected orientation and expected location of a candidate seam on the part to be welded based on the CAD model of the part. The expected orientation and expected position may be determined using annotations provided on the CAD model by the user/operator. Additionally or alternatively, the controller (eg, controller 108 of FIG. 1) may be operative to identify candidate seams based on model geometry. Object matching can be performed to match components on a part to components in a CAD model. In other words, the expected location and orientation of candidate seams can be identified based on object matching.

At step 1004, method 1000 captures image data of a workspace (e.g., workspace 101 of FIG. 1) using one or more sensors (e.g., sensor(s) 102 of FIG. 1). Including obtaining. The image data may include 2D and/or 3D images of the workspace. As mentioned above, the workspace may include one or more parts to be welded and fixtures and/or clamps that can securely hold the parts. In some examples, point clouds can be generated from image data. For example, images can be superimposed on each other to reconstruct and generate 3D image data. 3D image data can be matched against each other to generate point clouds.

In some examples, the sensor is configured to perform partial scans to reduce processing time for generating welding instructions. In other words, image data is collected from a small number of angles (eg, angles from which candidate seams are expected to be visible) instead of scanning the workspace from every angle. In such an example, the point cloud generated from the image data is a partial point cloud. For example, scanning and/or processing time can be reduced by generating partial point clouds that do not include portions of the part that the model indicates do not include seams to be welded.

At step 1006, method 1000 includes identifying candidate seams based on the image data, point cloud, and/or point cloud. For example, controller 108 (FIG. 1) can identify candidate seams using the techniques described above.

At step 1008, method 1000 includes identifying the actual location and orientation of the candidate seam. For example, in step 1002, a first subset of points may be identified as a modeled seam. At step 1006, a second subset of points may be identified as candidate seams. In some examples, the first subset of points and the second subset of points can be compared (eg, using the registration technique described above with respect to FIG. 6). The first subset of points can be transformed to determine the actual location and orientation of the candidate seam. In some examples, the comparison of the first subset of points and the second subset of points determines the tolerance (e.g., expected location and expected orientation of the candidate seam) of the first subset of points. can help you decide. In such an example, the first subset of points may be deformed based on a tolerance to determine the actual position and orientation of the candidate seam. In other words, the expected position and orientation of the candidate seam can be refined (in some examples, based on tolerances) to determine the actual position and orientation of the candidate seam. This deformation/refinement technique can take into account surface topography on the part that is not accurately represented in the CAD model (eg, in the CAD model in step 1002).

At step 1010, method 1000 includes generating welding instructions for a welding robot based on the actual location and orientation of the candidate seam. For example, path planning can be performed based on the actual location and orientation of the candidate seam.

Similar to method 900 of FIG. 9, once the actual location and orientation of the candidate seam is identified, method 1000 can include verifying the candidate seam using the techniques described above. However, in contrast to method 900, user interaction in method 1000 may not be necessary. This is because one or more seams to be welded may already be annotated in the CAD model. Thus, in some cases, a welding robot may be operated and controlled by performing method 1000 without user interaction.

In addition to or in place of the steps described above with respect to method 1000, a welding robot (e.g., robot 110 of FIG. 1) operates and controls the welding robot to perform autonomous welding as follows. It's okay. The controller 108 of the robot 110, and in particular the robot 11, scans the workspace containing the part to determine the position of the part within the workspace (e.g., the position of the part on a positioning device) and to determine the representation of the part (e.g., point group representation). Controller 108 may be provided with an annotated CAD model of the part. Controller 108 may then determine the expected location and expected orientation of the candidate seam on the part according to (or based on) the computer-aided design (CAD) model of the part and the representation of the part. For example, the controller 108 may be operative to identify candidate seams based on model geometry--performing object matching to match representational components or features (e.g., topographical features) of the part to the CAD model. controller 108 may be operative to use this object matching in determining the expected location and expected orientation of the candidate seam on the part. Once the expected position and orientation are determined, controller 108 may then determine the actual position and orientation of the candidate seam based at least in part on the representation of the part. Actual position and orientation may be determined using the transformation/refinement techniques described in step 1008.

FIG. 11 is a flowchart of a method 1100 for performing autonomous welding, according to various examples. More specifically, FIG. 11 depicts an example method 1100 of operation of manufacturing robot 110 (FIG. 1) in a robotic manufacturing environment. One or more parts 114 on which a manufacturing task (e.g., welding) is to be performed is moved using a fixture 116 onto another fixture 116 (e.g., a positioning device) in the manufacturing workspace 101. Assumed to be positioned and/or fixed at . Following placement of one or more parts 114, method 1100 may begin. As a first step, method 1100 includes scanning one or more parts (eg, block 1110). The scanning may be performed by one or more sensors 102 (e.g., a scanner not coupled to the robot 110), and the controller 108 determines the position of the part within the manufacturing workspace 101 and scans obtained from the sensors. The image data and/or the point cloud derived from the image or sensor data may be configured to identify one or more seams on the part (blocks 1112 and 1114). Parts and seams may be located and identified based on one of the techniques described with respect to FIG. Once the locations of the parts and seams are determined, controller 108 plots a path for manufacturing robot 110 along the identified seams (block 1116). The plotted path may include optimized movement parameters for the manufacturing robot 110 to complete the weld without colliding with itself or others within the manufacturing workspace 101. No human input is required in generating optimized movement parameters for manufacturing robot 110 to complete the weld. The path/trajectory may be planned based on one of the path planning techniques mentioned above.

The terms "position" and "orientation" are described as separate entities in the above disclosure. However, the term "position" when used in the context of a part means "the particular way in which the part is placed or arranged." The term "position" when used in the context of a seam means "the particular way in which a seam on a part is positioned or oriented." Thus, the position of the part/seam may essentially take into account the orientation of the part/seam. Therefore, "position" can include "direction". For example, a position can include the relative physical position or orientation (eg, angle) of a part or candidate seam.

Unless otherwise specified, "about," "approximately," or "substantially" before a value means +/-10 percent of the stated value. Unless otherwise specified, two objects described as "parallel" are side by side and have a distance between them that is constant or varies by no more than 10 percent. Unless otherwise specified, two objects described as perpendicular intersect at an angle ranging from 80 degrees to 100 degrees. Modifications are possible in the examples described, and other examples are possible within the scope of the claims.