A robotic fruit picking system

Robotic Fruit Picking System

Background of the invention. Field of the invention

[0002] The field of the invention relates to systems and methods for robotic fruit picking.

[0003] Portions of the disclosure of this patent document contain material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyrights.

2. Background technology

[0004] Horticultural producers rely heavily on manual labor to harvest their crops. Many varieties of fresh produce are hand-picked, including berries such as strawberries and raspberries, asparagus, table grapes and apples that can be eaten raw. Manual picking is currently necessary because the product is easily damaged and requires delicate handling, or because the plants themselves are valuable, producing fruit continuously over one or more growing seasons. Therefore, efficient but destructive mechanical methods for harvesting crops such as wheat are not feasible.

[0005] The reliance on manual labor poses several problems for producers:

[0006] Recruiting pickers for a short, hard picking season is risky and expensive. The supply of domestic pickers is almost non-existent, so farmers must recruit from overseas. However, immigration controls place a huge administrative burden on producers and increase the risk of labor shortages.

[0007] The supply and demand of low-skilled immigrant labor are unpredictable as they depend on weather conditions throughout the growing season and the economic environment. This creates significant labor price volatility.

[0008] In extreme cases, this may result in crops not being harvested in the field. For example, a 250-acre strawberry farm near Hereford lost more than £200,000 of produce due to labor shortages in 2007.

[0009] Human pickers give inconsistent results, directly impacting profitability (for example, baskets containing strawberries of inconsistent size or shape or showing signs of mishandling are often rejected by customers). Farmers use a variety of training and monitoring procedures to improve consistency, but these add significant cost.

[0010] Current techniques for robotic soft fruit harvesting often rely on complex hardware and naive robotic control systems. As a result, other soft fruit picking systems have not been commercially successful because they are expensive and require carefully controlled environments.

[0011] A few groups have developed robotic strawberry picking technology. However, robots are often costly and still require human operators to grade and post-process the fruit. In addition, robots are often not compatible with the tabletop growing systems used in Europe and are too expensive to compete with humans. Problems with expensive hardware and antiquated object recognition technology, and lack of mechanical flexibility to pick items other than carefully positioned vertically oriented strawberries, or unsuitable for picking soft fruit that cannot be handled other than its stalk . Therefore no product was provided.

[0012] Thus, to date, most solutions fall into at least the following two key areas:

[0013] They require growers to significantly change their working practices, or do not support the tabletop growing systems used in Europe.

[0014] They rely heavily on human operators. As a result, they use large machines with a disproportionately high production cost per unit of picking capacity compared to small automated machines produced in high volume.

[0015] They are too expensive to replace labor at current prices.

[0016] Elsewhere, several academic groups have also applied robotics and computer vision techniques (mostly equivalent to the era of the 1980s) to more general harvesting applications. However, the resulting system was too limited for commercial development.

[0017] Farmers need a reliable system to harvest their crops on demand with consistent quality and predictable cost. Such a system would allow farmers to purchase high-quality, consistent harvest capacity upfront at predictable prices, reducing their exposure to labor market price volatility. The machine will operate autonomously: traverse a field, orchard or multi-aisle; identify and locate produce ready for harvest; pick select crops; and finally grade, sort and store the picked produce into containers suitable for transfer to refrigeration.

Contents of the invention

The first aspect of the present invention is a kind of robot fruit picking system, described robot fruit picking system comprises autonomous robot, and described autonomous robot comprises following subsystem:

a positioning subsystem operable to achieve autonomous positioning of the robot using a computer-implemented guidance system, such as a computer vision guidance system;

[0020] at least one picking arm;

at least one picking head or other type of end effector mounted on each picking arm to cut the stem or branch of a particular fruit or bunch or to pick said fruit or bunch and then transfer said fruit or fruit bunches;

Computer vision subsystem, described computer vision subsystem is used for analyzing the image of the described fruit to be picked or stored;

Control subsystem, described control subsystem uses picking strategy programming or learning picking strategy;

a quality control (QC) subsystem for monitoring the quality of picked or pickable fruit and grading the fruit according to size and/or quality; and

[0025] A storage subsystem for receiving picked fruit and storing the fruit in containers for storage or transport, or in fruit baskets for retail sale.

[0026] We use the term 'picking head' to cover any type of end effector; an end effector is a device or devices at the end of a robotic arm that interacts with the environment - for example, a head or heads that are used to pick fruit the edible and edible part of the fruit, or grasp and cut the stem of the fruit.

Description of drawings

Aspects of the present invention will now be described by way of example with reference to the accompanying drawings, which respectively illustrate features of the present invention:

[0028] Figure 1 shows a top view (A) and a perspective view (B) of a robot suitable for picking fruit.

[0029] FIG. 2 shows a perspective view of a robot suitable for fruit picking with the wall in a picking position.

[0030] FIG. 3 shows an example of a robot suitable for fruit picking.

[0031] FIG. 4 shows another example of a robot suitable for fruit picking.

[0032] FIG. 5 shows another example of a robot suitable for fruit picking.

[0033] Figure 6 shows an early embodiment of the invention designed for picking table grown strawberries.

[0034] Figure 7 shows a system for mounting a 'vector cable' to the leg of a table on which the crop is growing using a metal bracket that is simply clamped to the leg.

[0035] FIG. 8 shows a number of line drawings with different views of the picking arm and picking head, alone or in combination.

[0036] FIG. 9 shows a cross-sectional view of an exemplary embodiment of a QC imaging chamber.

[0037] FIG. 10 shows a diagram of a space-saving solution for storing fruit basket trays within a picking robot.

[0038] FIG. 11 shows the main components of the end effector.

[0039] FIG. 12 shows the hook extending relative to the gripper/cutter.

[0040] FIG. 13 shows retraction of the hook relative to the gripper/cutter.

[0041] FIG. 14 shows various parts of the end effector, including the blade above the hook.

[0042] FIG. 15 shows a diagram illustrating movement of the hook to achieve capture of a plant stem.

[0043] Figure 16 shows a diagram showing plant stems being captured.

[0044] Figure 17 shows a diagram showing a product being clamped and cut from a parent plant.

[0045] FIG. 18 shows a diagram illustrating a release operation.

[0046] FIG. 19 shows the sequence of operations that make up the picking process.

[0047] FIG. 20 shows the main mechanical components of the ring and clamp assembly.

[0048] Figure 21 shows the ring and clamp assembly with the ring extended as shown in part 1.

[0049] FIG. 22 shows an exploded view of the ring and major components of the clamp assembly (ring omitted for clarity).

[0050] FIG. 23 shows components of the ring actuation mechanism.

[0051] FIG. 24 shows the ring/clamp assembly towards the approach vector of the target fruit.

[0052] FIG. 25 shows that the ring extends and the assembly moves parallel to the main axis of the target product and in the direction of the junction between the stem and the fruit.

[0053] Figure 26 shows the loop having traveled past the juncture of the stalk and the fruit, the target product is now selected and decision step 1 (Figure 19) can be applied.

[0054] FIG. 27 shows the ring retracted to control the position of the target product and the pincers actuated to grasp and cut the stalk of the fruit in a scissors-like motion.

[0055] FIG. 28 shows different elements of the cable management system.

[0056] FIG. 29 shows a view of the cable management system in situ within one of the joints of the arm.

[0057] FIG. 30 shows a series of views in which the cable guide is rotated within the cable housing.

[0058] FIG. 31 shows a cross-sectional view of a cable winding.

Detailed ways

[0059] The present invention relates to an innovative fruit picking system using a robotic picking machine capable of picking fruit completely autonomously and working effectively alongside humans picking fruit.

[0060] While this description focuses on robotic fruit picking systems, the systems and methods described may have broader application in other fields, such as robotic cat litter picking systems.

[0061] The picking system is suitable for a variety of different crops grown on plants (eg strawberries, tomatoes), bushes (eg raspberries, blueberries, grapes) and trees (eg apples, pears, loganberries). In this document, the term fruit shall include the edible and palatable parts of all fruits, vegetables and other kinds of products (including nuts, seeds, vegetables) picked from plants, and plant shall mean fruit-producing crops of all kinds (including shrubs, trees). For fruits that grow in clusters or bunches (e.g. grapes,

berries), fruit can refer to individual fruits or whole clusters.

[0062] Many plants continue to produce fruit over a long picking season and/or several years of the plant's life. Therefore, picking robots must not damage the fruit or the plant it grows on (including any unpicked fruit, whether ripe or unripe). Damage to plants/shrubs/trees can occur when the robot moves around the plant or during picking operations.

[0063] Our development efforts have been directly influenced by the needs of real commercial growers compared to the prior art. We will avoid high-cost hardware by utilizing state-of-the-art computer vision techniques, allowing us to use lower-cost off-the-shelf components. The appeal of this approach is that the marginal cost of making software is lower than the marginal cost of making complex hardware.

[0064] An intelligent robotic position control system has been developed that is capable of operating at high speeds without damaging the delicate picking fruit. While typical naive robotic control systems are great for performing repetitive tasks in controlled environments such as automotive factories, they cannot handle the variability and uncertainty inherent in tasks such as fruit picking. We address this problem using a state-of-the-art reinforcement learning approach that will allow our robotic control system to use the experience gained during picking to learn more efficient picking strategies.

The key parts of fruit picking robot are as follows:

[0066] A tracked rover capable of autonomously navigating along crop rows using a vision-based guidance system.

[0067] A computer vision system including a 3D stereo camera and image processing software to detect target fruits and decide whether to pick them and how to pick them.

[0068] A fast, 6-DOF robotic arm for positioning the picking head and camera.

[0069] A picking head comprising (i) means for cutting strawberry stalks and (ii) grasping the cut fruit for transfer.

[0070] A quality control subsystem for grading picked strawberries by size and quality.

[0071] A packaging subsystem for carrying out airborne basket loading of picked fruit.

Picking robot performs multiple functions fully automatically:

loading and unloading itself into and out of the transport vehicle;

Navigating among fruit producing plants, for example along rows of apple trees or strawberry plants;

Collaborate with other robotic and human pickers to efficiently divide the picking work;

Determining the position, orientation and shape of the fruit;

determine whether the fruit is suitable for picking;

separating ripe fruit from the tree;

Grading of fruit by size and other measures of suitability;

[0080] Transfer the picked fruit to a suitable storage container.

[0081] The picking system is innovative in several respects. In the following, some specific non-obvious inventive steps are highlighted with wording such as "A useful innovation is...".

1. system overview

Picking system comprises following important subsystems:

[0084] total positioning subsystem

Picking arm

Picking head

[0087] Computer Vision Subsystem

[0088] Control Subsystem

[0089] Quality Control (QC) subsystem

[0090] storage subsystem

[0091] Mapping subsystem

Management subsystem

[0093] The main purpose of a robot global positioning system is to physically move the entire robot along the ground. When the robot is within range of the target fruit, the picking arm moves an additional camera to allow the computer vision subsystem to locate the target fruit and determine its pose and suitability for picking. The picking arm also positions the picking head for picking and moves the picked fruit to the QC subsystem (and possibly the storage subsystem). The overall positioning system and picking arm operate under the control of the control subsystem, which uses input from the computer vision subsystem to decide where and when the mobile robot is. The main purpose of the picking head is to cut the fruit from the plant and hold it firmly for transfer to the QC and storage subsystems. Finally, the QC subsystem is responsible for grading the picked fruit for retail or other use and discarding unusable fruit.

[0094] Figure 1 shows a top view (a) and a perspective view (b) of a robot suitable for picking fruit. The robot includes a positioning subsystem operable to achieve autonomous positioning of the robot using a computer-implemented guidance system, such as a computer vision guidance system. Two picking arms (100) are shown for the described configuration. A picking head (101) is mounted on each picking arm (100) to cut the stem or branch of a particular fruit or bunch or to pick said fruit or bunch and then transfer the fruit or bunch. The picking head also includes the camera component (102) of the computer vision subsystem, which is responsible for analyzing the images of the fruit to be picked or stored. The control subsystem uses or learns the picking strategy. The quality control (QC) subsystem (103) monitors the quality of the picked or ready-to-harvest fruit and grades the fruit according to size and/or quality, and the storage subsystem (104) receives the picked fruit and stores it in Containers for storage or transport, or stored in fruit baskets for retail sale. Figure 2 shows a perspective view of the robot with the arm in a picking position.

[0095] Figures 3 to 5 show examples of robots suitable for picking fruit. Three different concept drawings show the robot configured in a number of different ways for different picking applications. Important system components shown include the tracked rover, the two picking arms and the associated quality control unit (QC). A plurality of trays are used to store baskets of picked fruit and are positioned such that a human operator can easily remove entire trays and replace them with empty trays. The disposal stream is located near the quality control unit for fruit that is not fit for sale. When a robot picks rotten or otherwise unsuitable fruit (by accident or design), it is often desirable to dispose of the rotten fruit into a suitable container inside the robot or on the ground. A useful time-saving innovation is to make the containers accessible through the disposal chute, the hole of which is located at the bottom of the QC unit, so that the arm can immediately drop the fruit without moving to an alternate container. A related innovation is to induce positive or negative air pressure (eg using a fan) in the chute or body of the imaging chamber to ensure that fungal spores from previously discarded fruit are kept away from healthy fruit in the imaging chamber. The two picking arms can be positioned asymmetrically (as shown in Figure 5) to increase speed of reach at the expense of picking speed.

[0096] Figure 6 shows an early embodiment of the invention designed for picking table grown strawberries and having a single picking arm and two storage trays.

[0097] These subsystems will be described in more detail in the following sections.

2. total positioning subsystem

[0099] The overall positioning subsystem is responsible for the movement of the entire robot on the ground (usually on the expected straight line between the current position and the input target position). The management subsystem uses the general positioning subsystem to move the robot. The overall positioning system includes the means to determine the current position and orientation of the robot, the means to realize the movement of the robot along the ground, and the control system to convert the information about the current position and orientation of the robot into motor control signals.

2.1. Pose Determination Component

[0101] The purpose of the pose determination component is to allow the robot to determine its current position and orientation in map coordinates for input to the control components. Coarse position estimates can be obtained using differential GPS, but these are not accurate enough to follow crop rows without collisions. Thus, a combination of additional sensors is used to more accurately determine heading along the row and lateral distance from the row. The combination may include an ultrasonic sensor for approximately determining distance from the row, a magnetic compass for determining heading, an accelerometer, and a forward or backward facing camera for determining direction relative to the crop row. Information from the sensors is fused with information from the GPS positioning system to obtain a more accurate estimate than either GPS or the sensors would obtain alone.

[0102] An innovative method that allows a robot to estimate its position and orientation relative to a row of crops is to measure its displacement ('vector line cable').

[0103] An innovative approach to measuring the displacement of the robot relative to the vector cable is to use a computer vision system to measure the projected position of the cable in the 2D image obtained by the mounted camera using the known position and orientation in the robot coordinate system . As a simplified illustration, the orientation of the horizontal cable in the image obtained by the vertically oriented camera has a simple linear relationship to the orientation of the robot. Typically, the edges of the cable will project onto a pair of lines in the image, which can be easily found by standard image processing techniques, for example by applying an edge detector and computing a Hough transform to find the long straight edges. The image positions of these lines are a function of the wire diameter, the pose of the camera relative to the wire, and intrinsic parameters of the camera (which can be predetermined). The pose of the camera relative to the cable can then be determined using standard optimization techniques and providing initialization by assuming that the robot is approximately aligned with the row. Knowing the approximate height of the camera above the ground, the remaining one-parameter blur (corresponding to the rotation of the camera about the cable axis) can be removed.

[0104] Another innovative method of determining position relative to a vector cable is to use a follower arm (or follower arm). It is connected by an articulated joint to the robot chassis at one end and to a truck running along a cable at the other end. The angle at the hinged joint (which can be measured, for example, using the resistance of a potentiometer that rotates with the hinge) can be used to determine the displacement relative to the cable. Two follower arms (e.g. one at the front and one at the rear) are sufficient to determine displacement and orientation.

[0105] A related innovation is a stand that allows the vector cable to be easily attached to the legs of a table where crops such as strawberries are commonly grown. This is shown in Figure 7. The brace allows the cable to be positioned a small, consistent distance above the floor, typically 20cm, so that a human can easily step over it while ducking under the elevated table and moving from row to row. A limitation of this approach is that if the follower arm becomes detached from the cable, the robot may receive false position information. A useful additional innovation is therefore to equip the truck with a microswitch positioned to break the circuit when the truck loses contact with the cable. This can be used to allow the control software to detect this failure condition and stop the robot (typically, the control software waits until it detects a loss of contact between the truck and the cable for a duration greater than some on the Bounce Truck). Since the follower arm may be subject to significant forces in the event of a crash or other type of failure of the control system, another useful innovation is to attach the outside of the follower arm to the inside using a magnetic coupling. Then, in the event of a failure, the parts of the follower arm can separate without permanent damage. Magnetic coupling can include the electrical connection needed to complete an electrical circuit (often the same electrical circuit interrupted by a microswitch in the truck). In this way, the separation of the follower arm can also trigger the control software to stop the robot. Another benefit of the magnetic coupling arrangement is the ease of attachment of the follower arm by a human supervisor.

[0106] An innovative aspect of the pose determination component is a computer vision-based system for determining the robot's advance and landscape position. Such a system can be used to drive a robot halfway between two crop rows or at a fixed distance from individual crop rows. In one embodiment of this concept, this is accomplished by training a regression function implemented using a convolutional neural network (or otherwise) to predict the heading and lateral position of the robot relative to the crop row from input images. Can be controlled by human remote control

A robot equipped with multiple forward and/or backward facing cameras is driven between representative crop rows to obtain training data. Because the human controller keeps the robot roughly centered between the rows (with heading parallel to the rows), each frame can be associated with approximate ground truth heading and lateral displacement information. Multiple cameras are used to provide training images corresponding to different approximate lateral displacements from the row. Training images corresponding to different robot captions can be obtained by resampling images obtained by the forward-looking camera with an appropriate 3×3 homography (which can be trivially computed from known intrinsic camera calibration parameters). [0107] A related innovation is the use of an infrared illuminator and a suitable infrared receiving camera to obtain additional training image data at night.

[0108] In another embodiment of this concept, the computer vision system is designed to detect (in images obtained by either the forward or backward facing cameras) the vertical legs of a table growing crops. Vertically oriented table legs define vertical lines in the world, and they project onto lines in perspective. Assuming that the legs of each table are evenly spaced, vertically oriented and arranged in a straight line, the projected image lines corresponding to sequences of three or more table legs are sufficient to determine the orientation of the calibrated camera and its lateral displacement with respect to the 3D coordinates defined by the legs Tie.

Figure 7 shows a system for mounting 'vector cables' to the legs of a table on which crops are grown (a) using metal brackets that simply clip onto the legs ( b). An n-shaped metal spring clip or strap (not shown) may be used to secure the cable (eg nylon cord) to the metal bracket. The following arm (seen in b) consists of a truck resting on a cable and an arm that attaches the truck to a robot (not shown). The brackets are shaped so that trucks will not be hindered when traversing them.

2.2. Motor control components

[0111] The purpose of the motor control component is to map the pose information provided by the pose determination component to motor control signals to move the robot in a given direction. It supports two kinds of motion: (i) by moving a given distance along a row of plants, and (ii) by moving to a given point along a desired straight line. The motor control system uses a PID controller to map the control input obtained from the pose determination component to the motor control signal.

2.3. Rover

[0113] An important component of a global positioning system is the rover, the means by which the robot moves on the ground. Typically, movement over the ground is achieved using powered wheels with tracks. A useful innovation is the mechanism that allows the rails to be removed so that the robot can also run on the rails.

3. picking arm

[0115] The picking arm is a robotic arm, with several (typically 6) degrees of freedom, mounted on the main body of the robot. While the purpose of the full positioning system is to move the entire robot along the ground, the purpose of the picking arm is to move the picking head (and its computer vision camera) into position to locate, position and pick the target fruit. Once it is in the picking position, the picking head performs a picking procedure that includes a series of mechanical actions including separation, gripping and cutting (refer to the description of the picking head below). The control subsystem selects picking locations to maximize picking performance based on desired measurements.

[0116] Before the picking head can be positioned to pick the target fruit, the computer vision subsystem must perform several important operations: (i) detect the target fruit, (ii) detect obstacles that may complicate the picking of the target fruit (e.g. leaves) , (iii) determine the pose and shape of the target fruit. To enable the computer vision subsystem to perform these functions, the picking arm can be equipped with a monocular or stereo camera, for example mounted at the end of the arm. The advantage of mounting the camera on the arm is that the camera can be moved to find an observation point that is free of occluded sources that would otherwise prevent reliable detection or localization of the fruit of interest (leaves, other fruit, etc.).

[0117] Finally, the picking arm must move the picking head into the proper posture for picking without colliding with the plant or the supporting infrastructure used to grow the plant or itself. This is achieved using the routing planning algorithm described in the Control Subsystem section below.

[0118] FIG. 8 shows multiple line drawings with different views of the picking arm and picking head, alone or in combination. The picking head shown includes the hook and camera system.

4. picking head

[0120] The purpose of the picking head is to cut the target fruit from the plant, hold it securely and release it while moving it to the QC and storage subsystems. Secondary objectives are to move foliage and other sources of occlusion away to allow detection and location of fruit, and to separate target fruit from the plant (before it is permanently cut off) to facilitate determination of suitability for picking.

[0121] Picking soft fruit such as strawberries is challenging because physical handling of the fruit can cause bruising, reducing marketability. Therefore, it is ideal to pick this fruit by cutting off the stem without dealing with the body of the fruit. An innovative aspect of our system is the use of a dry cutting picker head which works in three stages ('grab-grip-cut'):

1. Physical separation of the fruit from the rest of the plant ('grabbing'). This step facilitates (i) deciding whether the fruit is suitable for picking and (ii) increasing the chances that picking in step 2 will proceed successfully without damaging the remaining plant's target fruit before permanently cutting off the plant's fruit.

2. Grasp the picked fruit by its stem ('grab').

3. Severing the stem (above the point where it is grasped) to permanently separate the fruit from the plant ('cutting').

[0125] The introduction of a physical separation stage (occurring before the fruit is permanently severed from the plant) confers several benefits. Since the target fruit may be clogged with leaves or other fruit, pulling it off plan allows for a better view, allowing the computer vision system to more reliably determine whether the fruit is ready for picking and whether the picking procedure is likely to be successful (e.g., because other fruit is in the near the target). A related innovation is a mechanical gripper that can rotate the grasped fruit during said pre-picking inspection phase, for example by applying a twisting force to its stalk or otherwise. In this way, a camera or other sensor can gain information about parts of the fruit that would otherwise be invisible. One benefit of this innovation is that it can decide to delay picking fruit that looks immature on the hidden side.

[0126] A possible further innovation is to combine the grasping and cutting phases (2 and 3) by utilizing a clamping action to pull the stem onto one or more cutting blades.

[0127] Appendix A describes several innovative mechanical pickup designs incorporating some of these ideas.

[0128] For some soft fruits, such as raspberries, it is necessary to remove the fruit from the stem during the picking process. A useful innovation for such fruit is to pick the fruit by first cutting off and grasping its stem to then remove the body of the fruit from its stem in a subsequent operation. Compared to picking techniques that require holding or grasping the fruit body, there are important benefits if this method includes: (1) Minimizing handling of the fruit body (which can significantly reduce shelf life, e.g. due to pathogenic transfer) Treating device surfaces pathogens from fruit to fruit), (ii) the possibility to image the body of the picked fruit from all directions for quality control, and (iii) the possibility to remove the stem under controlled conditions.

[0129] Various methods can be used to remove the picked fruit from the stem. One innovative method is to use a compressed air spray to pull the fruit from its stem. This allows contact forces to be evenly distributed over a large contact area, minimizing bruising. Another possibility is to pull the fruit through a collar that is shaped to easily push the body of the fruit away from the stem. Depending on the specific fruit type, the collar may be designed to distribute the contact force over a large area of the fruit's body or to concentrate the contact force (perhaps by cutting edges similar to a row of needles) in a circular pattern around the stem. Related innovations are cleaning the collar after each use or providing the collar with a disposable surface to reduce the possibility of pathogen transfer from fruit to fruit.

[0130] Another innovation is to use the inertia of the body of the fruit to separate the body of the fruit from the container. This can be achieved by holding the fruit by its stalk and rotating it about an axis perpendicular to its stalk at a sufficiently high angular velocity. The advantage of this method is that the inertial force effectively acts on the entire mass of the fruit body, eliminating the need for contact forces on the surface (more likely to cause bruising because

They are applied over small contact areas, increasing localized pressure). A limitation of this method is that when the body of the fruit separates from the container, it flies away at a velocity tangent to the circle of rotation, which requires some means of stopping its motion slowly enough that it is not affected by bruises. Therefore, another innovation is to use the fruit or its stem in a reciprocating back-and-forth motion or an oscillating rotational motion in a direction substantially perpendicular to the stem, wherein the axis of rotation is substantially parallel to the stem. By making movements with proper frequency and proper amplitude, the main body of the fruit can be reliably separated from the shell without causing the fruit to fly out at high speed.

[0131] After picking, the picking head grabs the fruit as the picking arm transfers to the quality control (QC) subsystem. In a simple embodiment, the picking arm itself may be used to position the picked fruit within the imaging assembly of the QC subsystem before subsequently moving the fruit to storage. However, the time spent transferring the picked fruit to the QC/storage subsystem is wasted since the arm is not used for picking during the transfer. Therefore, a useful innovation is to include multiple picking units on individual multiplexed picking heads. This means that several fruits in the vicinity of a particular locality can be picked in succession before the arm has to perform the time consuming movement between the plant and the QC/storage unit and back. This means that transfer overhead can be amortized over more fruit picked, increasing productivity. This is especially advantageous in the common case of fruit bunches on plants/trees such that the robotic arm needs to move only a short distance to pick a few targets before transferring.

[0132] The picking units on the multi-way picking head must be arranged so that inactive picking units do not interfere with the operation of active picking units, or collide with arms or other objects. Innovative ways to achieve this include:

[0133] Mounting the picking units radially about an axis selected such that the inactive picking units are remote from the active picking units and pick the fruit.

[0134] Make each picking unit extend independently so that it can engage the fruit without other picking units interfering with the field.

[0135] Picking units typically have multiple moving parts, eg for hooking, cutting, etc., which may require independent drives. However, when multiplexing multiple units on individual picking heads, if each moving part is driven by its own actuator, the arm payload increases proportionally to the number of picking units, which will adversely affect Arm speed, accuracy and overall cost. Several innovative aspects of implementing the multi-head keep the overall mass of the multi-head low for fast and accurate arm movement:

[0136] Multiple picking functions on a picking unit may be driven by a single actuator or motor, selectively engaged by light-weight members, such as electromagnets; pins; rotating plates; or the like. This is challenging because different functions may require different actuator characteristics

[0137] A single motor or actuator can drive a function on all units on the head that are selectively engaged by electromagnets, pins, rotating plates, etc. It's fairly simple.

[0138] These functions can be actuated from elsewhere in the system by lightweight means, for example using Bowden cables, torsionally actuated cables/springs, pneumatic or hydraulic components.

5. Computer Vision Subsystem

[0140] The purpose of the computer vision subsystem is to locate the target fruits, determine their pose in the robot coordinate system, and determine whether they are suitable for picking, ie before the fruit is permanently separated from the plant.

[0141] To achieve this, the computer vision subsystem uses one or more cameras mounted on the end of the movable picking arm (or generally any other part of the robot, such as the chassis). A camera attached to a robotic arm moves under computer control in order to detect target fruits, estimate their pose (i.e. position and orientation), and determine that they may be suitable for picking. As the arm moves, the pose estimation and picking suitability metrics associated with each target fruit can be gradually refined. However,

This refinement stage takes time, which increases the time needed for picking. Therefore, an important innovation is a scheme for efficiently moving the robotic arm to optimize the trade-off between picking speed and picking accuracy (described in more detail in the Robot Control Subsystem section below).

[0142] The computer vision subsystem operates under the control of the robot control subsystem, which makes continuous decisions about what actions to perform next, such as moving an arm-mounted camera to a new viewpoint in order to find more more target fruit, or move the camera to a new point near the local area of the target fruit in order to improve the estimation of its position/orientation or picking suitability index.

In a nutshell, the working principle of the computer vision subsystem is as follows:

1. Under the control of the robot control subsystem, the camera captures images of the scene from multiple viewpoints.

2. Detect target fruit in captured image by pixel-by-pixel semantic segmentation.

3. For each detected fruit, recover an approximate estimate of pose and shape.

4. By combining information from multiple views with statistical prior knowledge, more accurate pose and shape estimates can be recovered. This was achieved by tuning the parameters of the generative model of strawberry appearance to maximize the agreement between predictions and images.

5. Estimate the probability of picking success for each detected fruit from visual and geometric cues.

6. Under the control of the control subsystem, additional images of specific target fruits can be captured from new viewpoints in order to increase the probability of successful picking.

[0150] The important steps are described in more detail below.

[0151] Image capture. An important challenge is controlling the exposure of the camera system to obtain consistently correctly exposed images. Failure to do so would increase the variability of the image, compromising the ability of machine learning-based object detection software to detect fruit accurately and reliably. One exposure control strategy is to acquire images of gray cards exposed to ambient lighting conditions. The image is then analyzed to determine adjustments to exposure times and/or color channel gains needed to ensure that the gray card appears to have predetermined target color values. A gray card can be placed on the robot chassis and can reach the picking arm and be oriented horizontally to measure the ambient lighting arriving from the approximate direction of the sky. However, a potential limitation of the described method is that the illumination of the gray card may not indicate the illumination of the plant or target fruit. Thus, in systems where a (stereo) camera is incorporated within the picking head, a useful innovation is to arrange that part of the picking head itself can be used as an exposure control target. The prerequisite is that the exposure control target must appear within the field of view of the camera. Suitable targets could be a gray card for imaging from the front or a translucent plastic diffuser for imaging from below.

[0152] Real-world lighting conditions can affect image quality, limiting the effectiveness of image processing operations, such as target fruit detection. For example, images obtained with a camera pointed directly at the sun on a cloudless day may exhibit lens flare. Thus, a useful innovation is to have the control system software use the weather forecast to schedule picking operations such that the robot's camera system is oriented to maximize the quality of the image data obtained as a function of expected lighting conditions over a given period of time. For example, on a farm where fruit grows in rows, on a day when it is predicted to be sunny, a robot might pick on one side of the row in the morning and on the other side of the row in the afternoon. On cloudy days, the robot can more efficiently pick on both sides of the row at the same time to amortize the cost of propelling the robot along the row with more fruit picked at each location. A related innovation is to dynamically adjust the viewpoint according to lighting conditions to maximize picking performance. Picking heads, for example, can be tilted downward in direct sunlight to avoid lens flare, even at the expense of reduced working volume.

[0153] Target detection. Automatic detection of target fruit in images obtained by a camera mounted on the picking arm or elsewhere. Machine learning methods are used to train detection algorithms to identify fruit in RGB color images (and/or depth images obtained by dense stereo or otherwise). To provide training data, images obtained from representative viewpoints are

Position and/or extent are manually annotated. Various implementations of this concept are possible:

1. A decision forest classifier or a convolutional neural network (CNN) can be trained to perform semantic segmentation, i.e. label pixels corresponding to ripe fruit, unripe fruit and other objects. Pixel labeling can be noisy, and evidence can be aggregated across multiple pixels by using clustering algorithms.

2. A CNN can be trained to distinguish image patches that contain the target fruit at their center from image patches that do not. A sliding window method can be used to determine the locations of all image patches that may contain the target fruit. Alternatively, semantic labeling algorithms1 can be used to identify possible image locations of target fruits for subsequent more accurate classification by (often computationally more expensive) CNNs.

[0156] Target pose determination. Picking heads for different types of fruit can function in different ways, for example by cutting the stalk or by twisting the fruit until the stalk is severed (see above and Appendix A). Depending on the picking head design, picking a target fruit may require first estimating the position and orientation (or pose) of the fruit or its stalk (in the following, fruit should be interpreted as the body of the fruit or its stalk, or both). A rigid body construction typically has 6 degrees of freedom (eg, the X, Y, Z coordinates of the fruit in a suitable world coordinate system and three angles describing its orientation relative to the world coordinate system's axes). Pose can be modeled as a 4x4 homography that maps homogeneous 3D points in the appropriate fruit coordinate system to the world coordinate system. The fruit coordinate system can be easily aligned with a specific type of fruit. For example, the origin of the coordinate system may be located at the intersection of the main body of the fruit and its stalk, and the first axis points in the direction of the stalk. Many types of fruit (such as strawberries and apples) and most kinds of fruit stalks have a shape with an approximate axis of rotational symmetry. This means that 5 degrees of freedom usually provide a sufficiently complete pose representation for picking purposes, i.e. the second and third axes of the fruit coordinate system can be oriented arbitrarily.

[0157] The robot uses images obtained from multiple viewpoints to determine the pose of the target fruit, for example using a stereo camera or a monocular camera mounted to a moving picking arm. For example, the detected position of the target fruit in two or more calibration views is sufficient to approximately determine its X, Y, Z position by triangulation. The orientation of the fruit or its stalk can then be estimated either through assumptions (eg, the assumption that the fruit hangs vertically) or recovered from image data.

[0158] A useful innovation is to use a learned regression function to directly map images of target fruits to their orientation in the camera coordinate system. This can be achieved using a machine learning approach where a suitable regression model is trained to predict two angles describing the orientation of the approximately rotationally symmetric fruit from images including monocular, stereo and depth images. This approach is effective for fruits such as strawberries, which have surface textures aligned with the main axis of the fruit. Suitable training images can be obtained using a camera mounted at the end of the robot arm. First, the arm is manually moved until the camera is roughly aligned with the appropriate fruit-based coordinate system and is at a fixed distance from the fruit's centroid. The arms are aligned so the fruit has a canonical orientation in the camera image, i.e., two or three angles used to describe the orientation in the camera coordinate system are 0. Then, the arm moves automatically, To obtain additional training images from different novel viewpoints, the relative orientations of the fruits are known. Sufficiently high-quality training data can be obtained by having a human operator visually judge the alignment between the camera and the fruit coordinate system by inspecting the scene and the video signal produced by the camera. Usually the training images are cropped such that the centroid of the detected fruit appears in the center of the frame and scaled such that the fruit occupies a constant size. A convolutional neural network or other regression model is then trained to predict fruit orientation in previously unseen images. Various image features provide information about the orientation of the fruit in the camera image frame (and can be exploited automatically by suitable machine learning methods), such as the density and orientation of any seeds on the surface of the fruit, calyx (the leaf-like part around the stem) location and the image location of the fruit stalk.

[0159] Since knowledge of the orientation of the fruit stalk can be very important for picking certain types of fruit (or other information about the orientation of the fruit's body), another useful innovation is to identify and delineate the fruit stalk in an image The fruit handle detection algorithm. A pixel-wise semantic labeling engine (e.g., a decision forest or CNN) can be trained by using manually annotated training images

to identify pixels located on the central axis of the stalk to implement a stalk detector. A line growing algorithm can then be used to delineate the visible portion of the stalk. If stereo or depth images are used, the fruit stem orientation can be determined in the 3D coordinate system by matching corresponding lines to individual fruit stems in two or more frames. Understanding the dense stereo matching problem is greatly facilitated by first performing a semantic segmentation of the scene (stem, target fruit). Assumptions about the range of depths likely to be occupied by fruit stems can be used to constrain the stereo matching problem.

[0160] Given an approximate pose estimate of the target fruit, it is possible that obtaining additional views will improve the pose estimate, e.g. by revealing informative parts of the fruit, such as the point where the handle is attached. A useful innovation is therefore an algorithm for predicting the degree to which additional views in the set of available viewpoints will most significantly improve the quality of the initial pose estimate. Pose estimates obtained using multiple views and statistical prior knowledge about the likely shape and pose of the target fruit can be fused using innovative model fitting methods (see below).

[0161] Size and shape determination and pose estimation refinement. Whether a target fruit is suitable for picking may depend on its shape and size, for example because a customer wants fruit within a certain range of diameters. Furthermore, certain parameters of the picking system may need to be adjusted taking into account the shape and size of the fruit, such as the trajectory of the picking head relative to the fruit during the initial 'grasping' phase of the picking movement (cf. above). Therefore, it may be beneficial to estimate the shape and size of the candidate fruit prior to picking and to refine the (possibly rough) pose estimate as determined above. This can be achieved using images of the fruit (including stereoscopic images) obtained from one or more viewpoints.

[0162] An innovative approach to recovering the 3D shape of a candidate fruit from one or more images is to tune the parameters of a generative model of fruit image appearance to maximize the agreement between the image and the model prediction, e.g. by using Gauss-Newton optimization . The method can also be used to refine a rough initial estimate of fruit position and orientation (provided above). A suitable model could be in the form of a (possibly textured) triangular 3D mesh projected into some perspective. The shape of the 3D mesh can be determined by a mathematical function of some parameters describing the shape of the fruit. A suitable function can be constructed by obtaining 3D models of a large number of fruits, followed by principal component analysis (or other dimensionality reduction strategies) to discover low-dimensional parameterizations of the fruit geometry. Another simpler but effective approach is to make such a model by hand, e.g. by assuming that the 3D shape of fruit can be interpreted as a volume of revolution applying parametric anisotropic scaling in a plane perpendicular to the axis. A suitable initialization for optimization can be obtained by using the 2D image shape (or the average 2D image shape of the fruit) to define the volume of rotation. The pose parameters can be initialized using the method described above. A major benefit of model fitting methods is that information from multiple viewpoints can be combined simultaneously. The agreement between real and predicted images can be measured, for example using the distance between real and predicted silhouettes, or for models including lighting or textures, measuring the sum of squared differences between pixel intensity values. A useful innovation is the use of geometric models to predict not only the surface appearance of the fruit, but also the shadows the fruit casts on itself under different controlled lighting conditions. Controlled lighting may be provided by one or more illuminators attached to the end of the robotic arm. Another useful innovation is to model the agreement using a composite cost function that includes terms that reflect the agreement between the contour and the stem.

[0163] Another benefit of the model fitting approach is that image evidence can be combined with statistical prior knowledge to obtain maximum likelihood estimates of shape and pose parameters. Statistical prior knowledge can be incorporated by penalizing unlikely parameter configurations that are unlikely according to the probability model. A valuable innovation is for pre-statistical purposes, simulating the way a mass of fruit hangs from its stalk under the influence of gravity. In a simple embodiment, the prior may reflect our knowledge that fruits (particularly large fruits) tend to hang vertically downward from their stalks. Such a prior can take the simple form of a probability distribution rather than a fruit orientation. More complex embodiments may take the form of joint distributions about the shape and size of the fruit, the pose of the fruit, and the shape of the stalk near the point of attachment of the fruit. Suitable probability distributions are usually formed by taking geometric measurements of fruit growth under representative conditions.

[0164] Some picking heads are designed so that candidate fruit can be physically separated from the plant and other fruit in the bunch prior to picking (see above). As an example, the 'hook' design of the picking head (see Appendix A) allows candidate fruit to be supported by its stalk, allowing it to hang at a predictable distance from a camera mounted to a robotic arm. A benefit of this innovation is that images (or stereo images) of the fruit can be captured from controlled viewpoints, helping to more accurately determine size and shape, for example through the shape of a silhouette.

[0165] Determination of Suitability for Picking. Attempts to pick the target fruit may or may not be successful. Successful picking generally means that (i) the picked fruit is fit for sale (e.g. ripe and undamaged) and transported to storage containers in stated conditions, (ii) no other parts of the plant or growing infrastructure were damaged during picking, and (iii) The picking arm is free from any collisions that might interfere with its continued operation. However, where rotting fruit is picked and discarded to prolong the life of the plant, it is not required that the picked fruit be in a marketable condition.

[0166] A valuable innovation is to determine the picking suitability of a target fruit by estimating the statistical probability of attempting to pick the target fruit. The probability can be estimated before attempting to pick the target fruit through a particular approach trajectory, so the control subsystem can use the probability to decide which fruit to pick next and how to pick it. For example, the easiest fruits to pick (ie the most likely to be successfully picked) may be picked first so that harder to pick fruits may be picked later, for example because they are partially hidden behind other fruits. The probability of picking success estimate may also be used to decide not to pick a particular target fruit, for example because the expected cost of picking in terms of damage to the plant or the picked fruit would not outweigh the benefit provided by picking one more fruit. The control subsystem is responsible for optimizing the picking schedule to achieve the best balance between picking speed and failure rate (see below).

[0167] An important innovation is a scheme that uses scene images obtained from viewpoints near the target fruit to estimate the probability of picking success. As an example, we could image the surface of a fruit by moving a camera (perhaps a stereo or depth camera) mounted at the end effector of a picking arm near its location. Various image measures can be used as indicators of the probability of picking success, including, for example, (i) the estimated pose and shape of the fruit and its stalk, (ii) the uncertainty associated with the recovered pose and shape estimates, (iii) the color The surface of the target fruit, (iv) the proximity of the detected obstacles, and (v) the observation range where the candidate fruit is visible.

[0168] A suitable statistical model for estimating the probability of picking success may take the form of a multivariate histogram or a Gaussian histogram defined over the space of all picking success indicators. An important innovation is to use picking success data obtained by worker robots to learn and improve the parameters of such a model. Since the quality control subsystem provides accurate judgments about the marketability of picked fruit, its output can be used as an indicator of successful or failed ground truth picking. As more data is generated, the model can be dynamically updated using online learning methods to quickly adapt picking behavior to the needs of a new farm or growing season. Multiple robots can share and update the same model.

[0169] Since the picking head may approach the target fruit through a range of possible trajectories (depending on the geometry of the obstacles and the degrees of freedom of the picking arm), the probability of picking success is modeled as a function of the assumed approach trajectories. In this way, the control subsystem can decide how to pick the fruit to achieve the best balance between picking time and picking success probability. During path planning operations, the probability of collision between the picking arm and the scene can be modeled using an explicit 3D model of the scene (as described in the Control Subsystem section below). However, another innovative approach is to use an implicit 3D model of the scene formed by the range of viewpoints from which the target fruit can be observed without occlusion. The fundamental insight is that if the target fruit is fully visible from a particular viewpoint, then the volume defined by the backprojection of the perimeter of the 2D image of the fruit must be empty between the camera and the fruit. Find unobstructed regions of space by identifying one or more viewpoints where the target fruit is not occluded. If no part of the picking head or arm deviates from said space area during picking, no collision should occur. when from a specific point of view

Obstruction of a target fruit by an obstacle between the fruit and the camera while observing can be detected by several means including, for example, stereo matching.

[0170] Another important innovation is the picking head, which can pull the target fruit away from the plant to more reliably determine the suitability of the fruit for picking before it is permanently cut from the plant. See Appendix A for the new picking head design.

6. Quality Control Subsystem

[0172] The main function of the quality control subsystem is to assign quality standards to individual picked fruit (or possibly individual picked fruit bunches for picking bunches of fruit). Depending on the type of fruit being picked and the intended customer, quality is a function of several properties of the fruit such as ripeness, colour, firmness, symmetry, size, stem length. Picked fruit may be assigned a grade classification reflecting its quality, such as Grade 1 (symmetrical) or Grade 2 (showing prominent surface creases) or Grade 3 (very deformed or unsuitable for sale). Fruit that is too low quality for retail sale can be discarded and stored separately for other applications, such as jam making. An important implementation challenge is ensuring that QC steps can be performed quickly to maximize the picking robot's productivity. A secondary function of the QC subsystem is to determine a more accurate estimate of fruit size and shape. The estimates can be used for several purposes, such as

For quality grading, since any asymmetry in the 3D shape of the fruit may be considered a reason for assigning a lower quality grade;

As a means of estimating fruit quality, thereby ensuring that the required fruit quality of each fruit basket is determined according to the requirements of the target customer, to determine the average or minimum quality of each fruit basket;

[0175] To facilitate a more precise placement of the fruit in the storage container, thereby minimizing the risk of bruising due to collisions.

[0176] The QC subsystem generates quality measurements for each picked fruit through a computer vision component that includes some cameras, some lights, and some software for image capture and analysis. Typically, the camera is arranged to obtain images of the entire surface of the picked fruit which has been suitably positioned, for example by a picking arm. For example, for a fruit like a strawberry, which may be held so as to hang vertically downward from its stalk, one camera may be positioned under the fruit, while several other cameras may be positioned radially around the vertical axis looking inward. However, one limitation of the described scheme is the large volume required to accommodate the camera (allowing for camera-object distance, maximum size of fruit, and tolerances inherent in fruit positioning). One solution might be to rotate the fruit in front of individual cameras to obtain multiple views - however, any undamped motion of the fruit after rotation could complicate imaging. Therefore, another useful innovation is the use of mirrors positioned and oriented in order to provide multiple virtual views of the fruit to individual cameras mounted under the fruit. The described approach greatly reduces the cost and size of the QC subsystem. The cameras and/or mirrors are usually arranged such that the fruit appears in all views against a common background in order to demarcate the fruit in the image.

[0177] Another useful innovation is to acquire multiple images under different lighting conditions. This could be achieved, for example, by arranging a series of LED lights in a circle around the fruit and activating them one at a time, with each light capturing one exposure. This innovation greatly increases the information content of the image, as the directional light creates shadows on the surface of the fruit and on the background screen suitable for positioning. Such shading can be used to gain more information about the 3D shape of the fruit, such as the location of any surface folds that can reduce marketability.

[0178] Using these images, image analysis software can measure the 3D shape of the fruit and detect various defects (eg, rot, bird damage, spray residue, bruises, mold, etc.). A useful first step is the semantic labeling step, which is used to segment the fruit from the background and generate per-pixel labels corresponding to fruit parts (eg calyx, body, achene, etc.). In the same way as the computer vision subsystem (which takes rough 3D geometry measurements before picking), the QC subsystem can generate models by tweaking

parameters to recover 3D geometry to maximize the consistency between model and image data. Likewise, statistical priors can be used to obtain maximum likelihood estimates of the shape parameter values. A useful innovation is to use estimates of fruit mass density to determine weight estimates from volume estimates. In this way we avoid the need to add the additional complexity of a mechanical weighing device.

[0179] Most aspects of quality judgments are somewhat subjective. While human experts can reasonably score picked fruit, they can have difficulty articulating exactly which factors produce a particular quality label. Thus, a useful innovation is to use quality label data provided by human experts to train a machine learning system to automatically assign quality labels to freshly picked fruit. This can be achieved by training an image classifier with training data consisting of (i) images of picked fruit obtained by QC hardware and (ii) associated quality labels provided by human experts. Various models can be used to map image data to quality labels, for example using handcrafted simple linear classifiers appropriate to the type of fruit in question. For example, in the case of strawberries, appropriate features might aim to capture information about geometric symmetry, seed density (which can indicate the drying of the fruit), ripeness, and surface folding. With enough training data, it is also possible to use convolutional neural networks to learn the mapping directly from images to quality labels.

[0180] FIG. 9 shows a cross-sectional view of an exemplary embodiment of a QC imaging chamber. Cameras (and possibly other sensors, such as cameras sensitive to specific (and possibly invisible) parts of the EM spectrum including IR, (ii) cameras and illuminators using polarized light, and (iii) specific chemical compounds Sensors can be emitted by the fruit.) positioned around the walls of the cylindrical imaging chamber, providing views of every part of the picked fruit surface. The picked fruit is grasped by a suitable end effector (shown here is the hook design described in Appendix A) and lowered into the chamber by a picking arm (of which only the head is shown). A useful innovation is to create a "slit" to reduce the amount of ambient light entering the QC unit - this is a small cylinder that sits on top of the aperture of the imaging chamber and blocks unwanted light from the sides. One difficulty associated with imaging fruit inside a QC imaging chamber is that fruit pieces can collect inside the chamber. A valuable innovation is therefore to equip the chamber with a base that can be pulled out and wiped clean after a period of use.

7. Storage Subsystem

[0182] The purpose of the storage subsystem is to store the picked fruit for transport by the robot until it can be unloaded for subsequent distribution. Because some types of fruit can be damaged by repeated handling, it is often necessary to pack the fruit in a retail-friendly manner immediately after picking. For example, in the case of fruits such as strawberries or raspberries, the picked fruit is usually transferred directly into fruit baskets, which are then transported to retailers. Typically, fruit baskets are stored in trays, each tray has 10 fruit baskets arranged in a 2 X 5 grid. When all baskets in each tray are filled, the tray can be removed from the robot and replaced with an empty tray.

[0183] Since oscillations caused by the movement of the robot on the ground can leave some fruits vulnerable, a useful innovation is to mount the trays via a suspension system (active or passive) in order to minimize the acceleration of the robot while moving over rough terrain .

[0184] Unloading an entire tray of picked fruit may require the robot to travel to the end of the row - so it would be advantageous for the robot to accommodate more trays to amortize the time cost of traveling to and from the end of the row rather than the picked fruit. However, it is an advantage that the robot is also small, so that it can be easily manipulated and stored. A useful innovation, therefore, is to equip the robot with tray supports that extend outward at each end, but can be detached or rotated (up or down) to reduce the robot's length.

[0185] Another useful innovation is to store the pallets in two vertically oriented stacks inside the robot body, as shown in FIG. 3 . One stack contains trays with not yet filled fruit baskets, the other stack contains trays with full fruit baskets. The picking arm transfers the picked fruit directly to the topmost tray. Once the fruit baskets in the tray are filled, the entire tray stack drops one tray deep

degrees to accommodate a new tray, which slides horizontally from top to top of the stack of trays to be filled. A stack of full pallets and a stack of not yet filled pallets rise one pallet depth to bring a new not yet filled pallet within reach of the robot arm. This design allows the robot to compactly store multiple pallets within a limited footprint-important for robots that must traverse narrow crop rows or be transported on transport vehicles.

[0186] Refrigeration of picked fruit shortly after picking can greatly extend shelf life. One advantage of the compact arrangement of the trays described above is that the entire tray can be stored in a refrigerated enclosure. In practice, however, the power requirements of the refrigeration unit on the robot may be greater than a convenient portable energy source (such as a rechargeable battery) can easily meet. Therefore, another useful innovation is to use one of various remote power transfer devices for fruit picking robots. One possibility is to use live overhead lines or railways, such as passenger trains. Another method is to use a power cable that connects one end to the robot and the other end to a fixed power point. As the robot travels along the crop row, the power cables can be stored in coils that are automatically wound and unwound, and such coils can be stored on drums located inside the robot or at the end of the crop row. As an alternative to delivering power directly to the robot, a coolant liquid could be circulated between the robot and the static refrigeration unit through flexible tubes. In this case, the robot can remove heat from the storage container using an internal heat exchanger.

[0187] FIG. 10 shows a diagram illustrating a space-saving solution for storing fruit basket trays within a picking robot. Trays of full fruit baskets are stored in one stack (shown on the right) and trays of empty fruit baskets are stored in another stack (shown on the left). The picking arm can place the picked fruit in the topmost tray. Once the uppermost tray on the right-hand side is full (A), the stack of full fruit basket trays descends downwards, the trays slide sideways (B), and the stack of empty trays rises (C).

[0188] Tray removal/replacement can be accomplished by a human operator or by automated means. A useful innovation draws the operator's attention to the need to change pallets by combining strobe lights on the robot itself with corresponding visual signals in the management user interface. Flashing lights that flash specific colors or have specific flash patterns can be beneficial in allowing operators to correlate visual signals in the UI with specific robots requiring pallet changes or other intervention. Another useful innovation is the idea of using a small, fast-moving robot working in tandem with a larger, slower picking robot. Small robots can automatically remove trays (or full fruit baskets) from picking robots and quickly send them to refrigeration units, where they can be chilled for subsequent distribution.

[0189] For certain types of fruit, the customer (supermarket, etc.) typically defines the size and mass of fruit per basket (or fruit in a basket). Typical requirements include:

i. The total weight of each complete fruit basket is within some allowable tolerance of the nominal value;

ii. Less than a portion of the fruit in each basket differs in size by more than a threshold percentage of the mean; and

iii. Less than a certain percentage of fruit in each fruit basket exhibits unusual shapes or blemishes.

[0193] Depending on the contract between the grower and the customer, pee that does not meet these requirements (or, for example, a tray containing one or more fruit baskets that do not meet these requirements) may be rejected by the customer, reducing the grower's profit. Business requirements can be modeled by a cost function that is a monotonically decreasing function of the grower's expected profit by providing fruit baskets or trays to customers. For example, a simple short network cost function may depend on a linear combination of the following factors: [0194] cost=w₀e+W].u+w₂.c+w₃.d

Wherein e represents that the weight of strawberry in the fruit basket exceeds the target weight, and u is an indicator variable, if the weight of the fruit basket is insufficient, it is 1, otherwise it is 0, and c is the measurement size range of the required strawberry quantity. Finally, d is a measure of how long it takes to place the strawberries in a particular basket, a result of how far the arm has to travel to reach the basket. The weights Wj reflect the relative importance of these factors to profitability, e.g. w1 reflects the cost of rejected pallets containing underweight fruit baskets, weighted for the risk that additional underweight fruit baskets would result in rejected pallets; similarly, w₃reflects more time spent placing strawberries in more

impact on overall machine productivity.

[0196] An interesting observation is that distributing the exact same picked fruit differently between the two networks may result in different total costs according to the above cost function. For example, more baskets can be filled with the same amount of fruit because meeting basket weights or other packaging requirements more precisely means less margin for error is required, or because placing Similar sized fruit reduces the likelihood of customers rejecting the tray. A useful innovation is therefore a strategy for automatically distributing picked fruit to multiple fruit baskets (or disposal containers) based on size and quality measures to minimize the statistical expectation of total cost, i.e., maximize expected profitability, based on the measures described earlier Ability for Growers. A software system can simultaneously maintain a more accurate and complete record of the contents of many suspects than a human picker. Thus, the robot can place the picked fruit into any of a number of partially filled fruit baskets (or discard fruit of inappropriate size or quality). However, this task is challenging because:

There may only be a limited number of partially filled fruit baskets;

- As the fruit basket fills up, the amount of space available for adding fruit decreases;

It is not advisable to move the picked fruit from basket to basket as it is time consuming and may damage the fruit; and

The size and quality of the fruit not yet picked is usually not a priori, so it is necessary to optimize the possible sequence of fruit picked and the associated quality and size classifications.

[0201] In a simple implementation of the above concept, each successive picked fruit can be placed to maximize incremental cost reduction according to the previously described cost measure. However, this greedy local optimization method does not yield the globally optimal fruit distribution. More sophisticated implementations work by optimizing the expected future cost of the stream of unpicked strawberries. While Whist may not be able to predict the size or quality of strawberries not yet picked, the statistical distribution of these attributes can be modeled. This means that global optimization of fruit placement can be achieved by Monte Carlo simulations or similar. For example, given (i) the known prior placement of strawberries in fruit baskets and (ii) the expectation of many samples of strawberries to be picked in the future, each fruit can be placed to minimize the total cost. A probability distribution (Gaussian, histogram, etc.) describing the size of the picked fruit and possibly other quality measures can be dynamically updated as the fruit is picked.

Note that the above cost function (w₃The final term in .d) can be used to ensure that the robot tends to place larger strawberries in farther baskets. Since baskets containing larger strawberries require fewer strawberries, this innovation minimizes time-consuming arm movements to distant baskets.

[0203] Sometimes the storage subsystem is unable to fit the picked fruit into any of the available fruit baskets without increasing the expected cost (i.e. reducing the expected profitability), e.g. because strawberries are too large to fit in any available space , or because its mass or size cannot be determined with high statistical confidence. So another useful innovation would be to have the robot place the fruit into individual storage containers for later detailed inspection and repackaging by human operators.

[0204] For bunch-picked fruit containing multiple individual fruits in the same branching structure, such as table grapes or vines, it may be important that no individual fruit is damaged or otherwise stained, for example because individual rotting fruit can Shorten the life or spoil the appearance of the whole fruit bunch. Therefore, a valuable innovation is a two-stage picking procedure, in which first the whole fruit bunch is picked, and second unsuitable individual fruit bunches are removed from it. In one implementation, this works as follows:

1. The first robotic arm picks the fruit bunch by cutting off the stalk.

2. Perform a visual inspection of the bunches to determine the location of any blemishes of individual fruit. During the visual inspection, the first robotic arm continues to hold the fruit bunch for visual inspection.

3. The second robotic arm trims off defective fruit in the bunch. Similar to that used for individual picking of individual fruits such as strawberries

The picking head can also be used to trim individual fruit from a bunch.

[0208] In another embodiment of this concept, the first robotic arm may transfer the fruit bunch to a static support for subsequent inspection and removal of unwanted individual fruits. In this way, only individual robot arms can be used.

[0209] In addition to deciding which fruit basket (or other container) should be placed to pick fruit, it may also be necessary or beneficial for the robot to determine where to place fruit in the target fruit basket. A key challenge is positioning the picked fruit to minimize bruising (or other types of damage) due to collisions with the basket's floor walls or other fruit already in the basket. In the context of fruit picking systems that work by grasping the stem of the fruit, another challenge is determining how high the fruit should be released into the fruit basket - too high and the fruit could be injured on impact, too low and the fruit could get caught in the basket. between the holder and the bottom of the fruit basket. Also, the picked fruit does not necessarily hang vertically, as the stems are not straight and somewhat stiff. Therefore, a useful innovation is to measure the vertical displacement between the base of the fruit and its grasping point, or the posture of picking the fruit, in the end-effector coordinate system so that the fruit can be released at an optimal height. This can be achieved by using a monocular or stereo camera to determine the position of the bottom of the picked fruit relative to the (assumed known) position of the jig. A related innovation is to use images of the fruit basket (obtained by a camera in the picking head or otherwise) to determine the location of other picked fruit already in the fruit basket. The position or release height can then be changed accordingly. Another useful innovation for fruits such as strawberries, which can be effectively held by their stems, is to orient the clamp so that the stems are held horizontally prior to releasing the fruit into the storage container. This allows the compliance of the stem segment between the gripper and the body of the fruit to cushion the landing of the fruit when it is placed in the container.

[0210] A related innovation is the automatic positioning and orientation of fruit to maximize visual appeal. This can be achieved, for example, by placing fruit with a consistent orientation.

[0211] Because the robot knows which fruit is placed in each basket, it can record the quality of the baskets. Therefore, a useful innovation is to mark each fruit basket with a barcode, which can be read by the robot and thus used to correlate the record of the specific returned fruit contained in the fruit basket.

8. Mapping Subsystem

[0213] The human supervisor uses the administrative user interface to indicate on the map where the robot should choose (see below). A prerequisite is a georeferenced 2D map of the environment, which defines (i) the area where the robot can freely choose any path (needed to traverse the terrain and avoid collisions with other robots) and (ii) the path that the robot must roughly follow, e.g. between rows of plants. Robots can pick from plants that are irregularly or regularly distributed, for example in rows.

[0214] A suitable map can be constructed by a human supervisor using the mapping subsystem. To facilitate map creation, the mapping subsystem allows human operators to easily define piecewise linear paths and polygonal areas. This is achieved in several ways:

- Use of geo-referenced aerial imagery and image annotation software. The UI allows the user to annotate the aerial imagery with the positions of the vertices of the polygonal area and the sequence of positions defining the path, for example through a series of mouse clicks. When annotating the start and end of crop rows, use an integer-based row indexing scheme to facilitate logical correspondence between start and end locations. [0216] Using a physical measuring device, its position can be accurately determined, for example by differential GPS. Users define area boundaries by manually positioning measurement tools, such as a series of points along a path, or at the vertices of a polygonal area. Simple UI devices such as buttons allow the user to initiate and terminate the definition of regions. Survey fixtures can be used to define (i) the physical locations of waypoints along a shared path, (ii) the vertices of a polygonal area where the robot can pick any path, & a) where a row of crops begins and ends. The survey equipment may be a device designed for hand-held use or a robotic vehicle capable of moving under radio remote control.

[0217] In agriculture, an important concern is that heavy robots can damage soft ground if too many robots take the same route on it (or if the same robot travels the same route too many times). Therefore, a useful innovation is to choose paths within the free area to distribute the routes as much as possible on the ground. Tunable parameters allow trade-offs between travel time and distance and degree of spread.

9. Management subsystem

[0219] The management subsystem (including its constituent management user interface) has several important functions:

- It allows a human supervisor to use a 2D map (created previously using the mapping subsystem) to define which crop rows should be picked.

It allows human supervisors to set operational parameter values used during picking and QC, such as target ranges for fruit size and ripeness, quality measures for deciding whether to discard or keep fruit, how to distribute between fruit baskets Fruit etc. [0222] It facilitates the movement of the robot around the farm.

- It divides the work between one or more robots and a human operator.

- It controls the movement of the robot along each row of strawberries.

- It allows the robot to signal a status or fault condition to a human supervisor.

[0226] It allows a human supervisor to immediately place any or all robots in a powered down state.

- It allows human supervisors to monitor the position and progress of all robots by displaying the positions of the robots on a map.

[0228] If there are multiple robots, they will cooperate to ensure that they can move around the same neighborhood without colliding.

[0229] If the robot is not desired to navigate fully autonomously around the site for safety or other reasons, it may be desirable for the robot to be able to drive temporarily under human remote control. Suitable controls may be provided by a radio remote control handset or a software user interface, eg displayed by a tablet computer.

[0230] In order to avoid the need for a human operator to individually drive multiple robots (e.g. from a storage container to a picking station), a valuable innovation is a means by which a series of robots can automatically follow each other driven under human control. individual "guided" robots. Our idea is that each robot in the chain follows its predecessor at a given approximate distance and takes approximately the same route on the ground.

[0231] A simple implementation of this idea is to use removable mechanical couplings to connect the second robot to the first and each successive robot to its predecessor. Alternatively, control signals for its motors may be obtained using means for measuring the direction and magnitude of the force transmitted by the robot's coupling with its predecessor. For example, a follower robot may always apply power to its wheels or tracks in such a way that it minimizes or otherwise modulates the forces in the mechanical couplings. In this way, all robots in the chain can share responsibility for providing power.

[0232] More sophisticated implementations of this concept eliminate the need for mechanical linkages by using a combination of sensors to allow robots to determine their absolute pose and an estimate of their pose relative to their neighbors in the chain. A communication network (eg, a WiFi network) may be used to allow all robots to share time-stamped pose estimates obtained by individual robots. An important benefit is that the potentially noisy relative and absolute pose estimates obtained by many individual robots can be combined to obtain a joint best estimate of the pose of all robots. In one such embodiment, the robot may be equipped with computer vision cameras designed to detect both absolute poses in the world coordinate system and their poses relative to their neighbors. The key elements of the described design are as follows:

Robots are designed to have visually distinct features with known positions or poses in the standard robot coordinate system

sign. For example, visually distinct markers could be attached to each robot at certain predetermined locations. Markers are generally designed for reliable automatic detection in camera images.

[0234] Attach a camera (or cameras) to each robot at a known pose in the standard robot coordinate system. By detecting the 2D position in its own camera image frame of a visually distinct feature belonging to the second robot, one robot can estimate its pose relative to the pose of the second robot (e.g., via a discrete linear transformation). The use of visually unique markers unique to each robot (such as a barcode or QR code or a unique pattern of flashes produced by flashing lights) provides a method by which a robot can be uniquely identified as following it or being followed by it robot.

[0235] One or more robots in the chain also maintain an estimate of their absolute pose in a suitable world coordinate system. This estimate can be obtained using a combination of information sources, such as differential GPS or computer vision-based simultaneous localization and mapping (SLAM) systems. Absolute position estimates from several (possibly noisy or inaccurate) sources can be fused to give a less noisy and more accurate estimate.

[0236] An inter-robot communication infrastructure, such as a wireless network, allows robots to communicate with each other. In this way, robots can ask other robots about their current pose relative to the robot in front of them. Providing pose information along with timestamps, for example, enables mobile robots to compensate for latency when fusing pose estimates.

[0237] In a robot chain, the absolute and relative position estimates obtained by all robots are fused to obtain a higher quality estimate of all robot poses.

[0238] Each robot uses a PID control system to achieve a desired pose relative to the lead robot's trajectory. Usually, the target position of the control system is obtained by finding the closest point on the trajectory of the guided robot. The previous orientation of the target robot at said point defines the target orientation of subsequent robots. A target speed can be set, for example to maintain a constant separation between all robots.

[0239] When picking, teams of robots may be spread out over large areas. Since the robots are visually similar, this could make it difficult for human supervisors to identify individual robots. To allow human supervisors to correlate robot positions displayed on a 2D map in the software UI with robot locations in the world, a useful innovation is to equip each robot with high-visibility strobe lights that respond to mouse clicks (or Other suitable UI gestures at the locations shown in the UI. Individual robots can be made more uniquely recognizable by using different colored flashes and different temporal lighting sequences. A related innovation is to turn the (possibly colored) light produced by the flashes upwards Leads to the multi-channeled roof where the crops are grown. This helps to identify robots hidden by tall crops or tables that grow certain crops such as strawberries.

[0240] Since an individual human supervisor may be in charge of multiple robots working simultaneously, it is useful if the UI exposes the controls (such as stopping and starting) of each robot in the team. One difficulty, however, is knowing which remote control setup is needed to control which robot. Therefore, where an emergency stop is required, the system is usually designed such that pressing the emergency stop button (eg, on the supervisor's tablet UI) will stop all robots that the supervisor is responsible for. This allows supervisors to determine which robot is which after securing safety.

[0241] While picking, the robot may encounter so-called failure situations that can only be resolved by human intervention. For example, a human might be required to remove and replace a complete fruit-picking tray or to disengage a robot from an obstacle causing its mechanics to get stuck. This requires moving human supervisors from robot to robot, for example by walking. To allow human supervisors to do this efficiently, a useful innovation is to use information about the robot's location and the urgency of their failure conditions (or impending failure situations) to plan the human supervisor's route among them . Route planning algorithms can be used, for example, to minimize the time for human supervisors to move between robots (and thus minimize the number of human supervisors required and their cost). Standard navigation algorithms need to be adapted to account for human operators operating at finite speeds in mobile robots

The fact of the degree of movement.

10. Robot Control Subsystem

10.1. Overview

[0244] While the robot is picking, the robot control subsystem makes continuous decisions about what to do next. An available set of actions could include (i) moving the entire robot forward or backward (e.g., along a row of plants), (ii) moving the picking arm and attached camera to previously unexplored viewpoints in order to detect more candidate fruit, (iii) move the picking arm and an additional camera around the local vicinity of some candidate fruit to improve the estimation of its pose or suitability for picking, and (iv) try to pick the candidate fruit (at a specific hypothesized position/orientation). Each of these actions has some expected costs and benefits. As an example, spending more time looking for fruit in a particular neighborhood increases the chances of picking more fruit (possibly increasing yield), but it just takes or takes more time (possibly reducing productivity). The purpose of a robotic control system is to arrange actions that ideally maximize expected profitability based on desired measurements.

[0245] In a simple embodiment, the robot control subsystem can move the entire robot and the picking head and camera in an alternating sequence of three phases. In the first phase, the arm is moved systematically, for example in a grid pattern, recording the image position of the detected fruit. In the second stage, the picking head and camera are sequentially moved to a location near each expected target fruit, collecting more image data or other information to determine (i) whether to pick the fruit and (ii) from which direction to choose. During the second phase, the system determines how much time to spend gathering more information about the target fruit based on a continuous accurate estimate of the probability of successfully picking the appropriate fruit (ie, probability of picking success). When the estimated probability of successful picking is greater than a certain threshold, picking should be performed. Otherwise, the control system may continue to move the arm until the picking success probability is greater than a threshold or some time limit has expired. In the third stage, the entire robot can move a fixed distance along the row of plants once all detected fruits have been detected or rejected for picking.

10.2. Overall position control

[0247] During picking, the control subsystem uses the general positioning subsystem to move the entire robot through the plant and within ranges appropriate for the fruit being picked, for example along a row of strawberry plants. Typically, the robot moves in a series of steps, pausing after each step to allow picking any fruit within the robot's reach. It is advantageous to use as few steps as possible, for example because the robot needs time to accelerate and decelerate during each step. Furthermore, because the time required to pick an incoming fruit depends on the relative position of the robot to the fruit, it is advantageous to position the robot so that the expected picking time is minimized. Therefore, a valuable innovation is to dynamically choose the step size and direction in order to try to maximize the expected picking efficiency according to a suitable model.

[0248] In a simple implementation of this concept, a computer vision camera can be used to detect target fruit that is nearby but outside the robot's current range. The robot can then either (i) reposition so as to minimize the expected picking time of the detected fruit, or (ii) move a greater distance if no suitable fruit is detected in its original vicinity. Additionally, a statistical model of possible fruit locations can be used to adjust the step size. The parameters of this statistical model can be dynamically improved during picking.

10.3. Robot Arm Path Planning

[0250] The picking arm and additional cameras are moved under the control of the control subsystem to find, locate and pick the target fruit. In order for the arm to move without colliding with itself or other obstacles, a path planning algorithm is used to find a collision-free configuration between the initial configuration (i.e., the vector of joint angles) and a new configuration that achieves the desired target end-effector pose path. Physical simulations based on 3D models of the geometry of the arm and scene can be used to test whether candidate paths will be collision-free. However, because finding collision-free paths at runtime can be time-consuming, one can pass one or more pairs of points in the configuration space

Physically simulating the motion of the robot to pre-identify such paths - thereby defining a graph (or "roadmap" where nodes correspond to configurations (and associated end-effector poses) and edges correspond to valid route. A useful innovation is to choose the cost (or "length") assigned to each edge of the graph to reflect a weighted sum of factors that reflect the overall commercial efficiency of the picking robot. These may include (i) the approximate time required, ( ii) energy costs (important in battery-powered robots), and (iii) the impact of component wear on the expected time to failure of robot components (this affects maintenance intervals, downtime, etc.). Path planning can then be performed as follows:

1. Search the nodes of the graph to find a configuration C0 that can be reached without conflict by linear movement from the initial configuration Ci.

2. Search the nodes of the graph to find the configuration C1 from which the target pose can be reached by simple linear movements without collisions. The configuration corresponding to the target pose can be determined by inverse kinematics, possibly using the configuration associated with each candidate configuration as a starting point for nonlinear optimization.

3. Find the shortest path in the graph between nodes C0 and C1, for example using Djikstra's algorithm or other methods.

One limitation of this approach is that the graph can only be precomputed for known scene geometry - and in principle the scene geometry can change every time the robot moves, for example along a row of crops This has inspired an interesting innovation of establishing a mapping between regions of space ("voxels") and the edges of the path graph graph, which correspond to configuration-space paths, which will cause the robot to interact with The regions intersect. Such a map can be easily constructed during a physical simulation of the robot's motion at each edge of the graph. By using voxels to approximate the real and potentially frequently changing scene geometry, those edges corresponding to paths that would cause the arm to collide with the scene can be quickly eliminated from the graph at runtime. A prior knowledge of the geometry of the growth infrastructure and the pose of the robot relative to it can be used to obtain a voxel-based suitable model of the approximate scene geometry. Alternatively, models can be dynamically formed by various means, such as depth cameras, ultrasound, or stereo vision.

[0255] In the context of fruit picking, certain types of collisions may not be catastrophic, for example, collisions between a slowly moving arm and surrounding leaves. Therefore, another useful innovation is to use path planning algorithms to model obstacles not as solid objects (mimicking e.g. bounding boxes), but as probabilistic models of the scene space occupancy of different types of obstacles with different material properties, e.g. leaves, Watering hose, grow bag. The path planning algorithm can then assign different costs to different types of collisions, such as an infinite cost for collisions with immovable objects and a lower (and possibly velocity-dependent) cost for collisions with leaves. By choosing the path with the lowest expected cost, path planning algorithms can maximize movement efficiency, for example by adjusting the trade-off between movement economy and probabilistic collisions with leaves.

10.4. Learning Control Strategy for Reinforcement Learning

[0257] Given an estimate of the approximate location of the target fruit, the system can gain more information about the target (eg, its shape and size, its suitability for picking, its pose) by obtaining more views from new viewpoints. Information from multiple views can be combined to produce more accurate judgments about the suitability of the fruit for picking and the suitability of specific method vectors. As a simple example, the average color of a target fruit across multiple views can be used to estimate ripeness. As another example, the best viewpoint may be selected by taking the viewpoint corresponding to the maximum confidence estimate of the stem orientation. However, obtaining more target views can be time-consuming. Therefore, it is important to (i) choose the viewpoint that will provide the most useful additional information for the lowest cost, and (ii) decide when to stop exploring more viewpoints and try to select the target or drop the target. For example: [0258] If the target fruit (or its stem) is partially occluded (by leaves, other fruit, etc.), it may be valuable to move in the direction needed to reduce the amount of occlusion. In general, it is desirable to find a viewpoint from which the entire fruit can be seen without occlusion, since such a viewpoint defines a volume of space through the backprojected silhouette in which the picking head (and the picked fruit)

Can move towards the target fruit without colliding with any other obstacles.

[0259] If the target fruit is viewed from a viewpoint whose pose (or the pose of its stalk) is difficult to determine for picking purposes, it may be valuable to move to a viewpoint where its pose is more easily ascertained.

[0260] If the target fruit is located near other target fruits, making it ambiguous which fruit belongs to which stalk, it may be valuable to have another view from the point of view that the stalk can be more easily associated with the target fruit.

[0261] If approaching the fruit to pick it from the current viewpoint would require time-consuming movement of the picking arm (e.g., because some movements require reconfiguring the joint angle of the robot arm), it is desirable to locate the strawberry from the corresponding viewpoint corresponding to a faster movement.

[0262] Multiple views are sometimes required to determine which fruit should be picked. Sometimes, it will be clear, but instead of gaining more views, it's better to move on, for example, leaving the fruit to be picked by human pickers. However, manually designing an effective control program can be very difficult.

[0263] The strategy for doing this is to use reinforcement learning to learn a control policy that decides what to do next. A control strategy maps certain states and current input views to new viewpoints. The state may include observations obtained using previous views and arm configurations, which may affect the cost of subsequent moves.

[0264] In order to train a control policy via reinforcement learning, it is necessary to define a utility function that rewards success (in this case choosing a marketable fruit) and penalizes costs (e.g. time spent, energy expended, etc.). This motivates the idea that control strategies could be trained while the robots are operating in the field, using their onboard QC gear to judge picking success using high-quality picking success information. An interesting innovation is that multiple picking robots can be used and explore the space of available control strategies in row, sharing the results among them (e.g. via a communication network or a central server) so that all robots can learn from the best-known control strategy. benefit from the strategy. However, one limitation of the described method is that it may take a significant amount of time to obtain enough training data for learning an effective control strategy. This yields an important innovation which, for training purposes, acquires images of a scene from a set of viewpoints arranged on a grid in camera pose space. Such a dataset can be captured by driving a picking robot under program control to visit each grid point in turn, acquiring (stereo) images of each grid point. Using a training set obtained in this way, reinforcement learning of the control policy can be achieved by simulating the motion of the robot in the available viewpoints, taking into account the cost associated with each motion. In the simulation, the robot can move between any viewpoints on the grid (under the control of the current control policy), perform image processing for each viewpoint, and decide to select the target fruit along a hypothetical physical path. It is impossible to be sure that picking the target fruit along a particular path will be successful in the physical world. However, for some target fruits, simply identifying the correct stalk in stereoscopic view provides a high probability of success. Therefore, we use the correct recognition of the stalk of a ripe fruit as a representation of success in reinforcement learning. Ground truth peduncle locations can be provided manually for the training set.

[0265] At the heart of reinforcement learning are some methods of evaluating the effectiveness of a particular control policy on a data set. In a nutshell, this is achieved by:

1. Setup cost=0

2. For each target fruit in the data set

3. Start from a randomly selected nearby viewpoint on the grid

4. Update state with current view (including current pose and fitness for picking estimation)

5. Use the current control strategy to map states to actions (in {move, pick, abandon})

6. Switch (action)

[0272] Move:

i. Move picking arm to new viewpoint (determined by current policy)

ii. Increase cost by moving costs (time-consuming, power-consuming functions, etc.)

iii. Go to 3

Picking:

i. Increase the cost by moving the picking arm along the picking trajectory (determined by current pose estimate)

ii. If the selection is successful, reduce the cost by the value of the successful selection

iii. Go to 1 and select the next target

[0280] Waiver:

i. Go to 1 and select the next target

[0282] Using this cost assessment scheme, we can compare the effectiveness of multiple control strategies and choose the best, for example through an exhaustive search of the available strategies.

10.5. Overall robot control

[0284] The reinforcement learning strategy described above involves a control strategy for locating fruit and determining its suitability for picking given an approximate initial estimate of its location. Note, however, that the same approach can also be used to train an overall control policy for the entire robot. This means expanding the space of available actions to include (i) moving the picking arm to a farther viewpoint (to find a rough initial position estimate of the target fruit) and (ii) moving along the crop row by a given amount. Another innovation was to extend the reinforcement learning scheme to include operations performed by human operators, such as manually picking hard-to-reach fruit.

11. Miscellaneous Innovations

1. Since picking robots maintain a continuous estimate of their position in a map coordinate system, they can collect geo-referenced data about the environment. A useful innovation, therefore, would be to have the robot record adverse conditions that might require subsequent human intervention, along with map coordinates and possibly photos of the scene. These conditions may include:

Increased damage to plants or infrastructure (e.g. due to harvesting failure or otherwise);

- Deciding to leave ripe fruit because picking would create too much risk of failure, or because the picking arm cannot access the fruit.

2. A useful and related idea is for the picking robot to store in computer memory the map coordinate frame positions of all detected fruit (whether ripe or unripe). This made several innovations possible:

a. One of the innovations is yield mapping, which enables farmers to detect problems early on such as disease, under-watering or over-watering. Of interest may be, for example, the density of fruit production or the proportion of unripe fruit that subsequently ripens into ripe fruit.

b. Another innovation is yield prediction. To pick ripe fruit in a timely manner, picking robots typically have to traverse each crop row every few days. By acquiring images of ripe and unripe fruit, they can measure the size of individual target fruits as they grow and ripen. Since most of the unripe fruits will ripen in time, this data helps in learning predictive models of future crop yields, e.g. in the next day, week, etc. Appropriate models can map current and historical ripeness and size estimates for individual target fruits as well as weather forecast information (e.g., hours of sunshine, temperature) for crop yield predictions.

c. Another related innovation reduces the time the robot has to spend searching for the target fruit during the picking target detection phase. During target detection, the system determines the approximate location of ripe and unripe target fruit. The system typically finds the target fruit by moving a camera at the end of the picking arm to obtain images from a wide range of viewpoints. However, spending more time looking for target fruit may increase yields, but it will only reduce picking rates. Therefore, a useful innovation is to store the map frame locations of unripe fruit that has been detected but not picked in computer memory, so that the robot can more quickly locate unpicked target fruit in subsequent crop row traversals. In a simple embodiment, previously detected but not yet

The location of the unpicked fruit allows the robot to return directly to the same location in subsequent traversals without spending time searching. In more complex embodiments, previous detections are used to form a probability density estimate that reflects the probability of finding a ripe fruit at a particular location in the map coordinate system space (eg, via other kernel density estimates). This density estimate can be used to help the search algorithm prioritize regions of space where ripe fruits are likely to be found, and prioritize regions where they are not present. For example, a simple search algorithm can obtain views from a denser sample of viewpoints in the region of space where ripe fruit is likely to be found. In a related innovation, yield prediction (as described in (b) above) can be used to account for temporal effects on the ripeness of previously unripe fruit.

3. Picking robots can be equipped to perform multiple functions in addition to picking, including the ability to spray weeds or pests with suitable herbicides and insecticides, or to reposition trusses (i.e., stalk structures) to facilitate fruit harvesting. growing or subsequent picking.

4. Use of repositionable adjustable arms to maximize picking efficiency for specific varieties of strawberries or stages of the growing season. The picking arms can also be positioned asymmetrically to increase the reach of the two cooperating arms (see Figure 5).

[0295] 5. In use, the robot may tend to one side or the other due to ground slope or local non-smoothness of the terrain. This effect can be exacerbated by the use of a suspension system (designed to reduce shock when the robot drives over bumps) due to the compliance of the suspension. An unfortunate consequence of tilting is that the position of the robot's picking arms won't be positioned as the plants were designed to do (they could be closer or farther away, higher or lower). This affects performance because (i) if the chassis rotates, the predefined camera pose chosen in the chassis coordinate system may no longer be optimal; (ii) the chassis-relative model of the environment geometry (used to prevent picking arms and environment ) may also be wrong. An obvious strategy to compensate for lean's effect on the environment geometry model is to expand the environment geometry in 3D to provide some tolerance for error, but this degrades performance by compromising the available working volume of the arm (in narrow crop row, space ) may already be at a premium). A valuable innovation is therefore a system that both measures how much the robot is tilting and compensates for it by adjusting models of the scene geometry and camera viewpoint accordingly. Inclination can be measured directly using an accelerometer (measuring vertical) or indirectly by measuring the position of a part of the robot in a coordinate system based on the crop row (e.g. using a vector cable following the boom or GPS). Tilting can then be allowed by applying an appropriate 3D transformation to the predefined camera pose and environment geometry. Another way to correct for tilt caused by sloping but smooth terrain is to dynamically adjust the lateral position of the robot track in the row so that the picking arm, despite the tilt, is closer to its designed position.

6. If the picked fruit is grasped by its stalk, it may swing like a pendulum after the robot arm is repositioned. This can lead to production costs because for certain manipulations (such as placing fruit into the QC imaging chamber or fruit basket), the posture of the fruit must be carefully controlled to avoid collisions - this may require pausing arm motion after repositioning until such amplitudes. Oscillations are reduced to acceptable levels. Therefore, a valuable innovation is the use of damping devices to reduce the oscillation amplitude as quickly as possible. In one embodiment, damping is achieved passively using soft grippers. In another embodiment, damping is actively achieved by adjusting the velocity of the end effector of the robotic arm so that the oscillations stop as quickly as possible. Estimates of the strawberry's mass and pendulum length can be used to design the deceleration profile (dynamic or otherwise) needed to reduce the amplitude or duration of the oscillations.

7. Many diseases and other defects that affect soft fruits (and reduce their quality grade) affect their visual appearance in images obtained by our QC imaging room. The first image processing step is to segment in each view the body of fruit from the calyx (and the background of the imaging chamber). Typically, this is achieved by labeling pixels using a decision forest. It may also be useful to divide the achenes (seeds) at the same time as described below. Next, we characterize the fruit's appearance using various quality measures (possibly isolated or combined innovations). Some features not mentioned before include:

a. Spatial distribution of achenes (eg strawberries) or drupe fruits (eg raspberries). For healthy fruits, achenes and

The drupelets are usually arranged very regularly, ie the distances between adjacent achenes and drupelets are usually locally similar. However, some diseases and other problems can disrupt this regular arrangement in the developing fruit. Therefore, an innovative concept is to use the measure of the regularity of the spatial arrangement of achenes or drupelets as an indicator of fruit quality. One approach is to first use computer vision to detect (in images obtained by our QC imaging room) the image locations of achenes or drupelets, and then assign a cost at each such point using an energy function that combines the lowest energy Points assigned to regular placement. Achenes can be detected by semantic labeling, for example by using a decision forest classifier to assign a table to each pixel. For example, nucleosomes can be detected by inducing specular reflections on the glossy surface of fruit using a point light source, followed by detection of local maxima in image brightness. Then, the cost sum of the points (or local clustering of points) can be used to represent fruit health.

b. Achene color. In e.g. strawberries, the achenes become redder when the fruit becomes overripe, and thinner when the fruit becomes rotten. Labeled training images of fruit with specific defects can be used to determine a threshold for color acceptability.

c. Color of pulp (excluding achenes). This is a good indicator of immaturity, overripeness and localized bruising. Relabeled training data can be used to determine acceptable thresholds.

d. The 3D shape of the fruit. An explicit 3D model of the fruit can be obtained by model fitting to contour and image intensity information, as described in the existing provisional application. However, by characterizing a 2D shape in each of multiple views, an "implicit" 3D model of the shape can be obtained more simply. An efficient strategy for taking one 2D view at a time by first determining the approximate major axis of the fruit, followed by characterizing the shape in a coordinate system aligned with the major axis. The long-axis position can be estimated, for example, by a line connecting the centroid of the fruit body and the image position of the gripper used to hold the fruit. The shape can be characterized by e.g. measuring the distance from the center of mass of the fruit to the edge of the fruit in each clock face direction between 2 o'clock and 10 o'clock given a 9-element vector (note we ignore the top clock face to avoid the calyx ). We can distinguish good (class 1) shapes from bad (class 2) shapes using a collection of training examples and associated expert-derived ground truth labels. A suitable strategy is k nearest neighbors in 9D shape space. Robustness to long axis localization errors is achieved by generating multiple 9D shape vectors for randomly perturbed versions of the long axes detected in the training images.

8. Typically, fruit picked is considered a Grade 2 (below standard stem) or Grade 3 (rejected) if any of our several independent quality indicators give a Grade 2 or Grade 3 ) (although it is possible to combine defect scores in other ways). Note that an important advantage of using image features specifically designed to detect specific kinds of defects rather than more black-box machine learning methods is that doing so allows us to provide growers with clear and intuitive explanations (or better visual indications) Why specific picked fruit gets a specific quality grade. This allows growers to tune meaningful acceptance thresholds for different types of defects. In practice, it is commercially important that growers make different decisions about the acceptability of different types of defects at different times of the season, taking into account the needs of different customers, and as a function of productivity and demand.

9. Another application of agricultural robots is the targeted use of chemicals such as herbicides and pesticides. By using computer vision to locate specific parts of plants or specific kinds of pathogens (insects, dry rot, wet rot, etc.), it may be advantageous for robots to apply the chemicals only where they are needed (as opposed to spraying Compared to the handling system of the whole crop, cost, pollution, etc. To facilitate this work with a robot that is otherwise used to pick it, it is clearly advantageous that the robot supports interchangeable end effectors, including one that can be used for picking and one that can be used for picking One for spraying. The latter typically requires piping along the length of the robotic arm, which can be difficult. A useful innovation, however, is to have the spraying end effector contain a small fraction of the liquid chemical - thereby avoiding the need for piping routing. Yes. This could be achieved using a cassette system, where a robotic arm would access a station on the chassis to collect chemical cartridges (which might be similar to those used in inkjet printers). Alternatively, the arm could access a cassette in the chassis to collect chemical cartridges. The required liquid chemical is drawn from the box into its reservoir or

The operator expels unused chemicals from its reservoir back into the box. It may be that several different types of chemicals are combined in dynamically programmable combinations for more optimal localized treatment, or that multiple cartridges contain many different chemical combinations.

[0304] Appendix A: Production Picking End Effectors

[0305] This appendix describes several innovative picking head designs.

[0306] Background

[0307] Fruit picking requires an end effector (such as can be fixed to a robotic arm) that is small enough, powerful, selective (so that only fruit that is suitable for sale is picked), and exclusive (so that no other fruit is picked). Picking operations must not damage plants or planting infrastructure such as grow bags. Additional constraints (low power, low cost, lightweight, durable) further limit the design.

[0308] The present invention addresses these issues with a lightweight end effector that enables reliable picking. The compact nature of the design greatly facilitates separating desired fruit from unwanted items.

[0309] Generally, picking of produce involves selecting the fruit to be picked, excluding items not intended to be picked (including immature fruit and growing infrastructure), grasping the fruit or its stem, and separating the fruit from its parent plants. Here, we disclose a number of embodiments of the invention to mechanically select, grab and cut products from host plants.

[0310] Hook Embodiment

[0311] A first embodiment of the invention uses hook selection and exclusion with dynamically programmable traces. Under the control of a picking arm (or other), the hook is mechanically swept (usually dynamically) across a selected volume of space, and any stems within said swept volume are collected into the hook. With the stem thus captured, the hook can be used to pull the target fruit (and potential bite sources like leaves or other fruit) out of the plant so that measurements of picking suitability (including visual, olfactory and tactile measurements) can be made. Such measurements inform the decision to pick or release the target fruit.

[0312] The hook can be made long and narrow (e.g. in the shape of the letter J) to minimize the volume that needs to be swept away when moving it (e.g. along its own long axis) towards the target fruit, thereby minimizing the The size of the gap needed (for example, between leaves, stems, or other fruit) so that the hook can reach the target fruit without bumping into it.

[0313] Note that it may be advantageous to position the long axis of the hook nearly coincident with the optical axis of the picking head camera (or almost halfway between the optical axes of the two eyes of the stereo camera). Doing so simplifies the problem of finding a non-collision path to the target fruit, since any target emerging from the camera's viewpoint (through other fruit or foliage) can generally be safely approached by moving along the line corresponding to the rays between the rays. The optical center and target of the camera. This reasoning eliminates the need to obtain a 3D model of the environment to plan a safe route to the target fruit.

[0314] At this stage, the hook movement can be reversed to release the product without damage, or the gripper and cutting mechanism can be actuated to hold the product and separate it from the stem prior to transport and release.

[0315] More detailed information about the mechanical aspects of the invention is now presented. Referring to Figure 11, the main features of the invention assembled are shown. The main components of the end effector are specifically: (1) hook, (2) gripper fitted inside the hook, (3) lower support and (4) blade. The cross-section of the hook is square and together with the blade creates a scissors cutting action. Screws and other support structures are not shown.

[0316] FIG. 12 shows the hook extended relative to the grabbing/cutting mechanism (one blade omitted prior to picking or picking). Figure 13 shows the hook retracted relative to the clamp/cutter with the gripper fitted on the hook to grasp the plant stem and the cutting mechanism in its post-actuated configuration (blades omitted). Figure 14 shows an exploded view of the main components in the described embodiment, including various parts of the end effector, including (4) the blade above the hook.

[0317] Referring to Figures 12 to 14, the main features are: (item 1) the hook (comprising the elongated portion and the hook, in the embodiment described is

semicircular) which (by spatial movement, Figure 15 item 5) allows selection and exclusion of unwanted items. The inside of the hook is shaped to form one half of the gripper mechanism, and to form one half of the scissors cutting surface. The tip is pointed in the described embodiment, maximizing selectivity and exclusivity and aiding positioning within the gripper/blade mechanism when retracted.

[0318] There is a lower support (Figures 11 to 13, item 3) that constrains the retracted hook, enhancing the clamping action and confining the blade adjacent to the hook cutting surface, improving cutting reliability. In said embodiment, the clamp (2) is shaped to fit into the inner surface of the hook, allowing the mechanism to be maximally compact. The described embodiment uses a flexible spring.

[0319] FIG. 15 is an example of the present invention showing the movement of the hook (5) to achieve capture of plant stems. The movement of the hook (5) captures the stem within the hook. Instead of having a fixed size cutting/clamping mechanism, in the present invention the hook is small (allowing good selection while maximizing rejection of unwanted stems) and the hook movement has variable size (in case of uncertainty in stem position selectivity allowed below).

[0320] Varying the catch (per product item) allows an optimal trade-off between selectivity (desired product) and exclusivity (unwanted product).

[0321] Grabbing and cutting actions are performed with the same movement by the actuating device (cf. FIGS. 12 and 13 ). When the hook is retracted relative to the gripper, the stem is first clamped between the hook and gripper (Figures 11 to 13, items 1 and 2). The clamp includes a spring that allows a range of stem sizes and allows the hook to continue retracting. After grasping is complete, the continued movement of the hook relative to the blade separates the product from the parent plant. Fruit release is achieved by extending the hook again.

[0322] Method of construction: Referring to Figure 11, in this embodiment the hook (1) is made of metal, such as steel. Clamp (2) is made of plastic (probably acetal if integrated flex is required). Blade (4) is made by knife steel, and lower support (3) is plastics.

[0323] Figure 16 shows the plant stem thus captured. Figure 17 shows the product picked and cut from the parental plant (optional operation). Fig. 18 shows a release operation (optional operation).

[0324] FIG. 19 shows the sequence of operations that make up the picking process. These are as follows:

[0325] Inspection products are performed by the control and computer vision subsystems described in the main text of this document.

[0326] Performing plant stem approach, positioning the hook on the plant stem close to the product desired to be picked.

[0327] Execution of selection items using cyclical movements determined based on the location of desired products and undesired products and infrastructure. Figure 15 item 5 shows an example of this movement.

[0328] Execute decision step 1 (as shown in FIG. 16 ) using the selected item captured in the hook. Since the fruit is so detached (but still attached to the parent plant), inspections (vision, smell, touch) are performed to detect the nature (and correctness) of the fruit. The outcome of this step is the decision to pick (production for harvest or disposal) or not to pick (unripe fruit, harvesting infrastructure, no harvest, incorrect harvest). If decision step 2 is reached, perform initial grading and quality control of the item for storage and use.

[0329] Release is performed with the hook catching an unwanted item, the action being the opposite of catching (FIG. 18).

[0330] Harvesting with successful capture of product ready to be harvested. The hook is retracted relative to the gripper/cutter (transitioning from the configuration in Figure 12 to the configuration in Figure 13). This enables a grasping operation followed by a cutting operation, separating the fruit from the parent plant (Figures 16 and 17 show before and after states).

[0331] Decision Step 2 is a secondary sensing operation performed when fruit is picked and transferred to another part of the machine.

The result of this decision is to store or process the fruit. Said decision is performed using a different sensor than the sensor in decision step 1 . This forms a more detailed assessment of the fruit so picked. The outcome of the described steps (together with the information in decision step 1) is a decision to store or process the fruit. In the case of a store, size and quality grades are used to determine where to store.

Stored fruit is moved to a storage area (size/quality class according to decision steps 1 and 2) and

Release by reversal of the picking operation (extending the hook relative to the grabber, releasing the fruit).

[0333] If the picked fruit is moldy or unsuitable for sale, it is discarded. Release operations are carried out as above for stores, except for transfers to disposal areas.

Ring embodiment

[0335] In a second embodiment of the invention, the previously described selection and exclusion phases of the picking sequence are performed by actuating, for example, loops of wire. The ring's diameter, position, and orientation are programmatically controlled and activated, allowing it to:

i. An arbitrarily small amount in proximity to the target fruit;

ii. An increase in diameter that is greater than the estimated diameter of the target fruit as it is parallel to and centered on the main axis of the target fruit and moves in the direction of the engagement between the target fruit and its stalk;

iii. It has an arbitrarily small diameter once it has moved past the junction of the stalk and the fruit of interest.

[0339] In this way, the loop will select the fruit stalk of the target fruit from other objects in the environment (eg, other products, leaves, growing infrastructure, etc.). The stem can then be manipulated so that the target fruit is kept away from other objects in the environment.

[0340] If rejected after decision step 1, the fruit can be released by:

i. increasing the diameter of the ring to at least the estimated diameter of the product of interest; and

[0342] ii. Move the ring parallel and center the main axis of the target created and away from the juncture created by the stem and target.

[0343] Once the product is released, a new picking operation can begin.

[0344] Clamp Embodiment

[0345] In a third embodiment of the invention, the selection, exclusion and grasping phases of the previously described picking sequence are performed by a set of tongs. The position, orientation and posture of the clamp (by which it indicates whether it is open, partially closed or closed) is programmatically controlled and actuated such that

i. They approach the stalk of the target fruit in a closed posture to minimize their swept volume;

ii. when arbitrarily close to the fruit stem, they are actuated to an open position;

iii. When they are adjacent to the fruit stem, they are in a partially closed position for simultaneous selection, exclusion and grasping phases; the clamps can be further actuated to a closed position so that the mounted blade moves perpendicular to the stem , using a scissors-like motion to cut it to the opposite side of the clamp.

[0349] If rejected after decision step 1, the fruit may be released in a third embodiment of the invention by step (iii) in the following paragraphs:

i. Making the jaw open; and

ii. Move the lower jaw along the previous approach vector and in the opposite direction. The product is then released and the picking operation resumes.

[0352] Ring-Clamp Embodiment

[0353] As shown in FIGS. 24 to 27, in a fourth embodiment of the invention, the second and third embodiments are combined such that the ring performs the selection and exclusion steps, and the clamps perform the grabbing and cutting steps.

[0354] FIG. 20 shows the main mechanical components of the ring and clamp assembly (with the clamp actuation mechanism omitted for clarity).

[0355] FIG. 21 shows the ring and clamp assembly with the ring extended as shown in assembly 11.

[0356] FIG. 22 shows an exploded view of the ring and major components of the clamp assembly (ring omitted for clarity).

[0357] FIG. 23 shows components of the ring actuation mechanism. The rings are extended and retracted by rotation of the rollers (16). Both drum and ring (11) are located within housings (12) and (17) to constrain the motion of the ring during actuation.

[0358] FIG. 24 shows the loop/tweezer assembly on the approach carrier towards the target fruit.

[0359] FIG. 25 shows that the ring extends and the assembly moves parallel to the main axis of the target product and in the direction of engagement between the stem and fruit.

[0360] FIG. 26 shows the loop having crossed the juncture of stem and fruit, the product of interest is now selected and decision step 1 ( FIG. 19 ) can be applied.

[0361] FIG. 27 shows the ring retracted to control the position of the target product and the jaws actuated to grasp and cut the fruit's stems in a scissors-like motion.

[0362] More detailed information on the mechanical aspects of the ring-clamp embodiment of the present invention is now presented. Figure 20 shows the main mechanical parts of the ring and clamp assembly, which are: (11) a ring sufficiently flexible in its operating plane - which is the plane perpendicular to the main axis of the target product; (12) housing for the ring, to constrain the degrees of freedom of the ring to its operating plane; (13) the blade is stationary mounted on one side of the clamp; (14) one side of the clamp, which may be elastic, to be able to grasp stems of various diameters; (15) The other side of the jaws, forming the opposite side of the grabbing mechanism and the scissors-like cutting mechanism; (16) The drum mounted with rings, by which the rings are extended or retracted by rotation; (17) The housing for the drum, which The confinement ring moves within the groove of the drum.

[0363] These components (except 11 and 12) are shown more clearly in the exploded view of FIG. 22 .

[0364] The ring itself (11) can be realized in several different ways. Their common feature is that they are flexible in the plane of operation and resistant to plastic deformation. This can be in the form of a single or multiple strands of metal or plastic wire, a series of static links connected such that adjacent links pivot about a common axis, or a series of stationary links overlapping adjacent links and pivoting about flex , connected to each linked contiguous member.

[0365] The housing of the ring (12) is shown as a hollow tube. It is statically mounted on the superstructure of the ring assembly. It has the characteristic that the inner surface has a low coefficient of friction, and its cross-sectional shape matches that of the ring. The casing can be made of plastic, rubber, or metal, and it can be reinforced with metal strands and lined with another material. It may also be in the form of a grove cut within the superstructure of the ring assembly and may have any cross-sectional shape.

[0366] Parts 16 and 17 constitute one embodiment of the actuation of the ring. It consists of a drum (16) rotating inside a housing (17). Figure 18 shows details of the drum and its assembly within the housing. The drum has two grooves on its outer circumference, and the shell has matching grooves. When assembled, the movement of these groove limiting rings is stationary relative to the drum and slides along the inner circumference of the housing. In the described embodiment, the housing is shown mounted perpendicular to the ring's operative plane. However, it can also be mounted in the operating plane of the ring to accommodate embodiments of the ring that are inflexible in other planes (such as a series of pivoting links). The actuation can have other implementations, such as a linear actuation that pulls on the ring to retract it, and is resisted by a spring arm to extend it.

[0367] The clamps of the clamp assembly (14 and 15) can be constructed of acetal for integrated flexures, or of a less flexible material (plastic or metal) and include resilient and pivoting The parts either have a rubberized inner surface to provide compliance. This compliance is a necessary characteristic of a clamp assembly to accommodate the clamping of shanks of different diameters with similar magnitudes of clamping force. The jaws are mounted on the superstructure and they can pivot with 1 degree of freedom in the same plane. They can be actuated in various ways, one embodiment is a Bowden cable, the other is a gear driven by means of a servo motor. The blade (13) is rigidly mounted on the flexible clamp (14), but this compliance is not static on the blade. When the jaws are moved to the closed position, the blades slide over the faces of the opposing jaws (15), creating a scissor-like cutting mechanism between them.

[0368] Miscellaneous Innovations

[0369] A variation of the hook motion (item 5 of FIG. 15) may be used. For example, rotation of the hook about 90 degrees along its main axis allows the hook itself to be roughly parallel to the product stem, allowing greater selectivity between stems that are close to each other. In clamping and cutting to

Before moving, the rotation is reversed.

[0370] To increase picking speed, the end effector holding the picked fruit (or the part of the end effector responsible for holding the fruit) can be disassembled and transferred to another part of the machine, and another copy (or variant) of the end effector Actuators can be used to pick more product. In this way, picking can work side-by-side with storage or processing operations, increasing speed. Additionally, one of the end effectors may be selected based on which one is best suited for the task of picking the particular fruit item to be picked.

[0371] The hook and gripper can capture the product into a smaller grab tray that can be removed from the end effector. This includes the gripper part (item 2 of Figure 11) as well as the hook, a subset of the hook, or an extra part that can be released from the machine (can fit inside the hook).

[0372] The hook can be reconfigured as a double bifurcated hook with a gripper/cutter on each side. This allows the capture hook movement to be either clockwise or counterclockwise (variation of item 5 in Figure 15).

[0373] The blade can be positioned in two positions: above and below the clamp (separate actuation). By twisting the hook approximately 180 degrees prior to operation and using the appropriate blade, the hook motion can again be clockwise or counterclockwise while still allowing the blade to be over the clamp for proper product retention by the rod.

[0374] After picking, the fruit can be placed in an imaging chamber for grading (see Quality Control section above). It is generally desirable that the picking head can be tilted downward while holding the fruit in the imaging chamber (as shown in Figure 9), because otherwise the design of the imaging chamber would be compromised by the need to avoid mechanical interference with the picking head. Thus, a useful innovation for an end effector that retains picked fruit by its stalk is to have the surface in contact with the stalk at an angle from vertical when the coring head is oriented so as not to slope downward. This allows the picked fruit to hang vertically compared to horizontally when the picking head is tilted downward. [0375] The hooks can be reshaped into squares, triangles or other shapes. During the (optional) release phase (Fig. 19, "Release"), the simple "L" shape replaces the hook and allows easier release of fruit or other items.

[0376] The cutter can be replaced with a less sharp blade for a more scissor-like rather than cutting action.

[0377] The gripper can be made of rubber (allowing use without a spring) or other material and positioned above or below the hook, although this is not optimal for compactness.

[0378] The cross-section of the hook can vary. In general, a flat inner surface is preferred to ensure a reliable cutting action.

[0379] A related innovation is a hook device designed to allow product rotation for a more complete inspection when deciding whether to pick or not (Fig. 19). For example, the mechanism of twisting its stem allows imaging of the product's inversion before it is picked.

[0380] If the picking arm collides with an immovable object (such as infrastructure) during picking, it may be able to stop automatically, for example by detecting that its expected position differs from its actual position. However, collisions may still result in damage to the arm or its end effector, or require time-consuming intervention by a human supervisor to safely move the arm away from tangled obstacles. The probability of collisions can be greatly reduced by designing the end effector and its motion path so as to minimize the volume of 3D space swept out when moving towards the target. However, the front of the end effector may still collide with immovable obstacles. In this context, the following innovations significantly help reduce the likelihood of robot or infrastructure damage and the need for human intervention: [0381] End effectors can be designed to deform under compressive forces. If the wire loop is sufficiently deformable, the loop design (above) naturally embodies this idea. If a hook design is used, the hooks can be designed to bend elastically or plastically if sufficient compressive force is applied in the longitudinal direction.

- By moving the end effector roughly in the direction normal to the plane of its smallest cross-sectional area to approach the target fruit, the probability of any obstacle colliding first with the deformable hook rather than with other non-deformable parts of the robot is increased.

[0383] As an alternative to making the end effector deformable, a rigid end effector can be mounted to a spring such that it will move backwards into the picking head if sufficient force is applied. Additionally, a microswitch could be used to detect backward movement of the end effector so that in the event of a collision, the arm could stop moving (or reverse its direction of motion). The length and stiffness of the spring should be calibrated so that the robot arm can come to a harmless (and subsequently reverse) stop before excessive force is applied to the obstacle.

Another useful innovation for the hook embodiment is to retract the hook into its support when not actively picking to reduce the chance of snagging.

[0385] APPENDIX B: ROTARY CABLE MANAGEMENT SYSTEM FOR A ROBOTIC ARM

[0386] In the following, we describe an innovative solution to the problem of running various types of cables through an articulated joint of a robotic arm. Here, a cable should be construed as meaning any flexible object intended to conduct matter or energy along its path, as a means of providing power or transmitting information or moving material. Said definition obviously includes (but is not limited to) cables and wires, optical fibers and pipes.

[0387] An important challenge here is to allow a sufficiently wide range of angular motion at each joint. This is especially important in robots, which don't just repeat pre-programmed paths of motion, but dynamically determine where to move based on observations of the environment. In the former case, the motion path can usually be designed such that the joint never exceeds its limit. But in the latter, the desired motion path may not be predictable in advance. Sometimes it may not be possible to move the robot arm directly from its current configuration to the desired target pose due to insufficient range of motion at one or more joints. When this occurs, more complex "reconfiguration" moves may be required to move the junction in question further from its end point. However, reconfiguration moves can be expensive in power or time, as they may require large movements of the robot at all joints. Increased range of motion at joints reduces the likelihood of reconfiguration moves required for the robot to reach new target poses.

[0388] Design Requirements

The main design requirements of the rotating cable management system:

- Allows joints to achieve large changes in rotation angle.

[0391] Has high reliability (equivalent to low stress and low stress inversion in cable assemblies - which can accelerate fatigue and plastic yield).

Ability to carry complex cables - a self-guiding robot is able to sense its environment and act on that basis, which means that high data rates are required to transmit enough information to act efficiently, fast enough to be effectively utilized time. Typically, high data rates require twisted pair cables or fiber optics, both of which are particularly sensitive to twisting and coiling actions.

[0393] Compact - A self-guided robotic system that interacts with its environment requires a small footprint in order to negotiate flexibly in the environment.

protected by the environment

[0395] Alternative solutions to these requirements are suboptimal and include:

[0396] WiFi (or other wireless communication channel). However, limited network bandwidth can make environments where multiple robots work nearby difficult, and it's not a solution for transferring power or conducting matter.

[0397] External Guide Cable - This requires an 'umbilical' which can be caught and damaged by the environment and severely limits the ability of the robot end effector to move relative to the umbilical.

[0398] Optical data transmission and inductive power transmission - high cost.

[0399] Slip rings (very high cost and size, reliable version).

Description

[0401] FIG. 28 shows different elements of a cable management system. The system comprises a cable housing (a) and a central cable guide (b) twisted relative to the housing (c), allowing a coil of cable to expand and contract like a clock spring when the cable guide is twisted. The system may also be configured to support one or more cable coils (d).

[0402] FIG. 29 shows a diagram of the cable management system in situ within one joint of the arm. One or more cables pass through the articulated joint of the robot arm. The cable guide is designed so that the cables pass through the central conduit to the next stage of the arm. The cables are well supported by the cable guides so only well-defined cable coils move. This is illustrated in Figure 30, a series of drawings showing the cable guide rotating within the cable housing. This arrangement minimizes the cyclical stress on the cable because it never bends in reverse, but only bends slightly more or less to accommodate the twist.

[0403] At one end, torsional motion is limited by tension in the coil. At the other end, the cable unrolls itself to the point where the largest part of the cable begins to rub against the inside of the next coil and bend back (a kind of winch lock), rather than the entire coil continuing to unwind. The design must be arranged with sufficient margin of error in fabrication, assembly, and operation so that these limits are never reached to extend the life of the cable.

[0404] FIG. 31 shows a cross-sectional view of a cable winding. When winding at one end of the travel, the cable is pulled taut and has more windings; on the other hand, the cable is pushed out to the edge of the housing and has fewer windings. At the two extremes (and between them) the curvature variation of the cable remains low, so the strain rate and fatigue seen eg for the copper and plastic parts of the cable is low - giving a long life for large overall displacements.

Concrete innovation:

[0406] Use the arrangement of the central cable guide and the housing to define cable coils that can be twisted relative to each other to accommodate low fatigue of the cables.

[0407]Use this arrangement with wires, cables, optical fibers, fiber optic cables, ducts, ribbon cables, individual cores.

The use of this arrangement is especially suitable for "flat" twisted pair cables, which are ideally sized to be flexible in one direction and self-supporting in the other, thus forming a stable coil

[0409] Multiple coil stacks can be stacked on top of each other so that many cables can be managed with a similar footprint, with cable guides guiding all cables through the central axis.

[0410] The housing can be made easy to open and close for easy construction of the cable. In many approaches, a hinge is included in the housing, and the housing is taken out of many parts and built around the coils.

- Shelves can be added between cables to provide a smooth running surface.

[0412] The cable can be lubricated to reduce wear and friction; add a lubricant, or use an inherently lubricating material in the cable, or add a film with the cable coiled to lubricate the surface.

[0413] An additional element can also be added to the central cable guide or the innermost part of the cable to reduce the angle of contact with the next coil and prevent the capstan from locking, thereby increasing travel and reducing stress.

[0414] Shielded cables are often used on data lines to improve immunity to noise emissions; however, shielded cables are generally less flexible and can be used in fewer configurations than unshielded configurations. By modifying the assembly to perform the shielding function, one can benefit from the flexibility and variety of unshielded cables while maintaining high integrity.

Shielding assemblies include making critical components (housing, cable ducts, shelves) out of metal, conductive filler material (e.g. carbon-loaded plastic) or conductive coating material (e.g. metallized plastic) and grounding them; or Place the entire assembly inside a conductive case. Effective shielding may require ground continuity between the rotating halves of the assembly, which can be achieved in a variety of ways, including coiling the ground conductor in the cable; or conductive flexible sealing elements); and capacitive coupling, by minimizing the gap between elements.

[0416] At low temperatures, cables typically degrade faster as the material approaches its glass transition temperature. Cool operation of the cable management system can be achieved by using the cables themselves as heaters, passing current through them to keep them warm.

[0417] · The temperature can be controlled by pumping hot and cold fluids through the pipes.

[0418] Some or all components may be made transparent for ease of manufacture, inspection and maintenance.

[0419] Appendix C: Functional Summary

[0420] This section summarizes the most important high-level features; implementations of the invention may include any combination of one or more of these high-level features, or any combination of any of these. Note that each feature may thus be a separate invention and may be combined with any one or more other features; however, the actual invention defined in the particulars is defined by the appended claims.

[0421] Advanced features are divided into the following categories:

Robot hardware functions or core robot functions

Operation optimization function

[0424] End effector function

[0425] Computer vision capabilities

[0426] AI/machine learning function

The picking process function

The method or application

[0429] There is inevitably some degree of overlap between these features. Therefore, this method of organizing features is not meant to be a strict division, but only a general high-level guide.

Robot hardware functions or core robot functions

[0431] In this section, we summarize the functions of robot hardware functions or core robot functions. The main feature is a robotic fruit picking system that includes an autonomous robot that includes the following subsystems:

a positioning subsystem operable to achieve autonomous positioning of the robot using a computer-implemented guidance system, such as a computer vision guidance system;

at least one picking arm;

at least one picking head or other type of end effector mounted on each picking arm to cut the stem or branch of a particular fruit or bunch or to pick said fruit or bunch, and then transferring said fruit or fruit bunch;

A computer vision subsystem for analyzing images of the fruit to be picked or stored;

[0436] A control subsystem that programs or learns a picking strategy with the picking strategy;

a quality control (QC) subsystem for monitoring the quality of picked or pickable fruit and grading the fruit according to size and/or quality; and

[0438] A storage subsystem for receiving picked fruit and storing said fruit in containers for storage or transport, or in fruit baskets for retail sale.

[0439] While the primary application of the system described is picking strawberries, raspberries and tomatoes, this method could potentially be repurposed outside of the fruit picking environment. For example, it can be used to pick trash or collect other kinds of items. The system can thus be summarized as follows:

A robotic picking system comprising an autonomous robot comprising the following subsystems:

a positioning subsystem operable to achieve autonomous positioning of the robot using a computer-implemented guidance system, such as a computer vision guidance system;

at least one picking arm;

[0443] at least one end effector, or other type of end effector, mounted on each picking arm to pick or collect an item and then transfer said item;

[0444] A computer vision subsystem for analyzing images of items to be picked or stored;

[0445] a control subsystem for programming or learning picking or gathering strategies;

[0446] Quality Control (QC) Subsystem for Monitoring Picked/Collected or Pickable/Collected Items and

[0447] A storage subsystem for receiving picked/collected items and storing said items in containers for storage or transport.

[0448] A number of optional features may be used in such a system, or may constitute independent features, which may be used independently of the system defined above. We list below. Although we specifically mention fruit picking systems, all of the following functions can be used outside of said environment, such as for collecting trash or actually collecting other items; in this appendix C, it is expressly envisioned in everything that follows that goes beyond Outline of fruit.

[0449] A robotic fruit picking system comprising a tracked or wheeled rover or vehicle capable of navigating autonomously using a computer vision based guidance system.

[0450] A robotic fruit picking system, wherein the computer vision subsystem includes at least one 3D stereo camera.

A robotic fruit picking system wherein said computer vision subsystem for analyzing images of fruit includes image processing software for detecting fruit, and said control subsystem includes software for deciding whether to pick said fruit and a user Optimal strategies for picking said fruit based on automatic update strategies, such as reinforcement learning-based strategies, including deep reinforcement learning.

[0452] A robotic fruit picking system wherein the control subsystem automatically learns fruit picking strategies using reinforcement learning.

[0453] A robotic fruit picking system wherein the picking arm has 6 degrees of freedom.

[0454] A robotic fruit picking system wherein said picking arm positions said end effector and camera each mounted on said picking arm.

A robotic fruit picking system, wherein said end effector comprises means for: (i) cutting said stalk or stem; and (ii) grasping said cut stalk or stem to place said fruit Transfer to the QC and storage subsystems.

[0456] A robotic fruit picking system wherein the robot automatically loads and unloads itself onto a storage container or transport vehicle.

[0457] A robotic fruit picking system wherein the robot autonomously navigates in a fruit producing plant, for example along rows of apple or strawberry plants, including table grown strawberry or raspberry plants.

[0458] A robotic fruit picking system that automatically collaborates with other robotic systems and human pickers to efficiently divide the picking work.

[0459] A robotic fruit picking system wherein the system automatically determines the location, orientation and shape of a target fruit.

[0460] A robotic fruit picking system wherein said system automatically determines whether fruit is suitable for picking based on factors automatically updateable in said quality control subsystem.

[0461] A robotic fruit picking system wherein the end effector separates the edible and palatable portion of the ripe fruit from its stem or stalk without contacting the edible portion.

[0462] A robotic fruit picking system in which the system automatically grades fruit based on size and other suitability measurements programmed or learned by the QC subsystem.

[0463] A robotic fruit picking system in which the system automatically transfers the picked fruit to a suitable storage container held in a storage subsystem without handling the edible and palatable parts of the fruit or any parts that may be bruised by handling Other sensitive parts of fruit.

[0464] A robotic fruit picking system in which the control subsystem minimizes the risk of damage to fruit or plants by the end effector or other parts of the robot using machine learning based picking strategies.

[0465] A robotic fruit picking system in which the picking arm moves an additional camera to allow the computer vision subsystem to locate the target fruit and determine its pose and suitability for picking.

[0466] A robotic fruit picking system in which the picking arm is a lightweight robotic arm with at least some joints having a range of motion of +/275 degrees that positions the end effector for picking and moves the picked fruit to the QC subsystem.

[0467] A robotic fruit picking system in which the control subsystem operates the overall positioning system and the picking arm.

[0468] A robotic fruit picking system in which the control subsystem uses input from the computer vision subsystem to analyze images of fruit to decide when and where to move the robot.

[0469] A robotic fruit picking system in which the QC subsystem is responsible for grading picked fruit for retail or other use, and for discarding unusable fruit.

[0470] A robotic fruit picking system in which the robot picks rotten or otherwise unsuitable fruit (by accident or design) and then disposes of the fruit into a suitable container within the robot or on the ground, and the container can be slided through the disposal The chute is close, and the hole for the disposal chute is located at the bottom of the QC unit, so that the arm can immediately drop the fruit without moving to another container.

[0471] A robotic fruit picking system in which positive or negative air pressure is induced in the disposal chute or imaging chamber (e.g. using a fan) to ensure fungal spores from previously discarded fruit are kept away from healthy fruit in the imaging chamber.

[0472] A robotic fruit picking system, wherein the system includes one or more 6-axis lightweight robotic picking arms with some or all joints having a range of motion of +/- 275 degrees.

[0473] A robotic fruit picking system, wherein the system includes two or more picking arms, and the picking arms are positioned asymmetrically on the robot.

[0474] A robotic fruit picking system in which the robot has rails that are movable, and if the rails are removed, the robot can run on the rails.

[0475] A robotic fruit picking system wherein the robot is equipped with a suspended mounted fruit tray.

[0476] A robotic fruit picking system in which the robot is equipped with a fruit holding tray mounted on a movable arm that moves from a first extended position to a second, more compact position.

[0477] A robotic fruit picking system wherein the robot is equipped with fruit holding trays arranged in two or more vertically oriented stacks.

[0478] A robotic fruit picking system wherein the robot is powered by a remote power source.

[0479] A robotic fruit picking system in which the robot has one or more lights (eg strobe lights) that are activated when fruit trays or holders need to be changed.

[0480] A robotic fruit picking system in which a fast moving robot learns from a slower moving machine doing fruit picking

The tray or holder is automatically removed from the robot.

[0481] A robotic fruit picking system in which the robot has one or more lights (eg, strobe lights) that activate in response to user input and illuminate an identification signal above the robot.

[0482] A robotic fruit picking system wherein the system includes an imaging or analysis chamber in which fruit is placed by a picking arm and then imaged or analyzed for grading or quality control purposes.

A robotic fruit picking system, wherein the system includes an imaging or analysis chamber, wherein fruit is imaged or analyzed for grading or quality control purposes, and wherein the imaging or analysis chamber includes a hole and a top slot or cylinder Or cover or baffle The aperture of the imaging or analysis chamber is designed to block unwanted light from entering the chamber while still allowing the fruit to lower or enter the chamber.

A robotic fruit picking system, wherein the system includes an imaging or analysis chamber, wherein fruit is imaged or analyzed for grading or quality control purposes, and wherein the imaging or analysis chamber includes one or more cameras and/or other Sensors such as cameras sensitive to specific (possibly invisible) parts of the EM spectrum including IR, (ii) cameras and illuminators using polarized light, and (iii) sensors specific to specific compounds that the fruit may emit. )

A robotic fruit picking system, wherein the system includes a cable management system for cables passing through an articulating joint of a robot, the cable management system including a cable housing and a central wire twisted relative to the housing A cable guide that allows the helix of the cable to expand and contract as the coil or joint rotates.

[0486] The robot may include a picking arm made up of several individual rigid bodies, each rigid body attached to another rigid body at an articulated joint, and there are associated with one or more of each articulated joint Cable housing. The cable guide may be configured such that the cable passes through the central conduit of the cable housing to the next body. The cable management system can be configured to minimize changes in the local curvature of the cables as the articulating joint moves through its full range of motion. The cable can be unshielded, then the housing provides the shielding. The cable can also be used to provide enough heat to reduce cable degradation

[0487] A robotic fruit picking system in which the picking arm is adjustable and repositionable to maximize picking efficiency for a particular crop variety or growing system, such as the height of a particular tabletop growing system.

A robotic fruit picking system wherein the system is configured to perform several functions in addition to picking, including the ability to spray weeds or pests with suitable herbicides and insecticides, or for relocation or Trim trusses to promote vigorous fruit growth or subsequent picking.

[0489] A robotic fruit picking system in which the robot estimates its position and orientation relative to a crop row by measuring its positional and/or directional displacement relative to tensioned cables.

[0490] A robotic fruit picking system in which the robot estimates its position and orientation relative to a row of crops by measuring displacement relative to a tensioned cable ("vector cable") extending along the row.

[0491] A robotic fruit picking system, wherein the robot includes one or more follower arms mounted to follow the robot.

[0492] A robotic fruit picking system in which a follower arm is attached at one end to a robot chassis by an articulated joint and at the other end to a truck running along a cable.

[0493] A robotic fruit picking system in which the angle at an articulation joint is measured to determine displacement relative to a cable.

[0494] A robotic fruit picking system in which the angle is measured from the resistance of a potentiometer.

[0495] A robotic fruit picking system in which two follower arms are used to determine displacement and orientation relative to a vector cable.

[0496] A robotic fruit picking system in which a computer vision guidance system measures displacement of the robot relative to a vector cable.

[0497] A robotic fruit picking system in which the computer vision system uses the known position and orientation in the robot's coordinate system to measure the projected position of the cable in the 2D image obtained by the mounted camera.

[0498] A robotic fruit picking system in which the stand allows the vector cable to be attached to the legs of a table where the crops are grown.

[0499] A robotic fruit picking system in which the truck is equipped with a microswitch positioned to lose contact with the cable when the truck disconnects the circuit.

A robotic fruit picking system in which an outer portion of a follower arm is attached to an inner portion of the follower arm using magnetic coupling so that in the event of a failure or other event, the portion of the follower arm can be moved without damage Case separation.

[0501] A robotic fruit picking system in which separation of the outer and inner portions of the follower arm triggers the control software to stop the robot.

Operation optimization function

[0503] In this section, we summarize the functions that contribute to the operational efficiency of the system.

[0504] A robotic fruit picking system in which a picking arm is controlled to optimize the trade-off between picking speed and picking accuracy.

[0505] A robotic fruit picking system wherein a control subsystem determines the suitability of a particular target fruit or group to pick via a particular approach trajectory by determining a statistical probability that attempts to pick said target will succeed.

[0506] A robotic fruit picking system wherein the estimated probability of picking success is determined from an image of the scene obtained from a viewpoint near a particular target fruit.

[0507] A robotic fruit picking system wherein determining the statistical probability is based on a multivariate statistical model, such as a Monte Carlo simulation.

[0508] A robotic fruit picking system in which a statistical model is trained and updated from picking success data obtained by a working robot.

[0509] A robotic fruit picking system in which a control subsystem uses an implicit 3D model of a scene formed by a range of viewpoints from which a target fruit can be viewed without occlusion to determine the probability of collision between a picking arm and an object.

[0510] A robotic fruit picking system wherein a control subsystem determines one or more viewpoints where target fruit appears to be unobstructed, and thus identifies unobstructed regions of space.

A robotic fruit picking system wherein the computer vision subsystem obtains a maximum likelihood estimate of the value of a shape parameter of a target fruit using statistical priors, the system then computes a volume estimate of the target, and based on the estimated target's weight .

[0512] A robotic fruit picking system in which the computer vision subsystem determines the size and shape of the fruit.

A robotic fruit picking system in which a computer vision subsystem determines the size and shape of fruit as a means of estimating fruit quality, thereby ensuring that desired quality of fruits are placed in each fruit basket.

A robotic fruit picking system in which picked fruit is automatically distributed into specific fruit baskets or containers based on measures of size and quality of the picked fruit to minimize overall Statistical expectations of costs.

A robotic fruit picking system describing the probability distribution of picked fruit size and other quality measures

Updates dynamically as fruit is picked.

[0516] A robotic fruit picking system in which the picking arm places larger strawberries in baskets farther from the bottom of the picking arm in order to minimize the number of time-consuming arm movements to distant baskets.

A robotic fruit picking system in which the picking arm places selected fruit in individual storage containers for subsequent detailed inspection and repackaging by a human operator, if the quality control subsystem identifies the selected fruit No need for detailed inspection by a human operator.

[0518] A robotic fruit picking system in which a control subsystem applies a two-stage picking procedure to a fruit bunch, wherein the whole bunch is first picked and then unsuitable individual fruits are removed from it.

[0519] A robotic fruit picking system in which the robot measures the pose of the picked fruit so that the fruit can be positioned at an optimal pose for imaging or analysis or released at an optimal height to drop into a fruit basket or container.

A robotic fruit picking system in which the robot determines the position of other picked fruit already in the basket or container and changes the release position or height into the basket or container accordingly to add new fruit to the fruit basket or container basket or container.

[0521] A robotic fruit picking system in which the robot automatically positions or positions picked fruit in a fruit basket or other container to maximize visual appeal.

[0522] A robotic fruit picking system in which the robot automatically generates a record of the quality or other attributes of the fruit in a particular fruit basket and adds a machine-readable image to the fruit basket that links back to the record.

[0523] A robotic fruit picking system in which a robot selects paths within free areas of the ground in order to distribute routes across the ground in a manner that optimizes the trade-off between travel time and damage to the ground.

[0524] A robotic fruit picking system in which a series of robots automatically follow individual "lead" robots driven under human control.

[0525] A robotic fruit picking system in which information about the location of several robots and the urgency of any or impending failure conditions affecting one or more robots is used to plan a route among a human supervisor.

[0526] A robotic fruit picking system in which the position of the robot relative to a target fruit is controlled to optimize picking performance, such as minimizing expected picking time.

A robotic fruit picking system in which a graph (or A "roadmap", where nodes correspond to configurations (and associated end-effector poses), and edges correspond to valid routes between configurations.

A robotic fruit picking system in which a robotic arm path plan is established by a mapping between regions of space ("voxels") and edges corresponding to a roadmap graph that configures a path in space, which will result in the robot at a certain Some or all of the regions intersect with said regions during its motion,

[0529] A robotic fruit picking system in which the system records adverse conditions in the environment that may require subsequent human intervention as well as map coordinates.

[0530] A robotic fruit picking system in which the system stores the location of all detected fruit (whether ripe or unripe) in computer memory in order to generate a yield map.

[0531] A robotic fruit picking system whose yield map enables farmers to identify problems such as disease or overwatering or overwatering.

[0532] A robotic fruit picking system wherein the system stores map coordinate frame locations of unripe fruits that have been detected but not picked in computer memory.

[0533] A robotic fruit picking system in which the yield map takes into account the effect on the ripeness of previously unripe fruit.

[0534] A robotic fruit picking system in which the system measures the degree of robot tilt and compensates for the degree of tilt by adjusting the geometry of the scene and a model of the camera viewpoint accordingly.

[0535] A robotic fruit picking system, wherein the system includes an accelerometer.

[0536] A robotic fruit picking system in which tilt is measured directly using an accelerometer, or indirectly by measuring the position of a part of the robot in a coordinate system based on crop rows.

[0537] The robotic fruit picking system is applied to a predefined camera pose and environment geometry, where appropriate 3D-to-3D transformations are required to correct for tilt.

[0538] A robotic fruit picking system in which the lateral position of a robotic track in a row is dynamically adjusted so that despite the tilt, the picking arm is closer to its designed position.

[0539] A robotic fruit picking system wherein damping achieved by employing a soft grip reduces fruit oscillations caused by picking fruit.

[0540] A robotic fruit picking system in which damping by adjusting the acceleration or velocity or motion of an end effector of a robotic arm reduces fruit oscillations caused by fruit picking.

[0541] A robotic fruit picking system in which the system estimates fruit mass and pendulum length.

[0542] A robotic fruit picking system in which the system designs deceleration or acceleration profiles (dynamic or otherwise) to minimize the amplitude or duration of oscillations.

[00543] End Effector Functions

[0544] In this section, we summarize the characteristics associated with end-effectors; we refer to end-effectors as "end-effectors".

A robotic fruit picking system wherein the end effector employs at least the following stages:

(i) a selection phase during which the fruit of interest is physically separated or separated from the plant or tree and/or other fruit growing on the plant/tree and/or planting infrastructure.

(ii) Cutting phase during which the fruit of interest is permanently severed from the plant/tree.

[0548] A robotic fruit picking system wherein the end effector comprises a hook.

[0549] A robotic fruit picking system in which fruit is removed from its original growth location during a selection phase.

[0550] Robotic fruit picking system where a decision phase is introduced after the selection phase and before the cut phase.

[0551] A robotic fruit picking system wherein the decision phase includes rotating the fruit by its stem or otherwise.

[0552] A robotic fruit picking system wherein the decision phase is used to determine whether to cut fruit, or the manner in which to cut fruit.

[0553] A robotic fruit picking system wherein the selection phase is reversible.

[0554] A robotic fruit picking system in which reversibility is achieved by changing the shape of the hook.

[0555] A robotic fruit picking system in which reversibility is achieved by movement or rotation of the hook.

[0556] A robotic fruit picking system wherein the system is capable of simultaneously gripping and cutting the stalk of a target fruit.

[0557] A robotic fruit picking system, wherein the system includes a plurality of picking units located on individual multiplexed end effectors.

A robotic fruit picking system in which multiple picking functions on a picking unit are controlled by a single actuator or motor

Driven, selectively engaged by lightweight components such as electromagnets, engaging pins, rotating vanes or similar.

[0559] A robotic fruit picking system in which a single motor or actuator drives a function of all units on the head that are selectively engaged by, for example, electromagnets, engaging pins, rotating fins, or similar members.

[0560] A robotic fruit picking system whose functions are driven by lightweight components from elsewhere in the system, for example using: Bowden cables, torsional drive cables/springs, pneumatic or hydraulic components.

[0561] A robotic fruit picking system in which an end effector pulls a target fruit from a plant to determine the suitability of the fruit for picking before it is permanently severed from the fruit.

[0562] A robotic fruit picking system wherein the end effector comprises a hook with a dynamically programmable trajectory.

A robotic fruit picking system wherein the end effector uses at least the following stages:

(i) a selection phase during which the fruit of interest is physically separated or separated from the tree and/or other fruit growing on the tree and/or planting infrastructure;

(ii) a cut-off phase during which the target fruit is permanently cut from the tree; and

[0566] wherein the selection and severing phases are performed by actuating a ring whose diameter, position and orientation are programmatically controlled.

A robotic fruit picking system wherein the end effector uses at least the following stages:

(i) a selection phase during which the fruit of interest is physically separated or separated from the tree and/or other fruit growing on the tree and/or planting infrastructure;

(ii) a cut-off phase during which the target fruit is permanently cut from the tree; and

[0570] and wherein the selection and severing phases are performed by a set of clamps, wherein the diameter and position of the clamps are programmatically controlled.

[0571] A robotic fruit picking system in which the jaw pose, for example open, partially closed or closed, is programmatically controlled.

[00572] Computer Vision Functionality

[0573] In this section, we summarize features related to the computer vision system for autonomous navigation and the computer vision subsystem for fruit imaging.

A robotic fruit picking system in which the computer vision-based system is used to determine the robot's travel relative to a row of crops using images obtained from a camera pointed forward or backward generally along a row of crops orientation and landscape position.

[0575] A robotic fruit picking system, wherein a computer vision based subsystem detects target fruit, and wherein the robot includes an end effector, wherein a portion of the end effector is used as an exposure control target.

[0576] A robotic fruit picking system in which control system software uses lighting conditions inferred or derived from weather forecasts as inputs to a control subsystem or computer vision subsystem to control picking strategies or operations.

[0577] A robotic fruit picking system in which a computer vision based subsystem detects target fruit and an end effector is able to physically separate candidate fruit from plants and other fruit in the cluster prior to picking.

[0578] A robotic fruit picking system in which mirrors are positioned and oriented to provide multiple virtual views of fruit.

[0579] A robotic fruit picking system in which a computer vision based system obtains multiple images of a target fruit under different lighting conditions and infers information about the shape of the target fruit.

[0580] A robotic fruit picking system in which a computer vision based subsystem uses image segmentation techniques to provide an indication of fruit health.

A robotic fruit picking system in which a computer vision-based subsystem detects the locations or points of fruit achenes or drupelets and uses an energy function that assigns energy functions to regularly arranged locations to provide a basis for these locations or placement of these locations Allocate costs.

A robotic fruit picking system in which achenes are detected using a semantic labeling approach, such as a decision forest classifier.

[0583] A robotic fruit picking system where the sum of cost over points provides an indication of fruit health.

[0584] A robotic fruit picking system wherein an indication of fruit health is provided by analyzing one or more of the following: the color of the achenes, the color of the fruit, or the 3D shape of the fruit.

[0585] A robotic fruit picking system using a neural network or other machine learning system trained from a database of existing images with relevant expert-derived ground truth labels.

[0586] A robotic fruit picking system wherein a computer vision based subsystem is used to sort fruit and wherein the system allows a grower to adjust thresholds used to sort fruit.

[0587] A robotic fruit picking system in which a computer vision based subsystem locates specific parts of plants or specific plants that require targeted topical application of chemicals such as herbicides or pesticides.

[0588] A robotic fruit picking system in which a computer vision based subsystem detects instances of specific classes of pathogens, eg: insects, dry rot, wet rot.

[0589] A robotic fruit picking system in which a computer vision subsystem detects drupelets or achenes using specular reflections induced by a single point light source on a fruit surface.

[0590] A robotic fruit picking system wherein one end effector is used for picking and the other end effector is used for spraying.

[0591] A robotic fruit picking system wherein the end effector is a spray end effector containing a small liquid chemical reservoir.

[0592] A robotic fruit picking system in which the picking arm visits a station on the chassis to collect chemical cartridges.

[0593] A robotic fruit picking system in which the picking arm accesses a box in the chassis to draw required liquid chemicals from the box into its reservoir or expel unused chemicals from its reservoir back into the box.

[0594] A robotic fruit picking system in which several different types of chemicals are combined in dynamically programmable combinations for more optimal local processing.

[0595] A robotic fruit picking system wherein multiple boxes contain multiple different chemical combinations.

[0596] AI/Machine Learning Functionality

[0597] In this section, we summarize functions related to AI or machine learning functions.

[0598] A robotic fruit picking system in which machine learning methods are used to train a detection algorithm to automatically detect target fruits.

[0599] A robotic fruit picking system in which the system identifies fruit in an RGB color image obtained by a camera.

[0600] A robotic fruit picking system wherein the system identifies fruit in depth images obtained by dense stereo or otherwise.

A robotic fruit picking system, wherein said training data is a data set in which the position and orientation of target fruits are annotated by hand in plant images representing those plants likely to be acquired by said camera.

[0602] A robotic fruit picking system in which a detection algorithm is trained to perform semantic segmentation on images captured by a camera.

A robotic fruit picking system in which semantic segmentation labels each image pixel, e.g. ripe fruit, unripe

Ripe fruit or other objects.

[0604] A robotic fruit picking system in which a clustering algorithm aggregates semantically segmented results.

A robotic fruit picking system wherein the machine learning method is a decision forest classifier.

A robotic fruit picking system wherein the machine learning method is a convolutional neural network.

[0607] A robotic fruit picking system in which a convolutional neural network is trained to distinguish image patches containing a target fruit at their centers from image patches not containing the target fruit.

[0608] A robotic fruit picking system in which a sliding window method is used to determine the locations of all images that may contain a target fruit.

[0609] A robotic fruit picking system in which semantic segmentation is used to identify possible image locations of target fruit for subsequent more accurate classification or pose determination via a CNN or other form of inference engine.

[0610] A robotic fruit picking system in which a machine learning approach with a regression model is used to predict angles describing the orientation of approximately rotationally symmetric fruit from images, including monocular, stereo, and depth images.

[0611] A robotic fruit picking system in which machine learning methods are used to train a detection algorithm to identify and delineate stems in images captured by a camera.

[0612] A robotic fruit picking system in which machine learning methods are used to train a predictive algorithm to predict how much improvement to an initial pose estimate of a target fruit might be revealed by a given additional viewpoint.

[0613] A robotic fruit picking system in which the system predicts which additional information, including which of a set of available viewpoints, is likely to be most valuable, including information most beneficial to overall productivity.

[0614] A robotic fruit picking system wherein the additional information is the location or point of attachment of the stem to the target fruit.

[0615] A robotic fruit picking system where additional information is the knowledge that fruit is visible without occlusion from a particular viewpoint.

[0616] A robotic fruit picking system where additional information is the knowledge that the space between the camera and the fruit is free of obstacles from a particular viewpoint.

[0617] A robotic fruit picking system, wherein the system recovers the 3D shape of the target fruit from one or more images of the target fruit obtained from one or more viewpoints, and wherein a generative model of the image appearance of the target fruit is used.

[0618] A robotic fruit picking system in which geometric and/or photometric model fitting methods are used to predict the surface appearance of a target fruit and the shadows the target fruit casts on itself under different controlled lighting conditions.

[0619] A robotic fruit picking system in which the cost function, i.e., the measure of coherence between images, is robust to occlusion, or the fruit is physically separated from the source of occlusion.

[0620] A robotic fruit picking system where machine learning methods are used to train a labeling algorithm to automatically assign labels to images captured by the system, where pre-labeled images provided by human experts are used to train the system.

A robotic fruit picking system in which labeled data provided by human experts is used to train a machine learning system to automatically assign quality labels to freshly picked fruit by training an image classifier with training data Includes (i) images of picked fruits obtained by a QC subsystem and (ii) associated quality labels provided by human experts.

[0622] A robotic fruit picking system in which the control policy subsystem is trained by reinforcement learning while the robot is running.

[0623] A robotic fruit picking system in which the control subsystem is trained by reinforcement learning and in which the training is done by simulating the motion of the robot using images of the real world environment captured in available viewpoints.

[0624] A robotic fruit picking system, wherein the control system is trained to predict picking success through reinforcement learning, and wherein the training is performed in a simulated picking environment.

A robotic fruit picking system, wherein the system is trained to predict picking success through reinforcement learning, wherein the predictor of picking success is to ensure that the predicted path of the end effector sweeps past fruit stalks containing the target and not other The 3D volume of the fruit stem.

[0626] A robotic fruit picking system in which the control subsystem is trained by reinforcement learning, including actions performed by a human operator.

[0627] A robotic fruit picking system wherein machine learning methods are used to train a model to predict yield predictions.

[0628] A robotic fruit picking system where the system records map coordinate system locations along with images of all detected fruits and the recorded data is used to train a model to estimate crop yield forecasts.

[0629] A robotic fruit picking system wherein the system uses picking success data obtained by a working robot to learn and improve parameters of a dynamically updatable statistical model for estimating probability of picking success.

[0630] Picking flow function

[0631] In this section, we summarize the functionality related to the picking process used by the system.

A robotic fruit picking system, wherein the system is operable to cut the stem of a fruit, wherein the fruit is picked by first severing and grasping its stem, and in a subsequent operation removing the body of the fruit from its stem remove.

[0633] A robotic fruit picking system, wherein the system is operable to use a jet of compressed air to sever fruit from its stem without the need to handle the body of the fruit.

[0634] A robotic fruit picking system, wherein the robot includes a collar shaped to facilitate forcing the body of the fruit away from its stem.

[0635] A robotic fruit picking system, wherein the system is operable to separate the body of the fruit from its stem by exploiting the inertia of the body of the target fruit to sever the stem from its fruit.

A robotic fruit picking system, wherein the system is operable to move fruit or its The fruit stalk is used to cut or cut off the stem of the fruit.

[0637] A robotic fruit picking system in which a path planning algorithm is used to model obstacles as probabilistic models of the scene space occupancy of different types of obstacles with different material properties.

A robotic fruit picking system, wherein the end effector is operable to cut the stem of fruit, wherein the system comprises a deformable end effector designed to operate under a compressive force down deformation.

[0639] A robotic fruit picking system, wherein the system is operable to cut a stem of fruit without manipulating the body of the fruit.

[0640] A robotic fruit picking system wherein the robot is operable at night with a computer vision system that operates at night and picks fruit when it is colder and thus stronger to minimize bruising.

[0641] A robotic fruit picking system wherein the end effector is operable to cleanly cut fruit stems without tearing to increase fruit yield.

[0642] A robotic fruit picking system wherein the quality control subsystem predicts the flavor or quality of the fruit based on the flavor or quality prediction and places the fruit in a specific storage container.

[0643] A robotic fruit picking system in which the prediction of the flavor or quality of a fruit is dependent on the analysis of growth trajectory data of the fruit measured over time.

Method or application

[0645] In this section, we summarize the functionality related to the method or application of the system.

A method of optimizing fruit yield prediction by imaging each fruit to determine ripeness or suitability for picking using the robotic fruit picking system defined above.

A method of optimizing fruit yield mapping for a fruit farm or farms by imaging each fruit to determine ripeness or suitability for picking using the robotic fruit picking system defined above.

A method of maximizing fruit shelf life by using the robotic fruit picking system defined above.

A method of selectively storing or basketing fruit with the best flavor or quality by using the robotic fruit picking system defined above.

[0650] The final aspect is fruit when picked using the robotic fruit picking system defined above. The fruit may be strawberries, including table grown strawberries. The fruit may be raspberries. The fruit may be apples, pears or peaches, or grapes, plums, cherries, or olives or tomatoes.

Note

[0652] It should be understood that the above-described arrangements are merely illustrative of the application of the principles of the invention. Numerous modifications and alternative arrangements can be devised without departing from the spirit and scope of the invention. While the invention has been illustrated in the drawings and fully described above, it will be apparent to those skilled in the art that the particularity and detail of what is presently believed to be the most practical and preferred example of the invention . Many modified techniques are possible without departing from the principles and concepts of the invention set forth herein.

CN 110139552 Β