Mobile human interface robot

The present invention relates to a mobile human interface robot.

[Explanation of related applications] The present application as a US patent application is US Patent Provisional Application No. 61 / 346,612 filed on May 20, 2010, US Patent Provisional Application No. 61 / 356,910 filed on June 21, 2010, US Patent Provisional Application No. 61 / 428,717 filed on December 30, 2010, US Patent Provisional Application No. 61 / 428,734 filed on December 30, 2010, filed on December 30, 2010. US Patent Provisional Application No. 61 / 428,759, US Patent Provisional Application No. 61 / 429,863 filed on January 5, 2011, US Patent Provisional Application No. 61 / 445,408 filed on February 22, 2011, Provisions of 35 U.SC § 119 (e) regarding US Patent Provisional Application No. 61 / 445,473 filed on February 22, 2011 and US Patent Provisional Application No. 61 / 478,849 filed on April 25, 2011. 35 U.SC §120 on US Patent Application No. 13 / 032,312 filed on February 22, 2011 and US Patent Application No. 13 / 032,228 filed on February 22, 2011. It is a claim application.

A robot is generally an electromechanical device guided by a computer or electronic programming. Mobile robots have the ability to move around in their environment and are not locked into one physical location. An example of a mobile robot commonly used today is an automatic guided vehicle or an automatic guided vehicle (AGV). AGVs are generally mobile robots that use a vision system or laser for tracing or navigating floor markers or wires. Mobile robots are found in industrial, military or security environments. These mobile robots also appear as consumer products for entertainment or for performing certain tasks such as vacuum cleaning and home assistance.

One aspect of the present invention is a mobile robot, the drive system, the controller in communication with the drive system, and the drive system at a height of about 1 foot (30.5 cm) or more above the ground. Provided is a mobile robot having a volume point cloud imaging device supported upward and directed so that a point cloud can be obtained from a spatial volume including a floor plane in the moving direction of the mobile robot. The controller receives a point group signal from the imaging device and, at least in part, sends a drive command based on the received point group signal to the drive system.

The embodiment of the present invention preferably includes one or more of the following features. In some embodiment, the controller is<u style="single">1000MIPS (million instruction (s) per second)</u>It consists of a computer capable of processing instructions exceeding. The imaging device emits light into the scene around the robot and captures an image of the scene along the driving direction of the robot, which are (a) three-dimensional depth image, (b) active lighting image and (c). ) Includes at least one of the ambient lighting images. The controller locates an object in the scene based on the image, and sends a drive command to the drive system based on the location of the object to steer the robot in the scene. In some embodiments, the imaging device determines the time-of-flight between the emission of light and the reception of reflected light from the scene, and the controller determines the distance to the reflecting surface of the object. Use the Time of Flight for this. In an additional embodiment, the imaging apparatus has a light source that emits light into the scene and an imager that receives the reflected light of the emitted light from the scene. The imager has an array of photodetector pixels. For example, a light source emits an optical pulse at a first power saving frequency and, upon receiving a sensor event, emits an optical pulse at a second active frequency. The sensor event should include a sensor signal that represents the presence of an object in the scene.

In some embodiment, the imaging apparatus includes a first part and a second part. The first part is configured to substantially emit light to the ground and receive reflected light from the emitted light from the ground. The second part is configured to emit light into a scene substantially above the ground and receive reflected light from the scene around the robot.

The imaging apparatus preferably has a speckle emitter that emits a speckle pattern of light into the scene along the driving direction of the robot, and an imager that receives the reflected light of the speckle pattern from an object in the scene. The controller stores a reference image of the speckle pattern reflected from the reference object in the scene. Reference images should be captured at different distances from the reference object. The controller compares at least one target image of the speckle pattern reflected from the target object in the scene with the reference image, thereby determining the distance of the reflecting surface of the target object. The controller preferably obtains the primary speckle pattern on the target object, and computer-calculates at least one of the cross-correlation and the non-correlation between the primary speckle pattern and the speckle pattern of the reference image.

Another aspect of the present invention provides a mobile robot that has a base and a holonomic drive system supported by the base. The drive system has first, second and third drive wheels, each located trilaterally spaced around the vertical central axis. Each drive wheel has a drive direction perpendicular to the radial axis with respect to the vertical center axis. The nonholonomic drive system steers the robot on the work surface of the scene. The robot further includes a controller in communication with the drive system, legs extending upward from the base and having variable heights, and a torso supported by the legs. The torso features shoulders with a bottom overhanging the base. The imaging sensor is mounted on the bottom of the fuselage with the imaging sensor facing downward along the front drive direction of the drive system. The imaging sensor captures a three-dimensional image of the scene around the robot.

In some embodiment, the imaging apparatus has a speckle emitter that emits a speckle pattern of light into the scene and an imager that receives the reflected light of the speckle pattern from an object in the scene. The controller stores a reference image of the speckle pattern reflected from the reference object in the scene. Reference images should be captured at different distances from the reference object. The controller compares at least one target image of the speckle pattern reflected from the target object in the scene with the reference image, thereby determining the distance of the reflecting surface of the target object.

The imaging sensor captures an image of the scene along the driving direction of the robot, which is at least one of (a) three-dimensional depth image, (b) active lighting image and (c) ambient lighting image. Including. In some embodiments, the controller locates an object in the scene based on an image comparison and sends a drive command to the drive system based on the location of the object to steer the robot in the scene. The controller preferably obtains the primary speckle pattern on the target object, and computer-calculates at least one of the cross-correlation and the non-correlation between the primary speckle pattern and the speckle pattern of the reference image.

In some embodiment, the imaging sensor is positioned at a height more than 2 feet (61.0 cm) above the work surface and points from the spatial volume, including the floor plane, in the direction of movement of the robot. It consists of a volume point cloud imaging device directed to obtain a swarm. The imaging sensor is preferably attached to the fuselage in a retracted state into the bottom surface of the fuselage so as to visually recognize the working surface in front of the drive wheels of the drive system. In addition, the imaging sensor should have a horizontal field of view of at least 45 ° and a vertical field of view of at least 40 ° and / or preferably in the range of about 1 meter to about 5 meters. The imaging sensor should scan left and right with respect to the front drive direction so as to widen the lateral field of view of the imaging sensor.

The imaging sensor should have a latency of about 44 ms. The imaging output of the imaging sensor should receive a time stamp that compensates for the latency. In some embodiments, the imaging sensor has a serial peripheral interface for communicating with the controller. The imaging sensor should be provided retracted into the body of the torso while maintaining its downward field of view (eg, to minimize catching on an object).

In some embodiment, the robot is placed around the base and provides a sonar detection curtain around the robot to detect objects that invade at least one of the legs and torso. It also has an array of upward-facing sonar proximity sensors. The array of sonar proximity sensors is oriented away from the torso (eg, vertically). The robot may further have a laser scanner in communication with the controller, which is centered in the forward drive direction and has a field of view substantially parallel to the work surface.

Each drive wheel has first and second rows of rollers provided along the perimeter of the drive wheel. Each roller has a rolling direction perpendicular to the rolling direction of the drive wheels. Each roller has an arcuate rolling surface. The rollers together form at least a substantially circular rolling surface of the drive hole.

From yet another perspective, an auto-propelled video conferencing platform for telepresence applications is provided in the drive system chassis that supports the drive system and above the drive system chassis.<u style="single">1000MIPS (million instruction (s) per second)</u>It has a computer capable of processing instructions exceeding, a display supported above the computer, and a camera supported above the computer and capable of moving at least one degree of freedom separately from the display. .. The camera has an objective lens positioned more than 3 feet (91.4 cm) above the ground and less than 10 percent of the display height from the apex of the display area of the display.

In some embodiment, the camera includes a volume point cloud imaging device positioned so that the point cloud can be obtained from a spatial volume adjacent to the robot, and the volume point cloud imaging device is about about above the ground. It is positioned at a height of more than 1 foot (30.5 cm) above and is directed so that the point cloud can be obtained from the spatial volume including the floor plane in the direction of movement of the robot. In addition, the camera should have a volume point cloud imaging device that is directed so that the point cloud can be obtained from a spatial volume located adjacent to the platform. In some embodiments, the display is at least 150 square inches (967.7 cm).²) Has a display area and can move with at least one degree of freedom. The objective lens of the camera preferably has a zoom lens.

A self-propelled video conferencing platform should have a battery configured to power the computer for at least 3 hours. The drive system preferably has an electric omnidirectional drive device. For example, an electric omnidirectional drive may have an electric omnidirectional drive that includes first, second and third drive wheels, each drive wheel trigonally around the vertical center axis. Spacing and supported by the drive system chassis. Each drive wheel has a drive direction perpendicular to the radial axis with respect to the vertical center axis.

The self-propelled video conferencing platform preferably has legs extending upward from the base and having variable height legs and a torso supported by the legs. The torso features shoulders with a bottom overhanging the base. The platform should further have a neck supported by the torso and a head supported by the neck. The neck pans and tilts the head with respect to the vertical center axis. The head supports both the display and the camera. The platform should have a torso imaging sensor that is mounted on the bottom of the torso and points downward along the forward drive direction of the drive system. The torso imaging sensor captures a three-dimensional image of the scene around the robot. In some embodiments, the platform has a head imaging sensor that is mounted on the head and captures a three-dimensional image of the scene around the robot.

One aspect of the present invention provides a method of operating a mobile robot. This method involves receiving three-dimensional depth image data, creating a local perceptual space corresponding to the environment around the robot, and detecting objects located above the ground and below the height of the robot. It has a step of classifying the corresponding part of the local perceptual space as an obstacle. This method classifies the part of the local perceptual space corresponding to the detected object located under the ground as an obstacle and the part of the local perceptual space corresponding to the unobstructed area on the ground as free space. It further has a step of classifying all the remaining unclassified local perceptual spaces as unknown. This method has the steps of executing a drive command to move it to a location within the environmental freedom that corresponds to the local perceptual space classified as at least one of the free space and the unknown.

The embodiment of the present invention preferably includes one or more of the following features. In some embodiment, the classification collapses over time if it is not sustained with updated 3D depth image data. The method further comprises evaluating a predictive robot path corresponding to a robotic drive command that can be executed by rejecting the robot path moving to a location having a corresponding local perceptual space classified as an obstacle or unknown. In some embodiments, the method further comprises the steps of creating a 3D voxel grid using 3D depth image data and the step of converting the 3D voxel grid into a 2D grid. Each cell of the two-dimensional lattice corresponds to a part of the local perceptual space. In an additional embodiment, this method has the steps of creating a grid corresponding to the local perceptual space. Each lattice cell has a corresponding classification of local perceptual space. For each grid cell classified as an obstacle or unknown, the method includes a step of searching for grid points within this grid cell and a step of performing a collision assessment. The collision evaluation may include a step of rejecting grid points located within the collision circle around the location of the robot. As a variant or additional example, the collision assessment may include a step of rejecting grid points located within a collision triangle centered on and / or where the robot is located.

In some embodiment, the method involves directing the field of view of an imaging sensor that provides 3D depth image data towards a region in the environment that corresponds to a local perceptual space classified as unknown. This method preferably includes a step of rejecting a drive command to move the robot to the robot position beyond the field of view of the imaging sensor that provides the three-dimensional depth image data. Further, this method has a step of rejecting a drive command that moves holonomically perpendicular to the forward drive direction of the robot when the robot is stationary for a threshold period. The imaging sensor that provides the 3D depth image data aligns with the front drive direction. In some embodiments, the method comprises accepting a drive command that moves holonomically perpendicular to the robot's forward drive direction while the robot is driving forward. Imaging sensors that provide 3D depth image data are aligned with the front drive direction and have a viewing angle of at least 45 °. This method confirms by the robot that the field of view of the imaging sensor that provides 3D depth image data extends to the location before the robot reaches a location in the environment that has a corresponding local perceptual space classified as unknown. Then, it is preferable to have a step of accepting a drive command to move to a place in the environment having a corresponding local perceptual space classified as unknown.

The three-dimensional depth image data is provided by a volume point cloud imaging device, which is attached to the robot and can obtain a point cloud from a spatial volume located adjacent to the robot, in some embodiment. For example, the 3D depth image data can be positioned at a height of 2 feet (61.0 cm) or more above the ground, and a point cloud can be obtained from the spatial volume including the floor plane in the direction of movement of the robot. It should be obtained by a volume point cloud imaging device that is directed as much as possible.

Another aspect of the invention provides a method of activating a mobile robot to follow a person. This method involves receiving 3D image data from a volume point cloud imaging device positioned so that a point cloud can be obtained from a spatial volume located adjacent to the robot, and the received 3D image data as an object. A person who corresponds to at least a portion of the filtered object, the step of segmenting, the step of filtering the object to remove objects larger than the first threshold size and smaller than the second threshold size. It has a step of identifying the robot and a step of moving at least a part of the robot with respect to the identified person.

In some embodiment, the 3D image data includes a 2D array of pixels. Each pixel contains depth information. This method preferably has steps to group pixels into objects based on the proximity of each pixel to adjacent pixels. In some embodiments, the method comprises moving the robot away from the identified person when the identified person is located within the robot's threshold distance. This method preferably has the step of maintaining the field of view of the imaging device on an identified person. Further, the method preferably has a step of driving the robot to maintain a follow-up distance between the robot and the identified person.

In some embodiment, this method sends a waypoint drive command to drive the robot within the follow-up distance of the identified person and / or to drive the robot between the robot and the identified person. It has a step to maintain the follow-up distance. This method preferably maintains the field of view of the imaging device in the person identified by at least one of panning and tilting the imaging device so that the corresponding field of view is directed towards at least substantially the identified person. It is good to have a step to do.

The imaging device is positioned at a height of at least about 1 foot (30.5 cm) above the ground and is directed so that the point cloud can be obtained from the spatial volume including the floor plane in the direction of movement of the robot. It is good to be. The first threshold size should include a height of about 8 feet (243.8 cm) and / or the second threshold size should include a height of about 3 feet (91.4 cm). This method preferably has a step of identifying a large number of people corresponding to the filtered object. It is good to track and reflect the motion trajectory of each identified person using a Kalman filter. The method may include, at least in part, the step of issuing a drive command based on the motion trajectory of at least one identified person.

From yet another point of view, the object detection method for mobile robots is the step of maneuvering the robot from one end of the work surface to the other, the step of emitting light to the scene around the robot, and the scene along the driving direction of the robot. Has a step of capturing the image of the robot. The image includes at least one of (a) three-dimensional depth image, (b) active lighting image and (c) ambient lighting image. This method involves locating an object in the scene based on an image, assigning a confidence level for the object location, and maneuvering the robot in the scene based on the object location and the corresponding confidence level. And further.

In some embodiment, this method involves creating an object occupancy map of the scene. This method preferably has a step of lowering the confidence level of each object location over time and finally updating each object location with a newly defined object location. In some embodiments, the method comprises maneuvering the robot to perform at least one of 1) contact with the object and 2) follow-up along the perimeter of the object. In an additional embodiment, the method comprises maneuvering the robot to avoid objects.

This method preferably has a step of emitting light into the scene in the form of intermittent pulses. For example, it is good to change the frequency of the emitted light pulse. In some embodiment, the method preferably has a step of emitting an optical pulse at a first power saving frequency and a step of emitting an optical pulse at a second active frequency upon receiving a sensor event. The sensor event should include a sensor signal that represents the presence of an object in the scene.

In some embodiment, this method involves the next step: emitting a speckle pattern of light into the scene, receiving reflected light from an object in the scene, and a reference within the scene. To store the reference image of the speckle pattern reflected from the object, to capture at least one target image of the speckle pattern reflected from the target object in the scene, and to determine the distance of the reflective surface of the target object. It has a step of creating a three-dimensional depth image of the scene by comparing at least one target image with a reference image. Reference images should be captured at different distances from the reference object. This method has a step of obtaining a primary speckle pattern on the target object and a step of computer calculation of at least one of the cross-correlation and the non-correlation between the primary speckle pattern and the speckle pattern of the reference image. good.

From another point of view, the object detection method for mobile robots is the step of emitting a speckle pattern of light into the scene around the robot while manipulating the robot from end to end of the work surface, and the target object in the scene. A step of receiving the reflected light of the emission speckle pattern reflected from the surface, a step of finding the distance between each reflecting surface of the target object, a step of creating a three-dimensional depth map of the target object, and a step of classifying the target object. Has.

In some embodiment, the method comprises the step of storing a reference image of a speckle pattern reflected from a reference object in the scene. Reference images should be captured at different distances from the reference object. This method has a step of finding a primary speckle pattern on the target object and a step of computer calculation of at least one of the cross-correlation and the non-correlation between the primary speckle pattern and the speckle pattern of the reference image. good.

In some embodiments, this method of light is performed, for example, by changing the frequency of the emitted light pulse.<u style="single">Speckle</u>It is preferable to have a step of emitting the pattern with intermittent pulses. This method preferably includes a step of capturing a frame of reflected light of an emission speckle pattern reflected from the surface of the target object at a given frame rate. The frame rate should be about 10Hz to about 90Hz. This method preferably has the step of analyzing the differences between speckle patterns captured in consecutive frames for the identification of the target object.

One aspect of the present invention provides a mobile robot system having a mobile robot and a remote computer computing device. The mobile robot is a volume type that is directed so that a point cloud can be obtained from the drive system, the controller that is in communication with the drive system, and the space volume that is in communication with the controller and is located adjacent to the robot. It has a point cloud imaging device. The remote computer computer is in communication with the mobile robot and executes the software program, which displays the 3D scene based on the volume point cloud captured by the imaging device and is within the 3D scene. View the rendered solid model of the robot. The software program receives the robot command and transmits it to the robot controller.

In some embodiment, the volume point cloud imaging device is supported above the drive system at a height of about 1 foot (30.5 cm) or more above the ground and the direction of movement of the mobile robot. It is directed so that a point cloud can be obtained from the spatial volume including the floor plane. The software application accommodates the robot model from a third perspective, preferably from a raised position behind the robot model, for example, the robot's actual orientation and / or attitude to a visible space volume located adjacent to the robot. It is preferable to display the orientation and / or orientation with respect to the displayed three-dimensional scene.

The computer computing device preferably has a touch screen and is configured to recognize touch gestures on the touch screen. For example, the computer computing device may be a tablet computer. The software application should determine the robot command based on the touch gesture received by the touch screen with respect to the robot model. In addition, software applications often seek changes in the position and / or kinematic state of the robot model, and determine robot commands to synchronize the robot with the modified robot model.

In some embodiment, the software application modifies the robot model and / or the view of the 3D scene based on the touch gesture received. Software applications should display directional pads that receive robot commands.

The software application should execute a post-collision detection algorithm to obtain the drive command. In some embodiments, the post-collision detection algorithm projects a drive path into a 3D scene from a first location to a second location until a collision with an object. Software applications should model the ground and point clouds as virtual surfaces of objects that can be collided by robots.

In some embodiment, the imaging device emits light into the scene around the robot and captures an image of the scene along the driving direction of the robot. The image includes at least one of (a) three-dimensional depth image, (b) active lighting image and (c) ambient lighting image. The controller locates the object in the scene based on the image. In some embodiments, the imaging apparatus has a light source that emits light into the scene and an imager that receives the reflected light of the emitted light from the scene. The light source should emit light in intermittent pulses. For example, a light source may emit an optical pulse at a first power saving frequency and, upon receiving a sensor event, emit an optical pulse at a second active frequency. The sensor event optionally includes a sensor signal indicating the presence of an object in the scene. The imager should have an array of photodetector pixels.

In an additional embodiment, the imaging device has a speckle emitter that emits a speckle pattern of light into the scene along the driving direction of the robot and an imager that receives the reflected light of the speckle pattern from an object in the scene. Have. The controller stores a reference image of the speckle pattern reflected from the reference object in the scene. Reference images should be captured at different distances from the reference object. The controller compares at least one target image of the speckle pattern reflected from the target object in the scene with the reference image, thereby determining the distance of the reflecting surface of the target object. The controller preferably obtains the primary speckle pattern on the target object, and computer-calculates at least one of the cross-correlation and the non-correlation between the primary speckle pattern and the speckle pattern of the reference image. In some embodiments, the imaging sensor scans left and right with respect to the front drive direction to widen the lateral field of view of the imaging sensor.

Another aspect of the present invention provides a method of operating a mobile robot. In this method, the step of receiving the 3D depth image data of the volume point group from the spatial volume located adjacent to the robot and the 3D scene are displayed on the remote computer computer based on the received 3D depth image data. It has a step of displaying a robot model corresponding to the robot in a three-dimensional scene, a step of receiving a robot command from a remote computer computing device, and a step of transmitting a robot command from the remote computer computing device to the robot.

In some embodiment, this method displays the orientation and orientation of the robot model with respect to the displayed 3D scene corresponding to the robot's actual orientation and orientation with respect to the visible space volume located adjacent to the robot. Have steps. This method preferably has a step of determining a robot command based on the touch gesture received with respect to the robot model. In some embodiments, the method comprises a step of obtaining a change in the position and / or kinematic state of the robot model and a step of determining a robot command to synchronize the robot with the modified robot model.

This method preferably has a step of changing the view of the robot model and / or the 3D scene based on the touch gesture received. In some embodiments, the method comprises displaying a directional pad that receives a robot command. This method preferably has a step of executing a post-collision detection algorithm for obtaining a drive command. The post-collision detection algorithm projects a drive path into a 3D scene from a first location to a second location until a collision with an object. It is good to model the ground and point clouds as virtual surfaces of objects that can collide with robots.

This method preferably has a step of rejecting a drive command to move the robot to the robot position beyond the field of view of the imaging sensor that provides the volume point cloud. The three-dimensional depth image data is preferably provided by a volume point cloud imaging device that is attached to the robot and can obtain a point cloud from a spatial volume located adjacent to the robot. The volume point cloud imaging device can be positioned at a height of about 1 foot (30.5 cm) or more above the ground and can obtain point clouds from the spatial volume including the floor plane in the direction of movement of the robot. It is good to be sent so that you can.

In some embodiment, this method emits light into the scene around the robot, captures an image of the scene along the driving direction of the robot, and the presence of objects in the scene based on the image. It has a step to locate it. The image includes at least one of (a) three-dimensional depth image, (b) active lighting image and (c) ambient lighting image. This method reduces the confidence level of each object location over time as a step of creating an object occupancy map of the scene and optionally assigning a confidence level for each object location, and finally each object presence. It is good to have a step to update the location with the newly defined object location.

In some embodiment, this method involves the next step: emitting a speckle pattern of light into the scene, receiving reflected light from an object in the scene, and a reference within the scene. It has a step of creating a three-dimensional depth image of a scene by storing a reference image of a speckle pattern reflected from an object. Reference images should be captured at different distances from the reference object. This method captures at least one target image of the speckle pattern reflected from the target object in the scene and compares the at least one target image with the reference image to determine the distance of the reflective surface of the target object. It is better to have more. Further, this method includes a step of obtaining a primary speckle pattern on the target object and a step of computer calculation of at least one of the cross-correlation and the non-correlation between the primary speckle pattern and the speckle pattern of the reference image. Is good.

In some embodiments, the method touches the step of receiving a touch gesture on a remote computer computer and / or during the 3D scene at the intersection with the 3D scene to determine the user's choice regarding the robot model. It has a step of projecting the location of the gesture screen along the frustum projection.

This method involves a step of identifying an unrecognizable mass within a group of volume points and a recognizable mass, such as an image captured by a robot's camera, overlaid on or replaced by an unrecognizable mass within a 3D scene. It is good to have a step to display.

Details of one or more specific examples of the present invention are described in the accompanying drawings and in the description below. Other aspects, other features and other advantages will become apparent from the description, drawings and claims.

<figref num="1">It is a perspective view of an exemplary mobile human interface robot.</figref><figref num="2">It is a schematic diagram of an exemplary mobile human interface robot.</figref><figref num="3">It is a perspective view seen from above of the example mobile human interface robot.</figref><figref num="4A">It is a perspective view seen from the front of the base of an example of a mobile human interface robot.</figref><figref num="4B">It is a perspective view seen from the back of the base shown in FIG. 4A.</figref><figref num="4C">It is a top view of the base shown in FIG. 4A.</figref><figref num="5A">FIG. 6 is a schematic front view of an exemplary base of an exemplary human interface robot.</figref><figref num="5B">FIG. 3 is a schematic plan view of an exemplary base of a mobile human interface robot.</figref><figref num="5C">It is a front view of the example holonomic wheel of a mobile human interface robot.</figref><figref num="5D">It is a side view of the wheel shown in FIG. 5C.</figref><figref num="6">It is a perspective view seen from the front of the body part of the example of a mobile human interface robot.</figref><figref num="7">It is a perspective view from the front of the neck of an example of a mobile human interface robot.</figref><figref num="8A">It is a schematic of the circuit configuration of an example of a mobile human interface robot.</figref><figref num="8B">It is a schematic of the circuit configuration of an example of a mobile human interface robot.</figref><figref num="8C">It is a schematic of the circuit configuration of an example of a mobile human interface robot.</figref><figref num="8D">It is a schematic of the circuit configuration of an example of a mobile human interface robot.</figref><figref num="8E">It is a schematic of the circuit configuration of an example of a mobile human interface robot.</figref><figref num="8F">It is a schematic of the circuit configuration of an example of a mobile human interface robot.</figref><figref num="8G">It is a schematic of the circuit configuration of an example of a mobile human interface robot.</figref><figref num="9">It is a schematic diagram of an exemplary mobile human interface robot.</figref><figref num="10A">FIG. 6 is a perspective view of an exemplary mobile human interface robot with a large number of sensors directed towards the ground.</figref><figref num="10B">FIG. 5 is a perspective view of an exemplary mobile robot having a large number of sensors oriented parallel to the ground.</figref><figref num="11">It is a schematic diagram of an exemplary imaging sensor that detects an object in the scene.</figref><figref num="12">It is a schematic of the example operation method of operating an imaging sensor.</figref><figref num="13">It is a schematic diagram of an exemplary three-dimensional (3-D) speckle camera that detects an object in a scene.</figref><figref num="14">It is a schematic diagram of an example operation method for operating a 3-D speckle camera.</figref><figref num="15">It is a schematic of an exemplary 3-D time-of-flight (TOF) camera that detects an object in the scene.</figref><figref num="16">It is a schematic diagram of an example operation method for operating a 3-D TOF camera.</figref><figref num="17A">It is a schematic diagram of an exemplary occupancy map.</figref><figref num="17B">It is a schematic diagram of a mobile robot having a visual sense of a scene in a work area.</figref><figref num="18A">It is a schematic diagram of an exemplary layout map.</figref><figref num="18B">It is a schematic diagram of an example robot map corresponding to the layout map shown in FIG. 18A.</figref><figref num="18C">It is a figure which shows the example operation method which operates a mobile robot to go through an environment using a layout map and a robot map.</figref><figref num="19A">FIG. 6 is a schematic of an exemplary layout map containing triangles of dense layout points.</figref><figref num="19B">It is a schematic diagram of an example robot map corresponding to the layout map shown in FIG. 19A.</figref><figref num="19C">It is a figure which shows the example operation method for finding the target robot map place using a layout map and a robot map.</figref><figref num="20A">It is a schematic of an exemplary layout map having centroids of dense layout points.</figref><figref num="20B">It is a schematic diagram of an example robot map corresponding to the layout map shown in FIG. 20A.</figref><figref num="20C">It is a figure which shows the example operation method for finding the target robot map place using a layout map and a robot map.</figref><figref num="21A">It is a schematic diagram of an example of the local perceptual space of a mobile human interface robot in a stationary state.</figref><figref num="21B">It is a schematic diagram of an example of the local perceptual space of a mobile human interface robot in a moving state.</figref><figref num="21C">It is a schematic diagram of an example of the local perceptual space of a mobile human interface robot in a stationary state.</figref><figref num="21D">It is a schematic diagram of an example of the local perceptual space of a mobile human interface robot in a moving state.</figref><figref num="21E">Corresponding sensory vision is an illustrative schematic representation of a mobile human interface robot in a state where the field is moving in close proximity around the corner.</figref><figref num="21F">Corresponding sensory vision at the corner<u style="single">Around</u>It is a schematic diagram of an example of a mobile human interface robot in a state of being widely moving.</figref><figref num="22">It is a schematic of an exemplary control system executed by a controller of a mobile human interface robot.</figref><figref num="23A">It is a perspective view of an exemplary mobile human interface robot that keeps the sensor field of view in contact with a person.</figref><figref num="23B">It is a schematic diagram of an example mobile human interface robot performed on a person.</figref><figref num="24A">FIG. 6 is a schematic representation of an exemplary human detection routine for a mobile human interface robot.</figref><figref num="24B">FIG. 6 is a schematic representation of an exemplary human detection routine for a mobile human interface robot.</figref><figref num="24C">FIG. 6 is a schematic representation of an exemplary human tracking routine for a mobile human interface robot.</figref><figref num="25A">It is a schematic diagram of an exemplary mobile human interface robot that follows a person around an obstacle.</figref><figref num="25B">It is a schematic of an example local map of a mobile human interface robot updated at a human location.</figref><figref num="25C">FIG. 6 is a schematic representation of an exemplary local map routine for a mobile human interface robot.</figref><figref num="26A">It is a schematic diagram of an exemplary mobile human interface robot rotating to face a person.</figref><figref num="26B">It is a schematic diagram of an example mobile human interface robot that keeps a follow-up distance from a person.</figref><figref num="26C">It is a schematic diagram of an example mobile human interface robot using a direct speed command for keeping a following distance from a person.</figref><figref num="26D">It is a schematic diagram of an example mobile human interface robot using the upper point command to keep the following distance from a person.</figref><figref num="27">FIG. 6 is a perspective view of an exemplary mobile human interface robot with a removable web pad.</figref><figref num="28A">It is a figure of a state in which a person is interacting with an exemplary mobile human interface robot.</figref><figref num="28B">It is a figure of a state in which a person is interacting with an exemplary mobile human interface robot.</figref><figref num="28C">It is a figure of a state in which a person is interacting with an exemplary mobile human interface robot.</figref><figref num="28D">It is a figure of a state in which a person is interacting with an exemplary mobile human interface robot.</figref><figref num="28E">It is a figure of a state in which a person is interacting with an exemplary mobile human interface robot.</figref><figref num="29">It is a schematic diagram which shows the example telephone system for initiating and carrying out communication with a mobile human interface robot.</figref><figref num="30">FIG. 6 is a schematic representation of an exemplary house having an object detection system.</figref><figref num="31">It is a schematic diagram of an example operation method for operating an object detection system.</figref><figref num="32">It is a schematic diagram of a person wearing a pendant in communication with an object detection system.</figref><figref num="33A">FIG. 6 is a schematic representation of a mobile robot system including a person operating a remote computer computing device in communication with an exemplary robot.</figref><figref num="33B">FIG. 6 is a schematic representation of an exemplary web pad running a software application that displays a layout map.</figref><figref num="33C">It is a schematic of an example web pad running a software application that displays a 3-D map floating on a 2-D layout map.</figref><figref num="33D">FIG. 6 is a schematic representation of an exemplary web pad running a software application that displays a 3-D map and directional pad.</figref><figref num="33E">It is a schematic diagram of a scene view of a software application.</figref><figref num="33F">It is a schematic diagram of a scene view of a software application.</figref><figref num="33G">It is a schematic diagram of a scene view of a software application.</figref><figref num="34">It is a schematic of an exemplary software architecture.</figref><figref num="35">FIG. 6 is a schematic projection of a truncated body from a touch screen of an exemplary web pad running a software application.</figref><figref num="36A">It is a schematic diagram of the virtual eye position (eye position) of the 3-D map displayed on the example web pad.</figref><figref num="36B">It is a schematic diagram of the virtual eye position (eye position) of the 3-D map displayed on the example web pad.</figref><figref num="36C">It is a schematic diagram of the line of sight from the virtual eye position to the target position.</figref><figref num="37A">It is a schematic diagram of the 3-D map displayed on the example web pad, and is the figure which shows the arrangement state of the recognizable image on the unrecognizable mass of a point cloud.</figref><figref num="37B">It is a schematic diagram of the 3-D map displayed on the example web pad, and is the figure which shows the arrangement state of the recognizable image on the unrecognizable mass of a point cloud.</figref><figref num="38">It is a schematic diagram of a remote computer computing device communicating with a robot.</figref>

In various figures, the same reference numerals indicate the same elements.

Mobile robots (also called mobile robots) can interact with or interface with people to provide a number of services ranging from home assistance (from home assistance to commercial assistance). In the home assistance example, mobile robots can assist older people in their daily work, such as medication, mobility assistance, and communication assistance (eg, video conferencing, telecommunications, internet access). Etc.), residential or on-site monitoring (inside and / or outside), human monitoring and / or provision of personal emergency response system (PERS), but not limited to these. In the case of a commercial assistant, the mobile robot can provide video conferencing (eg, in a hospital environment), a point-of-sale terminal (POS terminal), an interactive information / marketing terminal, and the like.

Referring to FIGS. 1 and 2, in some specific examples, the mobile robot 100 has a robot body 110 (or chassis) that determines a forward drive direction F. The robot 100 further includes a drive system 200, an interface module 300, and a sensor system 400, each of which is supported by the robot body 110 and of the robot 100.<u style="single">operation</u>It is in a communication state with the controller 500 that coordinates the movement with. Power supply 10<u style="single">5</u>(Eg, one or more batteries) should be supported in an electrical communication relationship with each of these components as needed, and this power supply will power each of these components as needed. Send to. For example, the controller 500 is 1,000 MIPS (<u style="single">million instruction (s) per second</u>) May include a computer capable of executing a number of instructions, and the power supply 105 provides the computer with sufficient battery power for more than 3 hours.

The robot body 110 has a base 120, a body 140 supported by at least one leg 130 extending upward from the base 120, and at least one leg 130 in the illustrated embodiment. The base 120 can at least partially support the drive system 200. The robot body 110 further has a neck 150 supported by a torso 140. The neck 150 supports the head 160, which supports at least a portion of the interface module 300. The base 120 has a low center of gravity CG of the base 120 to maintain mechanical stability._BAnd the low overall center of gravity CG of Robot 100_RSufficient weight to maintain (eg power supply 105 (battery)<u style="single">)</u>By supporting).

With reference to FIGS. 3 and 4A-4C, in some specific examples, the base 120 defines a ternary symmetric shape (eg, a triangular shape when viewed in plan view). For example, the base 120 may preferably have a base chassis 122, which is a first, second, and third base corresponding to each leg of the three-sided base 120 (see, eg, FIG. 4A). It supports the base body 124 with the body parts 124a, 124b, 124c. Each base body portion 124a, 124b, 124c is preferably movably supported by the base chassis 122 so that it moves independently of the base chassis 122 in response to contact with an object. The trigonal symmetry of the base 120 makes it possible to detect 360 ° bulging bumps around the robot 100. Each base body portion 124a, 124b, 124c preferably has a associated contact sensor (eg, capacitance sensor, reed switch, etc.) that detects the motion of the corresponding base body portion 124a, 124b, 124c with respect to the base chassis 122.

In some embodiment, the drive system 200 allows omnidirectional and / or nonholonomic motion control of the robot 100. As used herein, the term "omnidirectional" means that it is possible to perform substantially any planar direction, i.e., left and right (lateral), back and forth, and rotational movement. These directions are commonly referred to herein as x, y and θz, respectively. In addition, the term "holonomic" is used in a manner that is substantially consistent with the literature use of the term, with three planar degrees of freedom in the plane direction, namely two translations and one rotation. It means that you can do it. Therefore, it has the ability to move in a plane direction at a speed composed of virtually any ratio of three plane velocities (front-back, lateral and rotation) and these ratios in a virtually continuous manner. Has the ability to change.

Robot 100 can work in a human environment (eg, an environment typically designed for bipedal occupants) using a wheeled movement scheme. In some embodiment, the drive system 200 is first, first, equidistant (eg, 120 ° spaced) (ie, ternary symmetrically) about the vertical axis Z. It has 2nd and 3rd drive wheels 210a, 210b, 210c, but other configurations are also available. With reference to FIGS. 5A and 5B, the drive wheels 210a, 210b, 210c are laterally arcuate rolling surfaces (ie, rolling direction D)._R) It is preferable to have a contour shape curved in the lateral direction or the direction perpendicular to the direction, and this rolling surface can assist the maneuverability of the holonomic drive system 200. Each drive wheel 210a, 210b, 210c is coupled to a drive motor 220a, 220b, 220c, respectively, and this drive motor is independent of the other drive motors 220a, 220b, 220c. Can be driven forward (forward) and / or backward (backward). Each drive motor 220a-220c preferably has its own encoder 212 (FIG. 8C), which provides wheel rotation feedback to the controller 500. In some embodiments, each drive wheel 210a, 210b, 210c is attached to or near one of the three vertices of an equilateral triangle and is perpendicular to the bisector of the angle of the end of each triangle. It has a driving direction (front-back direction). By driving the ternary symmetric holonomic base 120 in the front-wheel drive direction F, the robot 100 can transition to a non-front-wheel drive direction for confinement or autonomous escape from the clutter, and then escape. Can be rotated and / or translated to drive along the front drive direction F after is resolved.

With reference to FIGS. 5C and 5D, in some embodiment, each drive wheel 210 has an inner row 232 and an outer row 234 of the rollers 230, each row in the rolling direction of the drive wheels 210. D_RRolling direction D perpendicular to_rHave. The rows 232,234 of the rollers 230 should be staggered (for example, as a result, one roller 230 belonging to the inner row 232 is equal between two adjacent rollers 230 belonging to the outer row 234. Positioned. The roller 230 makes an infinite slip perpendicular to the drive direction of the drive wheels 210. The roller 230 rolls in these rolling directions D._rIt comprises an arc-shaped (eg, convex) outer surface 235 perpendicular to, so that the rollers 230 together define a circular or substantially circular perimeter of the drive wheels 210. The contour shape of the roller 230 affects the overall contour shape of the drive wheels 210. For example, the rollers 230 may include an arc roller outer surface 235 that together constitutes a scalloped rolling surface (eg, as a tread for traction) of the drive wheels 210. However, by configuring the rollers 230 to have a contour shape that constitutes a circular rolling surface as a whole of the drive wheels 210, the robot 100 will not vibrate perpendicular to the wheel tread, but on a flat surface. It can move smoothly. When approaching an object at an angle, the staggered rows 232,234 of rollers 230 (radius r) can be used as a tread to climb an object that is as high as or nearly as high as the wheel radius R of the drive wheels 210.

In the embodiments shown in FIGS. 3-5B, the first drive wheel 210<u style="single">a</u>Is configured as a leading drive wheel along the front drive direction F, with the remaining two drive wheels 210b, 210c following it. In this configuration example, in order to drive forward, the controller 500 has the same second and third drive wheels 210b, 210c while the first drive wheel 210a is slipping along the front drive direction F. It is good to issue a drive command to drive in the forward rolling direction at a speed. Further, this drive wheel configuration allows the robot 100 to stop in a short time (eg, to produce a rapid negative acceleration with respect to the front drive direction F). This is due to the natural dynamic instability of the three-wheeled design. If the front drive direction F is along the bisector of the angle between the two front drive wheels, a short stop will cause the robot 100 to rotate around the two "front" wheels. A torque is generated to overturn this. On the other hand, by naturally traveling one drive wheel 210a forward, the robot 100 is prevented from falling forward even when it is necessary to support or make a sudden stop. However, when accelerating from a stopped state, the controller 500 will take the overall center of gravity CG of the robot 100._RThe moment of inertia I of the robot 100 can be taken into account from.

In another embodiment, the drive system 200 is positioned so that the angular bisector of the angle between the two drive wheels 210a, 210b is aligned with the front drive direction F of the robot 100, the first and second. It is preferable to be arranged so as to have the driving wheels 210a and 210b. In this configuration example, since the controller 500 is driven forward, it is preferable that the controller 500 drives the first and second drive wheels 210a and 210b in the forward rolling direction and at the same speed, and issues a drive command to drive the first and second drive wheels 210a and 210b. , The third drive wheel 210c remains idle. Since it turns left or right while driving forward, the controller 500 has the corresponding first or second drive wheels 210a, 210b.<u style="single">To</u>Relatively<u style="single">Fast speed or slow speed</u>It is good to issue a command to drive with. Drive systems 200 of other configurations can also be used. The drive wheels 210a, 210b, 210c can have a cylindrical, circular, elliptical or polygonal contour shape.

With reference to FIGS. 1 to 3 again, the base 120 supports at least one leg 130 extending upward from the base 120 in the Z direction. The legs 130 are preferably configured to have a variable height for raising and lowering the torso 140 with respect to the base 120. In some embodiment, each leg 130 has first and second leg portions 132,134 that move relative to each other (eg, nested or telescopic, linear and / or angular). Rather than continuously nesting small diameter extrusions into and out of each other or nesting from a relatively large diameter base extrusion, the second leg portion 134 is the first leg in the illustrated embodiment. Nesting along this on portion 132, thus allowing other components to be placed along the second leg portion 134 as well as potentially comparing the base 120 with the second leg portion 134. You can move close to the target. The leg 130 preferably has an actuator assembly 136 (FIG. 8C) that moves the second leg portion 134 relative to the first leg portion 132. The actuator assembly 136 preferably includes a lifting motor 138b and a motor driver 138a in contact with an encoder 138c that provides position feedback to the controller 500.

In general, the nested configuration is the center of gravity CG of the entire leg 130._LHas a continuously small diameter extrusion that nests in and out of a relatively large diameter extrusion at the base 120 to keep it as low as possible. In addition, strong and / or large diameter components may be provided at the bottom to handle the large torque generated at the base 120 when the legs 130 are fully extended. However, this approach raises two issues. First, when a relatively small diameter component is placed at the top of the leg 130, rain, dust or other granules tend to flow or fall along the extrusion, thereby between the extrusions. It enters the space of the above and thus interferes with the fitting of the extruded part. This creates a very difficult sealing problem when trying to still maintain full mobility / joint motion of the leg 130. Second, it may be desirable to attach a payload or ancillary equipment to the robot 100. One common place to attach to ancillary devices is the top of the torso 140. If the second leg portion 134 nests in and out of the first leg portion, the accessories and components are above the entire second leg portion 134 if it needs to move with the torso 140. Can only be attached to. If not, the component attached to the second leg 134 will limit the nested movement of the leg 130.

By nesting the second leg 134 along this on the first leg 130, the second leg 134 can move perpendicular to the base 120 with additional payload attachment points. I will provide a. With this type of configuration, water or air-carried granules do not enter the space between the legs 132,134 and flow along the torso 140 outside all leg 132s, 134 (eg, extrusions). This greatly simplifies the sealing of the joint or joint of the leg 130. In addition, the payload / accessory mounting features and / or second leg portion 134 of the torso 140 are always exposed and accessible no matter how the leg 130 is extended.

With reference to FIGS. 3 and 6, the leg 130 preferably supports the torso 140, which covers the base 120 and has shoulders 142 extending above it. In the illustrated embodiment, the torso 140 has a downward facing bottom surface 44 (eg, closer to the base) forming at least a portion of the shoulder 142 and an opposite upward facing top surface 146 between them. The side 148 extends. The torso 140 has various shapes or geometric shapes, such as a circle or an ellipse, with a central portion 141 supported by the leg 130 and a peripheral free portion 143 extending laterally beyond the lateral extent of the leg 130. It should be in the shape of a shape, thus forming an overhang that constitutes the downward facing surface 144. In some embodiments, the torso 140 has a polygonal or other complex shape that constitutes the shoulder, and the shoulder is an overhang that extends beyond the leg 130 while covering the base 120. There is.

Robot 100 preferably has one or more attachment ports 170 (eg, mechanical and / or electrical interconnect points) that accept the payload. The accessory port 170 is arranged so that the received payload does not obstruct or interfere with the sensor of the sensor system 400 (eg, attached to the bottom 144 and / or top 146 of the fuselage 140). good. In some embodiment, as shown in FIG. 6, the torso 140 is provided in the rear portion 149 of the torso 140, eg, the basket 3<u style="single">6</u>It has one or more accessory ports 170 that accept payloads in zeros so that sensors attached to the front part 147 of the fuselage 140 or other parts of the robot body 110 are not disturbed. It has become.

Referring again to FIGS. 1-3 and 7, the torso 140 supports the neck 150, which pans and engages the head 160 with the torso 140. In the illustrated embodiment, the neck 150 has a rotator 152 and a chiller 154. The rotator 152 has an angular motion range θ of about 90 ° to about 360 °._R(For example, around the Z axis) can be provided. Other ranges can also be adopted. Further, in some embodiments, the rotator 152 maintains a continuous 360 ° of head 160 relative to the torso 140 at an unrestricted rotation speed while maintaining electrical communication between the head 160 and the rest of the robot 100. Has an electrical connector or contact that allows the rotation of the robot. The tilter 154 may have the same or similar electrical connectors or contacts that allow rotation of the head 150 with respect to the torso 140 while maintaining electrical communication between the head 160 and the rest of the robot 100. .. The rotator 152 preferably has a rotator motor 151 coupled to or engaged with a ring 153 (eg, a toothed ring rack). The tilter 154 has its head at an angle θ with respect to the body 140, separately from the rotator 152._TCan be moved (eg, around the Y axis). In some embodiments, the tilta 154 has a tilta motor 155, the tilta motor 155 having the head 160 at an angle θ of ± 90 ° with respect to the Z axis._TMove between. Other ranges, such as ± 45 °, can also be adopted. The robot 100 is preferably configured such that the legs 130, torso 140, neck 150 and head 160 remain within the perimeter of the base 120 in order to maintain stable mobility of the robot 100. In the exemplary schematic shown in FIG. 8F, the neck 150 is the pan / tilt assembly 15.<u style="single">9</u>Has a pan / tilt assembly 15<u style="single">9</u>Includes Rotator 152 and Tilta 154 along with corresponding motor drivers 156a, 156b and encoders 158a, 158b.

8A to 8G are schematic diagrams of an example of the circuit configuration of the robot 100. 8A-8C provide an exemplary schematic of the circuit configuration for the base 120, which is a proximity sensor such as a sonar proximity sensor 410 and a cliff proximity sensor 420, contact sensor 430. , Laser scanner 440, sonar scanner 460 and drive system 200 may be accommodated. The base 120 may further accommodate the controller 500, the power supply 105 and the leg actuator assembly 136. The torso 140 preferably houses a microcontroller 145, a microphone 330, a speaker 340, a scanning 3-D image sensor 410a and a torso touch sensor system 480, and the torso touch sensor system 480 allows the controller 500 to be a user. Can receive and respond to contact or touch (eg, move the body 140 with respect to the base 120, pan and / or tilt the neck 150 and / or command in response. By sending it to the drive system 200). Neck 150 is a pan / tilt assembly 15<u style="single">9</u>Good to accommodate, pan / tilt assembly 15<u style="single">9</u>Is a compatible motor driver 156a and encoder 1<u style="single">5</u>Have 8a<u style="single">Rotator</u>152 and compatible motor driver 156b and encoder 1<u style="single">5</u>Have 8b<u style="single">Cirta</u>It is good to include 154. Head 160 has one or more web pads<u style="single">310</u>And camera<u style="single">320</u>Is good to accommodate.

With reference to FIGS. 1-4C and 9, in order to achieve reliable and robust autonomous motion, the sensor system 400 is sufficient to make intelligent decisions about what the robot 100 takes in the robot's environment. It is good to have several different types of sensors that can be used in relation to each other to give rise to the perception of a robotic environment. The sensor system 400 preferably has one or more types of sensors supported by the robot body 110, such as obstacle detection collision avoidance (ODOA) sensors, communication sensors, navigation sensors, and the like. Can be mentioned. For example, these sensors include proximity sensors, contact sensors, three-dimensional (3-D) imaging / depth map sensors, cameras (eg, visible and / or infrared cameras), sonars, radars, LIDAR (Light Detection And Ranging). (Light detection ranging, which may require an optical remote detection method that measures the characteristics of scattered light to detect a range and / or other information of a distant target), LADAR (Laser Detection) and Ranging), etc., but not limited to these. In some embodiment, the sensor system 400 is a ranged sonar sensor 410 (eg, nine around the base 120), a proximity cliff detector 420, a contact sensor 430, a laser scanner 440, one or two. It is preferable to have the above 3-D imaging / depth sensor 450 and imaging sonar 460.

There are several challenges in placing the sensor on a robot platform. First, the sensors need to be arranged so that they have the maximum coverage of the area of interest around the robot 100. Second, the sensors may need to be placed so that the robot 100 itself produces the absolute minimum degree of occlusion to the sensor, and in essence, the sensors are "these are" by the robot itself. It must not be placed so that it is "blindfolded". Third, the placement and installation of the sensors should not be exposed to the rest of the platform's industrial design. In terms of aesthetics, a robot with a sensor mounted inconspicuously can be considered more "attractive" than a robot without it. In terms of practicality, the sensor should be mounted so that it does not interfere with normal robot operation (does not get caught in obstacles, etc.).

In some embodiment, the sensor system 400 is in communication with the controller 500 and within one or more zones or parts of the robot 100 to detect nearby or invading obstacles. Arranged (eg, base body portion 124a, 1 of robot body 110)<u style="single">24</u>b, 1<u style="single">24</u>It has a set or array of proximity sensors 410,420 located at or near c. Proximity sensors 410,420 are focused infrared (IR) emitter-sensor elements, sonar sensors, ultrasonic sensors and / or imaging sensors (eg, eg) that provide signals to controller 500 when an object is within a given range of robot 100. 3-D depth map image sensor) is good.

In the embodiments shown in FIGS. 4A-4C, the robot 100 is a sonar in an array configured around the base 120 (eg, at substantially equal spacing) and with an upward field of view. It has a shape proximity sensor 410. First, second and third sonar proximity sensors 410a, 410b, 410c are provided at or near the first (front) base body portion 124a and are at least one of the sonar proximity sensors. Is located near the outermost radial edge 125a of the first base body 124a. The fourth, fifth and sixth sonar proximity sensors 410d, 410e, 410f are the second (<u style="single">left</u>Provided at or near the base body portion 124b (on the side), at least one of the sonar proximity sensors is located near the outermost radial edge 125b of the second base body 124b. Seventh, eighth and ninth sonar proximity sensors 410g, 410h, 410i are provided at or near the third (right) base body portion 124c, and at least one of the sonar proximity sensors is It is located near the outermost radial edge 125c of the third base body 124c. This configuration provides at least three detection zones.

In some embodiments, a set of sonar proximity sensors 410 (eg, 410a-410i) provided around the base 120 is arranged to face upwards (eg, substantially in the Z direction) and optionally. , Angled outward from the Z-axis, thus forming a detection curtain 412 around the robot 100. The sonar proximity sensors 410a-410i do not guide sonar emissions (radiated waves) upwards or at least towards other parts of the robot body 110 (eg, do not detect motion of the robot body 110 with respect to itself). It is better to have a shroud or emission guide 414. The emission guide 414 is preferably of a shell or semi-shell type. In the illustrated embodiment, the base 120 extends laterally beyond the leg 130 and the sonar proximity sensors 410 (eg, 410a-410i) are at the base 120 around the leg 130 (eg, substantially the base). (Along the perimeter of 120). In addition, the upward facing sonar proximity sensors 410 are spaced apart from each other to form a continuous or substantially continuous sonar detection curtain 412 around the leg 130. The sonar detection curtain 412 can be used to detect obstacles with elevated lateral protrusions, such as table tops, shelves, and the like.

The upward-viewing sonar proximity sensor 410 mainly provides a function of looking at an object located in a horizontal plane, for example, the upper surface of a table. Due to these aspect ratios, these objects may not be visible to other sensors in other sensor systems, such as the laser scanner 440 or the imaging sensor 450, and may therefore pose problems for the robot 100. .. An upward-viewing sonar proximity sensor 410, provided along the perimeter of the base 120, provides a means of visually recognizing or detecting objects / obstacles of these types. In addition, the sonar proximity sensor 410 is placed around the widest part around the base, tilted slightly outward so that it is not shielded or obstructed by the body 120 or head 160 of the robot 100. Well, thus, the result is no false belief in detecting parts of the Robot 100 itself. In some embodiment, the sonar proximity sensor 410 is provided around the torso 140 outside the field of view of the sonar proximity sensor 410 and thus is attached to the payload or ancillary device, such as the basket 3.<u style="single">6</u>Arranged (upper and outward) to leave behind a volume that can freely accept zeros. The sonar proximity sensor 410 should be retracted into the base body 124 so as not to be visible and to provide an external feature that could get caught or hit an obstacle.

The sensor system 400 serves as a backup for one or more sonar proximity sensors 410 (eg, in the direction opposite to the front drive direction F) to detect obstacles. For example, it is preferable to have a rear proximity sensor 410j). The rear sonar proximity sensor 410j should have an emission guide 414 that directs its sonar detection field 412. In addition, the rear sonar proximity sensor 410j can be used for distance measurement to determine the distance between the robot 100 and a detected object (eg, as a "backup alert") in the field of view of the rear sonar proximity sensor 410j. In some embodiments, the rear sonar proximity sensor 410j is provided retracted into the base 120 so as not to provide visual or functional irregularities of the housing form.

With reference to FIGS. 3 and 4B, in some embodiment, the robot 100 enables drive wheels 210a, 210b, 210b to detect cliffs before the drive wheels 210a, 210b, 210c hit the cliffs (eg, stairs). It has a cliff proximity sensor 420 located near or around the 210c. For example, the cliff proximity sensor 420 may be disposed at or near each of the outermost radial edges 125a-125c of the base body 124a-124c and at locations between them. In some cases, cliff detection was tilted towards each other using infrared (IR) proximity or actual range detection and forming overlapping emission and detection fields and thus detection zones where the floor should be predicted. It is carried out using an infrared emitter 422 and an infrared detector 424. IR proximity detection should have a relatively narrow field of view, may be determined by the surface albedo for reliability, and should have varying range accuracy for each surface. As a result, it is advisable to place a number of separate sensors around the robot 100 to properly detect cliffs from multiple locations on the robot 100. In addition, IR proximity sensors typically cannot distinguish between cliffs and, for example, safety events immediately after the robot 100 has climbed the threshold.

The cliff proximity sensor 420 can detect when the robot 100 encounters a falling edge of the floor, for example, when the robot encounters a series of stairs. The controller 500 (which executes the control system) can execute a behavior that causes the robot 100 to take an action such as a change in the moving direction when the edge is detected. In some embodiment, the sensor system 400 has one or more secondary cliff sensors (eg, other sensors configured to perform cliff detection and optionally other forms of detection). Cliff detection proximity sensor 420 is configured to provide early detection of cliffs and to provide data to distinguish between actual cliffs and safety events (eg, climbing a threshold), and these fields of view are robots. It should be positioned downwards and outwards to include at least a portion of the body 110 and a region located away from the robot body 110. In some embodiment, the controller 500 increases the distance through the edge of the supporting work surface (eg, the floor), the edge of the work surface and / or the distance between the robot body 110 and the work surface. Perform a cliff detection routine to identify and detect. In this embodiment, 1) early detection of potential cliffs (which may allow for rapid movement speeds in unknown environments), 2) controller 500 is a cliff event in the true sense of the word. Increased reliability of autonomous mobility by receiving cliff imaging information from the cliff detection proximity sensor 420 to know if it is unsafe or if it can safely cross the cliff (eg, up the threshold). , 3) Allows reduction of false beliefs in cliffs (eg, due to the use of edge detection for multiple separate IR proximity sensors with a narrow field of view). An additional sensor, placed as a "wheel drop" sensor, is used for the sake of security and to detect situations where the range detection camera is unable to reliably detect certain types of cliffs. it can.

With the detection of thresholds and stairs, the robot 100 can effectively plan to either cross the threshold that can be climbed or avoid stairs that are too high. This also applies to randomly placed objects on the work surface that the Robot 100 may or may not be able to safely cross. In the case of obstacles or thresholds that Robot 100 sees, it is necessary for Robot 100 to be able to make a smooth transition to maximize smoothness and minimize instability due to sudden acceleration. You can go up knowing these heights, which can moderately slow down if done. In some embodiment, sill and staircase detection is based on object height on the work surface along with geometric recognition (eg, sill or stain with electrical cable, eg sock distinction). The threshold can be recognized by edge detection. The controller 500 receives imaging data from the cliff detection proximity sensor 420 (or another imaging sensor attached to the robot 100), executes an edge detection routine, and issues a drive command based on the result of the edge detection routine. be able to. The controller 500 can also identify an object by using pattern recognition. The threshold detection allows the robot 100 to orient its orientation with respect to the threshold to maximize its ability to climb smooth stairs.

Proximity sensors 410,420 can function alone or, as a variant, in combination with one or more contact sensors 430 (eg, bump switches) for the sake of completeness. For example, one or more contact or bump sensors 430 attached to the robot body 110 can detect whether the robot 100 has physically encountered an obstacle. Such sensors can utilize physical properties within the robot 100, such as capacitance or physical displacement, to ascertain when the robot encounters an obstacle. In some embodiment, each base body portion 124a, 124b, 124c of the base 120 detects the motion of the corresponding base body portion 124a, 124b, 124c with respect to the base chassis 122 (see, eg, FIG. 4A). Has a related contact sensor 430 (eg, capacitive sensor, reed switch, etc.). For example, the base bodies 124a, 124b, 124c can move radially with respect to the Z axis of the base chassis 122 to allow three-way bump detection.

With reference to FIGS. 1 to 4C, 9 and 10A, in some embodiment, the sensor system 400 has a laser scanner 440 mounted on the front portion of the robot body 110 and in communication with the controller 500. .. In the illustrated embodiment, the laser scanner 440 is the first base.<u style="single">Body</u>Attached to base 120 facing forward (eg, having a field of view along forward drive direction F) at or above 124a (eg, to obtain maximum imaging coverage along forward drive direction F). There is. In addition, placing the laser scanner at or near the anterior tip of the triangular base 120 causes the outside angle of the robot base (eg, 300 °) to be greater than the field of view 442 (eg, about 285 °) of the laser scanner 440. Large, thus, means that the base 120 is prevented from obstructing or obstructing the detection field of view 442 of the laser scanner 440. Do not block the field of view of the laser scanner 440 to minimize the portion of the laser scanner that protrudes beyond the base body 124 (for example, for aesthetics and to minimize the risk of getting caught in obstacles). It is better to be provided in a retracted state in the base body 124 as much as possible.

The laser scanner 440 scans the area around the robot 100, and the controller 500 uses the signal received from the laser scanner 440 to create an environment map or object map of the scanned area. Controller 500 can use object maps for navigation, obstacle detection and obstacle avoidance. In addition, the controller 500 can use sensory inputs from other sensors in the sensor system 400 to create and / or navigate object maps.

In some embodiments, the laser scanner 440 is a scanning lidar, which uses a laser that quickly scans the area in one direction as the "main" scanning line and a depth for each pixel generated in this line. A time-of-flight imaging element that uses a phase difference or similar technique can be used to assign to (return to a two-dimensional depth line in the scan plane). To create a 3D map, lidar can perform an "auxiliary" scan in a second direction (eg, by "nodding" the scanner). This mechanical scanning technique, if not captured, is a technology such as "flash" LIDAR / LADAR and "Swiss Ranger" focal plane imaging element sensors, depth at each pixel or series of depths at each pixel. It can be perfected by technology using semiconductor stacks to allow time-of-flight calculations for a complete 2-D matrix of pixels (by an encoded illuminator or illumination laser).

The sensor system 400 preferably has one or more three-dimensional (3-D) image sensors 450 in communication with the controller 500. If the 3-D image sensor 450 has a limited field of view, the controller 500 or sensor system 400 operates the 3-D image sensor 450a in a lateral scanning fashion to create a relatively wide field of view, thereby providing a robust ODOA. Can be carried out.

With reference to FIGS. 2 and 4A to 4C again, the sensor system 400 is the overall center of gravity CG of the robot 100._RIt is preferable to have an inertial measurement unit (IMU) 470 in communication with the controller 500 to measure and monitor the moment of inertia of the robot 100 with respect to.

The controller 500 can monitor the deviation of the feedback from the IMU470 to the threshold signal corresponding to normal unrestricted operation. For example, if the robot begins to lean from an upright position, the robot will say "<u style="single">Close line</u>Or be disturbed in a different way, or someone may suddenly add a heavy payload. In these cases, it is necessary to take urgent measures (including, but not limited to, recalibration and / or generation of audio / visual warnings) to ensure the safe operation of Robot 100. In some cases.

Since the robot 100 can work in a human environment, the robot 100 can interact with humans and work in a space designed for humans (without considering the constraints of the robot). Robot 100 can limit its drive speed and acceleration when in a crowded, restrained or highly dynamic environment, for example at a cocktail party or a busy hospital. However, the robot 100 may encounter situations where it is safe to drive relatively quickly, for example in a long, empty corridor, for example if something crosses the robot's path of motion. It is possible to decelerate suddenly.

When accelerating from a stop, controller 500 will take the robot's overall center of gravity CG_RIt is possible to prevent the robot from tipping over in consideration of the moment of inertia of the robot 100 from. The controller 500 can use a model of its attitude including its current moment of inertia. When supporting the payload, the controller 500 is the overall center of gravity CG_RIt is possible to monitor the motion of the robot moment of inertia by measuring the effect of the load on the robot. For example, the torso 140 and / or the neck 150 may have a strain gauge to measure strain. If this is not possible, the controller 500 should add a test registration command to the drive wheels 210 and use the IMU470 to measure the actual linear and angular acceleration of the robot to empirically set safety limits. Can be done.

During a sudden deceleration, the commanded load on the second and third drive wheels 210b, 210c (rear wheels) is reduced, during which the first drive wheels 210a (front wheels) slip in the forward drive direction. Supports robot 100. If the loads on the second and third drive wheels 210b, 210c (rear wheels) are asymmetric, the robot 100 will "yaw", which reduces dynamic stability. An IMU470 (eg, a gyro) can be used to detect this sway and issue commands to the second and third drive wheels 210b, 210c to reorient the robot 100.

With reference to FIGS. 1 to 3, 9 and 10A, in some embodiment examples, the robot 100 is attached to the front portion of the robot body 110 and has a field of view along the front drive direction F ( For example, it has a scanning 3-D image sensor 450a (so that it has the maximum imaging coverage along the driving direction F of the robot). The scanning 3-D image sensor 450a can be used primarily for obstacle detection / obstacle avoidance (ODOA). In the illustrated embodiment, for example, the shoulder 142 on the torso 140 in a retracted state (eg, flush with or beyond the bottom surface 144) as shown in FIG. Mounted below or on the bottom 144, it prevents contact between the user and the scanning 3-D image sensor 450a. The scanning 3-D image sensor 450a points downward in front of the robot 100 for obstacle detection and obstacle avoidance (ODOA) (eg, with interference from the base 120 or other parts of the robot body 110). It is preferable that the robot body is arranged so as to have a field of view 452, practically downward and facing away from the robot body 110. By placing the scanning 3-D image sensor 450a on or near the front edge of the body 140, the field of view of the 3-D image sensor 450 (eg, about 285 °) is reduced to the 3-D image sensor 450. In contrast, it can be less than the outer surface angle of the torso 140 (eg, 300 °), thus obstructing or obstructing the detection field of view 452 of the scanning 3-D image sensor 450a. Is blocked. In addition, the scanning 3-D image sensor 450a (and associated actuators) does not block its field of view (eg, for aesthetic purposes and to minimize catching on obstacles) as much as possible on the torso 140. It is better to install it in a retracted state. The distracting scanning motion of the scanning 3-D image sensor 450a is invisible to the user and thus reduces the experience of distracting interactions. Unlike protruding sensors or features, the retracted scanning 3-D image sensor 450a interacts unintentionally with the environment, especially when moving or scanning (people, obstacles). Etc.) will not be likely to occur. This is because there are virtually no moving parts that extend beyond the envelope of the torso 140.

In some embodiment, the sensor system 400 has an additional 3-D image sensor 450 provided at the base 120, legs 130, neck 150 and / or head 160. In the embodiment shown in FIG. 1, the robot 100 has a 3-D image sensor 450 provided on a base 120, a body 140, and a head 160. In the embodiment shown in FIG. 2, the robot 100 has a 3-D image sensor 450 provided on a base 120, a body 140 and a head 160. In the embodiment shown in FIG. 9, the robot 100 has a 3-D image sensor 450 provided on the legs 130, the torso 140 and the neck 150. Other forms can also be adopted. One 3-D image sensor 450 (eg, attached to the neck 150 above the head 160) can be used for people's recognition, gesture recognition and / or video conferencing, while the other. Another 3-D image sensor 450 (eg, attached to base 120 and / or leg 130) can be used for navigation and / or obstacle detection and obstacle avoidance.

A forward-facing 3-D image sensor 450 on the neck 150 and / or head 160 can be used to recognize the face and / or gestures of individuals around the robot 100. For example, using the signal input from the 3-D image sensor 450 provided on the head 160, the controller 500 creates a 3D map of the user's face that is visible and / or captured, and the created 3D map. Can be recognized by comparing with a known 3-D image of a person's face and confirming a match with one of the known 3-D facial images. Face recognition is preferably used to identify the user as an acceptable user of Robot 100. In addition, one or more of the 3-D image sensors 450 determine an individual gesture viewed by the robot 100 and, optionally, the determined gesture (eg, pointing, waving). Can be used to react on the basis of cues and / or hand signals). For example, the controller 500 can issue a drive command in response to a recognized hand pointing in a particular direction.

FIG. 10B is a schematic diagram of the robot 900 equipped with the camera 910, the sonar sensor 920 and the laser range finder (distance meter) 930, and the camera 910, the sonar sensor 920 and the laser range finder 930 are all attached to the robot body 905. Each of these has a field of view parallel to or substantially parallel to the ground G. With this configuration, it is possible to detect an object located at a distance. In the embodiment, the laser range finder 930 detects an object near the ground G, and the ring-shaped ultrasonic sensor (sonar) 920 detects an object located further above the ground G, and the camera The 910 captures most of the scene from a high vantage point. An important feature of this design is that the sensors 910, 920, 930 are all oriented parallel to the ground G. One advantage of this configuration is that the robot 900 travels before the distance to the object measured using one or more of the sensors 910,920,930 is before the robot 900 touches the object in the corresponding given direction. It means that computer calculation can be simplified in the sense that it is also a distance that can be achieved. The drawback of this configuration is that many levels of detection are required to get good coverage around the robot. This may be prohibited from a cost and computer computing standpoint, which often results in large gaps in the sensory field of view of all of Robot 900's sensors 910,920,930.

In some embodiment, the robot has a sonar scanner 460 to obtain acoustic imaging of the area around the robot 100. In the embodiments shown in FIGS. 1 and 3, the sonar scanner 460 is provided in the anterior portion of the base 120.

With reference to FIGS. 1, 3B and 10A, in some embodiment, the robot 100 is a laser scanner or laser rangefinding 440 for complete detection and a sonar proximity sensor facing backwards for safety. We are using 410j, both of which are directed parallel to the ground G. The robot 100 is preferably equipped with first and second 3-D image sensors 450a, 450b (depth cameras) to allow robust detection around the robot 100. The first 3-D image sensor 450a is attached to the torso 140 in a state of being pointed downward at a constant angle with respect to the ground G. By tilting the first 3-D image sensor 450a downward, the robot 100 receives a dense sensor coverage of the area immediately in front of or adjacent to the robot 100, which coverage of the robot 100 in the forward direction. Suitable for short-time travel. The backward-facing sonar 410j detects objects when the robot is moving backwards. If backward movement is common to the robot 100, the robot 100 is a third 3-D image sensor 450 oriented back and forth to provide dense sensor coverage of the area immediately behind or adjacent to the robot 100. It is good to have.

The second 3-D image sensor 450b is attached to the head 160, which allows the neck 150 to be panned and tilted. The second 3-D image sensor 450b should be useful for remote drive. This is because it allows a person as an operator to see where the robot 100 is going. The neck 150 allows the operator to tilt and / or pan the second 3-D image sensor 450b to see both near and far objects. Panning the second 3-D image sensor 450b widens the associated horizontal field of view. During high speed travel, the robot 100 tilts the second 3-D image sensor 450b slightly downward to widen the field of view of both the 3-D image sensors 450a and 450b in the overall or combined state, and the robot 100 becomes an obstacle. Can be given enough time to avoid (because high speed generally means that the time to respond to an object is short). At slow speeds, the robot 100 can tilt the second 3-D image sensor 450b upwards or substantially parallel to the ground G to track the person the robot 100 is following. In addition, while driving at a relatively low speed, the robot 100 can pan the second 3-D image sensor 450b to widen its field of view around the robot 100. The first 3-D image sensor 450a should remain stationary when the robot is driving to increase the robot's perceptual range (eg, it should not move relative to the base 120). ..

The 3-D image sensor 450 may be able to generate data in the following formats: (i) depth maps, (ii) intensity images using reflectance and / or (iii) normal intensity images. is there. The 3-D image sensor 450 can obtain such data by image pattern matching, flight time measurements and / or phase lag shifts of light emitted from the source and reflected from the target.

In some embodiment, the inference or control software that can be executed on a processor (eg, the processor of the robot controller 500) is a combination of algorithms that is executed using various types of data generated by the sensor system 400. Use. The inference software processes the data collected from the sensor system 400 and outputs data for making navigational decisions about where the robot 100 can move, for example, without colliding with obstacles. Accumulation of imaging data over time around the robot allows inference software to apply effective methods to selected segments of detected images to improve depth measurements on the 3-D image sensor 450. .. This should include the use of suitable temporary and spatial averaging techniques.

The reliability of performing robotic collision-free movements is based on (i) confidence levels obtained by high-level inference over time and (ii) depth perception sensors that accumulate three main forms of data for analysis. Of course, the three main formats of data are (a) depth image, (b) active lighting image and (c) ambient lighting image. Algorithms for recognizing various types of data should be run for each of the images obtained by the depth perception imaging sensor 450. The collected data can improve the reliability level as compared with a system using only one of various data.

The 3-D image sensor 450 can obtain an image including depth and brightness data from a scene around the robot 100 containing one or more objects (eg, a sensor-visible portion of a room or work area). The controller 500 is preferably configured to determine occupancy data about the object based on the capture of reflected light from the scene. In addition, the controller 500 issues drive commands to the drive system 200, at least in part, based on occupied data to bypass obstacles (ie, objects in the scene) in some embodiments. The 3-D image sensor 450 can repeatedly capture the scene depth image for real-time determination by the controller 500 and move the robot 100 through the scene without colliding with an object in the scene. For example, the speed or frequency at which depth image data is obtained by the 3-D image sensor 450 can be controlled by the shutter speed of the 3-D image sensor 450. In addition, the controller 500 receives an event trigger (eg, from another sensor component of the sensor system 400 (eg, from proximity sensors 410,420) and informs the controller 500 that there is an object nearby or is at risk. The 500 can allow the 3-D image sensor 450 to capture depth images and increase the frequency of obtaining occupancy information in response to event triggers.

Referring to FIG. 11, in some embodiment, the 3-D imaging sensor 450 has a light source 1172 that emits light into a scene 10, eg, a region around the robot 100 (eg, a room). The imaging sensor 450 captures reflected light from scene 10 including reflected light originating from light source 1172 (eg, as a scene depth image) imager 1174 (eg, photoelectric pixels 1174p arranged in an array). It is better to have more. In some embodiments, the imaging sensor 450 has a light source lens 1176 and / or a detector lens 1178 that manipulate (eg, speckle or focus) the emitted and received reflected light, respectively. The robot controller 500 or a sensor controller (not shown) in communication with the robot controller 500 receives an optical signal from the imager 1174 (eg, pixel 1174p) to match the image pattern and / or to reflect the reflected light captured by the imager 1174. Determine depth information for object 12 in scene 10 based on time-of-flight characteristics.

FIG. 12 provides an exemplary configuration or flow diagram 1200 of an operation or step in which the imaging sensor 450 is operated. Further referring to FIG. 10A, the operation receives the reflected light from the step (1202) of emitting light to scene 10 around the robot 100 and the emitted light from scene 10 on an imager (eg, an array of photoelectric pixels). Includes step (1204). The operation is a step in which the controller 500 receives the photodetection signal from the imager (1206), and a step in which the image data derived from the photodetection signal is used to detect one or more features of the object 12 in the scene 10 (1206). It further includes 1208) and the step (1210) of tracking the location of the detection feature of the object 12 in the scene 10 using the image depth data derived from the photodetection signal. The operation is the step of repeating the step of emitting light (1202) (1212), the step of receiving the reflected light of light (1204), the step of receiving the light detection signal (1206), the step of detecting the feature of the object (1208), and the step. It is preferable to include the step (1210) of tracking the position of the object feature so as to increase the resolution of the image data or the image depth data and / or provide a confidence level.

The iterative step (1212) can be performed at a relatively slow speed (eg, slow frame rate) to obtain a relatively high resolution, an intermediate speed or a high speed with a relatively low resolution. The frequency of repeating steps (1212) should be adjustable by the robot controller 500. In some embodiment, controller 500 can increase or decrease the frequency of iterative steps (1212) upon receiving an event trigger. For example, a presented item in a scene may cause an increased frequency of repeating steps (1212) to detect a potentially prominent object 12 (eg, doorway, threshold or cliff) in scene 10. Can trigger an event to do. In additional embodiments, elapsed time events between the detected objects 12 may slow down the frequency of repeating steps (1212) or stop for a given period of time (eg, sleep until caused by another event). become). In some embodiments, one or more detection steps (1208) of object 12 in scene 10 are repeated to trigger a feature detection event, thereby increasing the rate at which image depth data is obtained. Make step (1212) relatively frequent. The relatively high collection speed of image depth data allows for relatively reliable tracking of features within the scene.

The operation further includes step (1214) of outputting navigation data to bypass the object 12 in the scene 10. In some embodiment, the controller 500 uses the output navigation data to issue a drive command to the drive system 200, thereby moving the robot 100 in a way that avoids collision with the object 12.

In some embodiment, the sensor system 400 detects a large number of objects 12 in the scene 10 around the robot 100, and the controller 500 tracks the position of each of the detected objects 12. The controller 500 can create an occupancy map of an object 12 in an area around the robot 100, eg, a bounded area of a room. The controller 500 can use the image depth data of the sensor system 400 to match the scene 10 with a portion of the occupancy map and update the occupancy map at the location of the tracked object 12.

Referring to FIG. 13, in some embodiment, the 3-D image sensor 450 has a three-dimensional (3-D) speckle camera 1300, which is an image due to speckle non-correlation. Allows mapping. The speckle camera 1300 emits a speckle pattern into scene 10 (as a target area) speckle emitter 1310 (eg, infrared UV and / or visible light) and speckle on the surface of object 12 in scene 10. It has an imager 1320 that captures the image of the pattern.

The speckle emitter 1310 preferably has a light source 1312, such as a laser, that reflects and thus emits a beam of light into the diffuser 1314 as a speckle pattern into scene 10 for projection. The imager 1320 preferably has an objective optical element sensor 1322, which focuses the image on an image sensor 1324 having an array of photodetectors 1326, such as a CCD or CMOS technology utilization image sensor. Although the optical axes of the speckle emitter 1310 and the imager 1320 are shown to be located on the same straight line in the non-correlation mode, for example, the optical axes of the speckle emitter 1310 and the imager 1320 are shown to be on the non-aligned line Well, on the other hand, for example in cross-correlation mode, the imaging axis will deviate from the emission axis.

The speckle emitter 1310 emits a speckle pattern into scene 10, and the imager 1320 emits various object distances Z from the speckle emitter 1310._nCaptures the reference image of the speckle pattern in scene 10 in the range (eg, where the Z axis can be determined by the optical axis of the imager 1320). In the illustrated embodiment, the projected speckle pattern reference image is at various distances from the origin, eg Z.₁, Z₂, Z₃It is captured by a series of planes such as the reference place marked with. The difference ΔZ between the reference images should be configurable at a threshold distance (eg, 5 mm) or adjustable by controller 500 (eg, in response to a triggered event). The speckle camera 1300 archives the captured reference images and assigns them to their respective emission distances, thereby allowing the speckle pattern to be uncorrelated with the distance from the speckle emitter 1310, thereby following: Allows distance measurement of the object 12 captured in the image. Reference distance Z where ΔZ is adjacent₁, Z₂, Z₃, ... Place Z, assuming it is approximately equal to the distance between each other_AFor example, Z speckle pattern for object 12 at₂It can be correlated with the reference image of the speckle pattern captured at. On the other hand, Z_BFor example, Z speckle pattern for object 12 at₃It can be correlated with the reference image of the place. These correlation measurements give an appropriate distance to the object 12 from the origin. Since the object 12 is mapped in three dimensions, the speckle camera 1300 or the controller 500 that receives information from the speckle camera 1300 can use the local cross-correlation with the reference image given the closest collation.

Other details and features relating to 3-D image mapping using speckle ranging by, for example, triangulation or speckle cross-correlation using non-correlation, which can be combined with the details and features described herein, are international. Application PCT / IL 2006/000335, which is found in this international application, is cited by reference and the entire description thereof is incorporated herein by reference.

FIG. 14 shows an exemplary configuration or flow of operations or steps for operating the speckle camera 1300. The operation involves emitting a speckle pattern into scene 10 (1402) and capturing a reference image (eg, a reference image of reference object 12) at various distances from the speckle emitter 1310 (1404). Including. The operation further includes the step of emitting the speckle pattern to the target object 12 in the scene 10 (1406) and the step of capturing the target image of the speckle pattern on the object 12 (1408). The operation is a step (1410) of comparing the target image (target image of the speckle object) with various reference images to identify the reference pattern having the strongest correlation with the speckle pattern on the target object 12 and the target in the scene 10. Further including the step (1412) of finding the estimated distance range of the object 12. This further includes the step of locating the major speckle pattern on the object 12 and the step of finding the reference image having the speckle pattern most strongly correlated with the major speckle pattern on the object 12. The distance range can be obtained from the corresponding distance of the reference image.<u style="single">Since the above-mentioned reference object and target object are included in the object 12 (in other words, the object 12 includes various objects other than the reference object and the target object), the object with respect to the reference object and the target object. The same code "12" as 12 is attached.</u>

The operation optionally creates a 3-D map of the surface of the object 12 by local cross-correlation between the speckle pattern on the object 12 and the identified reference pattern, for example to locate the object 12 in the scene. Includes step (1414). It finds the steps to locate the main speckle pattern on object 12 and the respective offsets between the main speckle pattern on multiple regions of object 12 in the target image and the main speckle pattern in the identified reference image. It is good to include a step to derive a three-dimensional (3-D) map of the object. The use of solid-state components for 3-D mapping of scenes provides a relatively inexpensive means for robot navigation systems.

Typically, at least some of the various distances are axially separated by a distance longer than the axial length of the main speckle pattern at each distance. The step of comparing the target image with the reference image has the step of computer-computing the respective cross-correlation between the target image and at least some of the reference images and the strongest cross-correlation with the target image. It is good to include a step to select a reference image.

The operation includes a step (1416) that repeats steps (1402 to 1412) or steps (1406 to 1412) and optionally a step (1414) (eg, continuously) that tracks the movement of object 12 in scene 10. Is good. For example, the speckle camera 1300 can capture a series of target images while the object 12 is moving for comparison with a reference image.

Other details and features relating to 3-D image mapping using speckle ranging that can be combined with the details and features described herein are described in US Pat. No. 7,433,024, US Patent Application Publication No. 2008/0106746. Book (Title of Invention: Depth-varying light fields for three dimensional sensing), US Patent Application Publication No. 2010/0118123 (Title of Invention: Depth Mapping Using Projected Patterns), US Patent Application Publication No. 2010/0034457 Book (Title of Invention: Modeling Of Humanoid Forms From Depth Maps), US Patent Application Publication No. 2010/0020078 (Title of Invention: Depth Mapping Using Multi-Beam Illumination), US Patent Application Publication No. 2009/0185274 (Invention Name: Depth Mapping Using Multi-Beam Illumination) Invention title: Optical Designs For Zero Order Reduction), US Patent Application Publication No. 2009/0096783 (Title of Invention: Three-Dimensional Sensing Using Speckle Patterns), US Patent Application Publication No. 2008/0240502 (Title of Invention: Depth Mapping Using Projected Patterns) and US Patent Application It is found in Publication No. 2008/0106746 (Title of the Invention: Depth-Varying Light Fields For Three Dimensional Sensing), these patent documents are cited by reference, and the entire description thereof is incorporated herein by reference.

Referring to FIG. 15, in some embodiment, the 3-D imaging sensor 450 has a 3-D time-of-flight (TOF) camera 1500 for obtaining depth image data. The 3-D TOF camera 1500 is a processing resource that is in contact with the light source 1510, the complementary metal oxide semiconductor (CMOS) sensor 1520 (or charge-coupled device (CCD)), the lens 1530 and the light source 1510, and the CMOS sensor 1520. It has a control logic or camera controller 1540 with resources) (and / or robot controller 500). The light source 1510 is preferably a laser or light emitting diode (LED) with an intensity adjusted by a periodic high frequency signal. In some embodiments, the light source 1510 has a focusing lens 1512. The CMOS sensor 1520 preferably has pixel detectors 1522 arranged in an array or pixel detectors 1522 in other arrangements, in which case each pixel detector 1522 has the intensity of the photonic energy hit by it. And the phase can be detected. In some embodiments, each pixel detector 1522 has a dedicated detector circuit 1524 that processes the detected charge output of the associated pixel detector 1522. The lens 1530 focuses or focuses the light reflected from the scene 10 containing one or more objects 12 of interest on the CMOS sensor 1520. The camera controller 1540 provides a series of operations to format the pixel data obtained by the CMOS sensor 1520 into a depth map and a luminance image. In some embodiments, the 3-D TOF camera 1500 has an input / output (IO) 1550 (eg, in contact with the robot controller 500), memory 1560 and / or camera controller 1540 and / or pixel detector. It also has a clock 1570 in contact with the 1522 (eg, detector circuit 1524).

FIG. 16 shows an exemplary flow of operations or steps for operating the 3-D TOF camera 1500. The operation begins the step (1602) of emitting a light pulse (eg, infrared, ultraviolet and / or visible light) into scene 10 and the timing of the flight time of the light pulse (eg, counting the clock pulse of clock 1570). Includes step (1604). The operation involves receiving the reflected light of the emitted light from one or more surfaces of the object 12 in the scene 10 (1606). The reflected light is at various distances Z from the light source 1510._nIt should be the reflected light from the surface of the object 12 located at. The reflected light is received on the pixel detector 1522 of the CMOS sensor 1520 through the lens 1530. Operation is CMOS sensor 152<u style="single">0</u>Includes step (1608) of receiving the time of flight for each light pulse reflected light received on each corresponding pixel detector 1522. During the round trip flight time (TOF) of the optical pulse, the detector circuit 152 of each pixel detector 1522<u style="single">4</u>The counter of is accumulating clock pulses. The large number of accumulated clock pulses represent a long TOF and therefore a long distance between the light reflection point on the imaged object 12 and the light source 150. The operation further includes finding the distance between the reflecting surfaces of the object 12 for each received light pulse reflected light (1610) and optionally constructing a three-dimensional object surface (1612). In some embodiment, the operation includes repeating steps (1602-1610) (1614) and optionally tracking the movement of object 12 in scene 10 (1612).

Other details and features relating to 3-D time-of-flight imaging that can be combined with the details and features described herein are described in U.S. Pat. No. 6,323,942 (Invention title: CMOS Compatible 3-D Image Sensor), U.S.A. Patent No. 6,515,740 (Title of Invention: Methods for CMOS-compatible Three-Dimensional Image Sensing Using Quantum Efficiency Modulation), International Application PCT / US02 / 16621 (Title of Invention: Method and System to Enhance Dynamic Range Conversion) Usable with CMOS Three-Dimensional Imaging), these patent documents are cited by reference, and the entire description thereof is incorporated herein by reference.

In some embodiment, the 3-D imaging sensor 450 has three forms of information: (1) Depth information (eg, correspondence on scene 12 from each pixel detector 1522 of the CMOS sensor 1520). (To location), (2) ambient light intensity at each pixel detector location and (3) active illumination intensity at each pixel detector location. The depth information allows the position of the detected object 12 to be tracked over time, especially with respect to the proximity of the object to the robot deployment location. Active illumination intensity and ambient light intensity are different types of luminance images. The active illumination intensity is captured from the reflected light of the active light (eg, provided by light source 1510) reflected by the target object 12. The peripheral light image is that of the peripheral light reflected by the target object 12. Together, the two images provide additional robustness, especially when the lighting conditions are poor (eg, too dark or excessively high ambient lighting).

Image segmentation and classification algorithms can be used to classify and detect the position of object 12 in scene 10. The information provided by these algorithms as well as the distance measurement information obtained from the imaging sensor 450 can be used by the robot controller 500 or other processing resources. The imaging sensor 450 can operate on a time-of-flight principle, specifically in a tuned light pattern reflected from scene 10 that includes techniques for adjusting the sensitivity of photodiodes that filter ambient light. It can operate based on the detectable phase lag.

Robot 100 includes 1) mapping, location and navigation, 2) object detection and object avoidance (ODOA), 3) object hunting (eg, to find people), 4) gesture recognition (eg, for companion robots). ), 5) People and face detection, 6) People tracking, 7) Monitoring of object manipulation by Robot 100 and the use of Imaging Sensor 450 for other suitable applications for autonomous movement of Robot 100. it can.

In some embodiment, at least one of the 3-D image sensors 450 is on the robot 100 at a height above 1 foot (30.5 cm) or 2 feet (61.0 cm) above the ground. A point cloud imaging device (eg, speckle or time of flight) directed (eg, by omnidirectional drive system 200) that can be attached and obtain a point cloud from a spatial volume that includes a floor plane in the direction of movement of the robot. It should be a camera). In the embodiments shown in FIGS. 1 and 3, the first 3-D image sensor 450a is at least 1 or 2 feet above the ground and at a height above the ground (or about 1 or 2 feet above the ground). Forward drive direction F to capture an image of the volume including the floor (eg, volumetric point cloud) while being mounted and driven at base 120 (eg, for object detection and object avoidance). It is good to be sent along. The second 3-D image sensor 450b is located on the head 160 (eg, about 3 feet (91.5 cm) above the ground) so that skeleton recognition and definition point clouds can be obtained from the spatial volume located adjacent to the robot 100. ) Or 4 feet (122.0 cm) or more, at an upper height position) shown mounted. The controller 500 can run skeleton / digital recognition software to analyze the captured volume point group data.

It may be important to correctly detect the object 12 with the imaging sensor 450 regardless of the ambient light conditions. In many environments, lighting conditions range from direct sunlight to bright fluorescent lighting to dim shades, resulting in significant changes in the surface texture and basic reflectance of the object 12. The lighting may vary from scene 10 to scene 10 within a given location. In some embodiment, the image sensor 450 is used to identify and resolve a person and an object 12 in all situations with relatively little effect from ambient light conditions (eg, peripheral illumination rejection). It is good to be done.

In some embodiment, the VGA resolution of the imaging sensor 450 is 640 horizontal pixels x 480 horizontal pixels, but other resolutions, such as 320 x 240 (eg, for short range sensors), are also available. ..

The imaging sensor 450 preferably has a pulsed laser and a camera diaphragm that acts as a band filter in the time domain to see the object 12 only within a specified range. The variable aperture of the imaging sensor 450 can be used to detect the object 12 at various distances. Further, the pulsed high power laser can be used for outdoor applications.

Tables 1 and 2 (listed below) provide exemplary features, parameters and / or uses of the Imaging Sensor 450 for a variety of applications. The sensor 1 is preferably used as a general-purpose imaging sensor 450. Sensors 2 and 3 are preferably attached to a robot that interacts with humans, and sensors 4 and 5 are preferably attached to a coverage or cleaning robot.

<tables num="1"><img id="000002" he="190" wi="159" file="JP5963372B2_D0001.tif" img-format="tif" img-content="drawing" /></tables> table 1<tables num="2"><img id="000003" he="206" wi="159" file="JP5963372B2_D0001.tif" img-format="tif" img-content="drawing" /></tables> Table 2

The minimum sensor latency allows the object 12 to be seen quickly enough to avoid it when the robot 100 is in motion. The latency of the imaging sensor 450 should be a factor in the real-time response to the detected and recognized user gestures. In some embodiments, the imaging sensor 450 has a latency of about 44 milliseconds. The image captured by the imaging sensor 450 may preferably mean an attribute time stamp, which is used to determine the robot pose in which the image was captured while translating or rotating in space. it can.

The serial peripheral interface bus (SPI), which is in communication with the controller 500, should be used for communication with the imaging sensor 450. The use of the SPI interface for the Imaging Sensor 450 does not limit its use for multinode distributed sensor / actuator systems and does not limit its use for Ethernet-enabled devices such as microprocessors or field programmable gate arrays (FPGAs). ), Which in this case is described in US Patent Application No. 61 / 305,069 (Title of Invention: Mobile Roboto Communication System) filed on February 16, 2010. As such, data becomes available through Ethernet and EtherIO systems, this patent document is cited by reference and the entire description is incorporated herein by reference.

Since SPI is a restricted protocol, interrupt pins should be available to interface to the imaging sensor 450 that strobes or migrates when performing image capture. The interrupt pin allows communication with the controller 500 when a frame is captured. This allows the controller 500 to know that the data can be read at any time. In addition, interrupt pins can be used by the controller 500 to capture a time stamp that indicates when the image was taken. The imaging output of the imaging sensor 450 is often time stamped (eg, by the global clock of controller 500), which can be referenced to compensate for latency. In addition, time stamped imaging outputs from multiple imaging sensors 450 (eg, different parts of scene 10) can be synchronized and combined (eg, stitched together). The EtherIO system should capture the interrupt time (for interrupt pins) and make it available to the high-level equipment and software of the EtherIO system. Robot 100 preferably has a multi-node distributed sensor / actuator system that performs a clock synchronization scheme, such as IEEE 1588, which inventor can be applied to data captured from the imaging sensor 450. ..

Both the SPI interface and EtherIO should be memory address driven interfaces. For example, byte / word (word) / double word (double word) data can be read from the imaging sensor 450 via the SPI interface and available in the memory space of the EtherIO system. For example, local registers and memory provided within the FPGA, such as direct memory access (DMA) memory, can be used to control the EtherIO nodes of the EtherIO system.

In some cases, the robot 100 may need to scan the imaging sensor 450 left and right or sideways (eg, to look around object 12 or shield 16 (FIG. 17A)). For the robot 100, which is differentially steered, this is when the robot 100 is rotated in place by the drive system 200 or the mirror, prism, variable angle micromirror or MEMS mirror array associated with the imaging sensor 450. It is good to include.

Viewing angle θ smaller than 360 °_VThe field of view 452 of the imaging sensor 450 with can be magnified 360 ° by optics such as omnidirectional, fisheye, catadioptric (eg parabolic mirror, telecentric lens), panamorph mirror and lens. .. Since the controller 500 can utilize the imaging sensor 450 specifically for distance measurement, it is not necessarily used for images or images that are visible to humans (eg, for communication with humans). Distortion of the illumination of the light source 1172 (eg, warping) and / or image capture by the imager 1174 (Fig. 11) with optics is acceptable for ranging (eg, 3-D speckle camera 1300 and / or 3). -As with the D-TOF camera 1500).

In some cases, the imaging sensor 450 may be a black object 12, a surface of a changing albedo, a highly reflective object 12, a strong 3-D structure, a self-similar or periodic structure or a field of view 452 or this. It may be difficult to recognize and measure an object (for example, at or outside the horizontal and vertical viewing angles) located just beyond. In such cases, other sensors in the sensor system 400 can be used to complement the imaging sensor 450 or act as a redundant means of the imaging sensor 450.

In some embodiment, the light source 1172 (eg, 3-D speckle camera 1300 and / or 3-D TOF camera 1500) is an infrared (IR) laser, IR pattern illuminator or other IR illuminator. Has. Black objects, especially black cloth or carpet, can absorb IR and therefore cannot return strong enough reflected light for recognition by the imager 1174. In this case, either a secondary detection mode (eg, sonar) or an automatic calibration technique for expressed albedo differences may be required to improve the recognition of black objects.

An object 12 having a high refractive index or an object 12 having a considerably high specular enhancement (for example, a columnar or spherical object) may make distance measurement difficult for the imaging sensor 450. Similarly, an object 12 that is extremely sensitive to the wavelength of light detected by the imaging sensor 450 may pose a similar problem. Objects 12 made of glass, such as doors and windows, can be extremely reflective and, when distanced, look as if they were free space (finite distance) or a first non-mirror surface. Distance measurement is performed as reflected light on the reflective surface. As a result, the robot 100 does not see the object 12 as an obstacle, and as a result, the robot 100 may collide with a window or a door, which may cause damage to the robot or the object 12. To avoid this, the controller 500 should run one or more algorithms looking for surface discontinuities that match typical windowpane or doorway dimensions (consisting of straight lines). Then, these surfaces may be presumed to be obstacles rather than free space. Another embodiment of detecting a reflective object in the path of a robot is to use a reflective sensor that detects its own reflected light. Upon careful approach of an obstacle or object 12, reflection sensors can be used to determine if a specular object is in front or if the robot is safely occupying space.

For 3-D speckle camera 1300, light source 131<u style="single">2</u>May not form a recognizable pattern on the surface of the highly reflective object 12, or the imager 1320 may not recognize the speckle reflected light from the highly reflective object 12. In the case of the 3-D TOF camera 1500, the highly reflective object 12 is the range or distance (not to the object itself) to another object 12 reflected in the object 12 by the 3-D TOF camera 1500. May result in a multi-path situation. To cure IR failure modes, the sensor system 400 should employ acoustic flight time, millimeter wave radar, stereo or other vision technology that can use the slightest reflected light within Scene 10.

The mesh object 12 can make distance measurement difficult for the imaging sensor 450. If the object 12 is not present immediately behind the mesh of a particular porosity, the mesh will appear as a solid obstacle 12. However, if the object 12 moves behind the mesh and in the case of the 3-D speckle camera 1300, the speckle can reflect from the object 12 behind the mesh, and the object will be positioned even if it is behind the mesh. Even if you do, you will see it in the depth map, not in the mesh. If information about the pre-contributions to the identification of the mesh is available (before the object 12 moves behind the mesh), such information can be used to register the location of the mesh in future occupancy maps. By receiving information about the stochastic correlation of the receiving speckle maple at various distances, the controller 500 can locate a large number of porous or mesh objects 12 that are in line with the imaging sensor 450.

The controller 500 can utilize the imaging data from the imaging sensor 450 for color / size / dimension blob matching. By identifying the separate objects 12 in the scene 10, the robot 100 can not only avoid the collision but also search for the object 12. The human interface robot 100 may need to identify a person and a target object 12 against the background of a home or office environment. Controller 500 images the imaging sensor 450 as if the depth map was a single grayscale map looking for the same "color" (ie, depth continuity) to produce the continuous object 12 in scene 10. One or more colormap blob discovery algorithms can be run on the depth map derived from the data. Object matching is further enhanced by allowing segmentation of the color space as well as the depth space by enhancing the decision on how to segment the object 12 using the color map. The controller 500 can first detect the object 12 by depth and then further segment the object 12 by color. This allows the robot 100 to identify two objects 12 that are close to or in contact with each other with different optical qualities.

In the embodiment where the sensor system 400 has only one imaging sensor 450 (eg, a camera) for object detection, the imaging sensor 450 has a problem in imaging the surface in the absence of scene texture. , It may not be possible to analyze the scale of the scene. In addition, mirror and / or mirror enhancement of object 12 can result in saturation of a group of pixels 1174p in the imager 1174 (eg, saturation of the corresponding portion of the captured image), and in color images, mirror enhancement is a different perspective. Some may appear to be separate, which interferes with image matching for, for example, the speckle camera 1300.

By using or collecting two or more sensors for object detection, it is possible to provide a relatively robust and highly redundant sensor system 400. For example, flash lidar generally has a low dynamic range and rotating scanners generally require long inspection times, but these types of sensors can be useful for object detection. In some embodiment, the sensor system 400 is in contact with the controller 500, in addition to the imaging sensor 450 (eg, 3-D speckle camera 1300 and / or 3-D TOF camera 1500), as well as flash lidar. And / or have a rotating scanner. The controller 500 identifies the object 12 using the detection signals from the imaging sensor 450 and the flash laser radar and / or rotary scanner, determines the distance of the object 12 from the robot 100, and obtains a 3-D map of the surface of the object 12. Along with creating / or you can create or update the Occupation Map 1700. With the 3-D speckle camera 1300 and / or the 3-D TOF camera 1500, by initializing distance measurement, filling low texture areas, detecting depth discontinuities and / or fixing the scale. Can address the weaknesses of color or stereo cameras.

In the embodiment using the 3-D speckle camera 1300, the speckle pattern emitted by the speckle emitter 1310 should be rotationally invariant with respect to the imager 1320. further. An additional camera 1300 (color or stereo camera) used in conjunction with the 3-D speckle camera 1300 and / or the 3-D TOF camera 1500 is used to handle ego rotation, tilt, perspective and / or scale (distance). It is advisable to employ a feature detector that is somewhat or completely scale-rotating affine-invariant. Scale-invariant feature transformation (or SIFT) is an algorithm for detecting and / or explaining local features in an image. SIFT is a controller for object recognition, robot mapping and navigation, 3-D modeling, gesture recognition, video tracking and collation movement.<u style="single">500</u>(Sensor system<u style="single">400</u>Can be used by (using data from). SIFT enables the placement of signs with respect to feature parts in scene 10 as scale-invariant and rotation-invariant conversions, and when the identified feature parts in scene 10 are located far away or rotated. Even if there is, it can help to get the identified features in scene 10 again. For example, applying SIFT to a normal image allows the recognition of a moving object 12 (eg, face or button or some text), the object 12 having the same brightness or color pattern and becoming larger or smaller. It is identified whether it has been rotated or rotated. Other of the transformations that are affine invariant and can explain skew or distortion to identify the object 12 from an angle can be employed. The sensor system 400 and / or the controller 500 includes SIFT, RIFT, affine SIFT, RIFT, G-RIF, SURF, PCA-SIFT, GLOH.PCA-SIFT, SIFTw / FAST corner detection and / or scale-invariant vocabulary tree and / Or SIFTw / Irregular By adopting Orientation Histogram Binning, it is possible to provide scale-invariant feature recognition (for example, by a color or stereo camera).

In some embodiment, controller 500 executes a program or routine that employs SIFT and / or other transformations for object detection and / or identification. The controller 500 can receive image data from the image sensor 450 or IR camera, such as colors, black and white. In some embodiments, the image sensor 450 is a 3-D speckle IR camera capable of providing image data without speckle illumination to identify features without the advantage of speckle ranging. Controller 500 can identify or tag features or objects 12 previously mapped to the 3-D scene from speckle distance measurement. Depth maps can be used to filter and improve the SIFT recognition rate applied to camera-imaged features and / or to simplify scale invariance (movement during range). And changes are both known and can be associated with scales). SIFT transforms may be useful for frame-by-frame position change standardized and / or shifted depth map data, with inertial tracking, odometry, proprioception and / or beacon criteria. The robot can track such depth map data. For example, the transformations applied to scale and rotation immutability may still be effective in recognizing local features of the depth map if the depth map is momentum-fed in the direction of the features.

Other details and features for SIFT-like or other feature descriptors to 3-D data that can be combined with details and features as described herein are Se S, Lowe, David. David G, Little J, "Vision-Based Mobile Robot Localization and Mapping Using Scale-"<u style="single">Features (Vision-based mobile robot localization and mapping using scale-invariant features</u>) , Proceedings of the Institute of Electrical and Electronic Engineers International Conference on Robotics and Automation (ICRA) (Proceedings) of the IEEE International Conference on Robotics and Automation (ICRA), 2001, p.2051 or F Rothganger, S Lasebnik, C Schmid and J. Ponce), "3D Object Modeling and Recognition Using Local Affine-Invariant", "3D Object Modeling and Recognition Using Local Affine-Invariant" Image Descriptors and Multi-View Spatial constrains), International Conference on Computer Vision (ICCV), 2004, Iruna Gordon and David G, Lowe, Wat and Hoa: Three Dee of the<u style="single">Ekto Rikaguni</u>Shan with Accurate Pose (<u style="single">What and where: 3D object recognition with accurate pose</u>) , Toward Category-Level Object Recognition, Springer-Verlag, 2006, p67-82, cited by reference to these non-patent documents. , The entire contents of these descriptions are incorporated in this specification.

Other details and features regarding 3-DSIFT-suitable techniques in recognizing human actions, including falling, are Ivan Laptev and Tony Lindeberg,<u style="single">「</u>Local descriptors for spatio-temporal recognition<u style="single">」</u>, European Conference on Spatial Coherence for Visual Motion Analysis, Springer Lecture Notes in Visual Motion Analysis, ECCV'04 Workshop on Spatial Coherence for Visual Motion Analysis Springer Lecture Notes in Computer Science, No. 3667, 2004, p.91-103, Ivan Laptev, Barbara Caputo, Christian Schuldt and Tony Tony Lindeberg, "Local Velocity-Adapted Motion Events for Spaceio-Temporal Lecture Notes"<u style="single">Local velocity-adapted motion events for spatio-temporal recognition</u>) , Computer Vision and Image Understanding 108, p.207-229, 2007, Paul Scovanner, S Ali, M. M Shah, "A Three Dimensional Shift Descriptor and It's Application<u style="single">Too Action</u>N Rika Gunishan (<u style="single">A 3-dimensional sift descriptor and its application to action recognition</u>) , Proceeding of the 15th International Conference on Multimedia, p.357-360, 2007, J Niebles, H. One (H Wang), Fei-Fei Li, "Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words (Temporal Words)<u style="single">Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words</u>) , Proceeding of the British Machine Vision Conference (BMVC), and these non-patent documents are cited by reference. , The entire contents of these descriptions are incorporated in this specification.

The controller 500 is an imaging sensor 450 (eg, a depth map sensor) in creating a 3-D map of the surface of object 12 to fill holes from depth discontinuities and fix the metric scale of the 3-D model. Is better to use. The sensor pose can be estimated using structure-from-motion reinforced with depth map sensor range data. A typical structure from motion pipeline should include viewpoint invariant feature estimation, in-camera feature matching and bundling methods.

Software means that combine the features of a color / stereo camera with an imaging sensor 450 (eg, 3-D speckle camera 1300 and / or TOF camera 1500) include (1) sensor pose estimation and (2) depth map estimation. And (3) 3-D mesh estimation. In the sensor pose estimation, the position and orientation of the sensor package for each image capture are obtained. Depth map estimation provides a high resolution depth map for each image. In 3-D mesh estimation, sensor pose estimation and depth maps can be used to identify objects of interest.

In some embodiment, it is preferable to use the color or stereo camera 320 (Fig. 9) and the 3-D speckle camera 1300 or the 3-D TOF camera 1500 at the same time. Standoff distances of 1 meter and 45 ° field of view 452 can provide reasonable circuit time and overlap between views. If at least two pixels are needed for 50% detection, it is better to use a color camera with at least 1 megapixel resolution for a lens with a 45 ° field of view 452, proportionally for a 60 ° or wider field of view 452. Larger resolution is used.

Depth maps can reliably assign a collection of pixels from a color / stereo image to an accurate surface, although depth map sensors may have relatively low resolution and range accuracy. Thereby, the stereo visual error due to the absence of texture can be reduced, and the variation search range and the computer processing cost can be reduced by demarcating the range up to, for example, 5 cm intervals.

With reference to FIG. 10A again, the first and second 3-D image sensors 450a and 450b can be used to improve the mapping of the robot's environment and create a robot map. This is because the first 3-D image sensor 450a can be used to map nearby objects, and the second 3-D image sensor 450b can be used to map distant objects. is there.

With reference to FIGS. 17A and 17B, in some cases the robot 100 receives an occupancy map 1700 for the object 12 in the scene 10 and / or work area 5, or the robot controller 500 receives the imaging sensor 450 over time. For example, an occupation map 1700 is generated (and can be updated) based on the image data and / or the image depth data received from the second 3-D image sensor 450b). In addition to locating the robot 100 in scene 10 (eg, the environment around the robot 100), the robot 100 uses the sensor system 400 to move to other locations in the connected space (eg, work area 5). can do. The robot 100 maps a nearby area around the robot 100 to identify a relatively nearby object 12 of a short-range imaging sensor 450a (eg, of the body 140 as shown in FIGS. 1 and 3). A long-range image sensor 450 (eg, as shown in FIGS. 1 and 3) that maps a relatively large space around the robot 100 (mounted on the underside) and identifies an object 12 located relatively far away. It is better to have (attached to the head 160). Robot 100 uses the occupancy map 1700 to identify or confirm known objects 12 as well as shields 16 in scene 10 (eg, object 12 should or should be confirmed from the current vantage point). It is possible to identify a place that cannot be confirmed (a place that cannot be confirmed). The robot 100 can register a shield 16 or a new object 12 in the scene 10, and the shield 16 or a new object 12 to confirm the location of the new object 12 or any object 12 in the shield 16. You can try to bypass the object 12. In addition, Robot 100 can use Occupancy Map 1700 to locate and track the movement of Object 12 in Scene 10. For example, imaging sensors 450, 450a, The 450b can detect a new position 12'of the object 12 in the scene 10 without detecting the mapped position of the object 12 in the scene 10. The robot 100 can register the position of the old object 12 as the shield 16 and try to bypass the shield 16 to confirm the location of the object 12. The robot 100 can compare the new image depth data with the previous image depth data (eg, map 1700) and assign a confidence level for the location of the object 12 in the scene 10. The location confidence level of object 12 in scene 10 can time out after a threshold period. The sensor system 400 can update the location confidence level of each object 12 after each imaging cycle of the sensor system 400. In some embodiments, the new shield 16 detected within the shield detection period (eg, less than 10 seconds) (eg, the object 12 missing from the occupancy map 1700) is "living" in scene 10. It may mean object 12 (eg, moving object 12).

In some embodiment, the second object of interest 12b located behind the detected first object 12a in the scene 10 may not be initially detected as the shield 16 in the scene 10. It can be said that the shielding portion 16 is an area in the scene 10 that cannot be easily detected or visually recognized by the imaging sensors 450, 450a, 450b. In the illustrated embodiment, the sensor system 400 of the robot 100 (eg, or a portion thereof, eg, imaging sensors 450, 450a, 450b) has a viewing angle θ to view the scene 10._VIt has a field of view 452 with any angle between 0 ° and 360 °). In some embodiments, the imaging sensors 450, 450a, 450b have a 360 ° viewing angle θ._VOn the other hand, in other embodiments, the imaging sensors 450, 450a, 450b have a viewing angle θ of less than 360 ° (eg, about 45 ° to 180 °)._VHave. Viewing angle θ_VIn the embodiment where is less than 360 °, the imaging sensors 450, 450a, 450b (or these components) have a 360 ° viewing angle θ._VCan be rotated relative to the robot body 110 to achieve. The imaging sensors 450, 450a, 450b have a horizontal viewing angle θ._VHVertical viewing angle θ that is the same as or different from this_VVCan have. For example, the imaging sensors 450, 450a, 450b have a horizontal viewing angle θ of at least 45 °._VHAnd a vertical viewing angle of at least 40 ° θ_VVCan have. In some embodiment, the imaging sensors 450, 450a, 450b or parts thereof are the robot body 110 and the drive system.<u style="single">200</u>Can move against. In addition, to detect the second object 12b, the robot 100 is driven around scene 10 in one or more directions (eg, working).<u style="single">region</u>(By translation and / or rotation on 5) the imaging sensors 450, 450a, 450b can be moved to obtain a vantage point that allows the detection of the second object 12b. Robotic or independent motion of the imaging sensors 450, 450a, 450b or parts thereof can also solve the problem of monocularity.

It is good to assign a confidence level to the detected location or tracked movement of the object 12 in the work area 5. For example, when creating or updating the occupancy map 1700, the controller 500 may assign a confidence level for each object 12 on the map 1700. The confidence level should be directly proportional to the likelihood that object 12 actually exists in work area 5 as indicated on map 1700. The confidence level can be determined by many factors, such as the number and type of sensors used to detect the object 12. For example, the contact sensor 430 can provide the highest level of confidence because the contact sensor 430 detects the actual contact with the object 12 by the robot 100. The imaging sensor 450 can provide different confidence levels that may be higher than the proximity sensor 430. Information received from two or more sensors in the sensor system 400 can be collected or stored to provide a relatively high level of confidence for any single sensor.

Odometri is the use of data from actuator motion to estimate changes in position (mileage) over time. In some embodiments, an encoder is attached to the drive system 200 to measure the number of wheel revolutions and thus the mileage of the robot 100. The controller 500 should use odometry when assessing the level of confidence regarding the location of an object. In some embodiment, the sensor system 400 has an odometer and / or an angular velocity sensor (eg, a gyroscope or IMU470) to detect the mileage of the robot 100. A gyroscope is a device that measures or maintains orientation based on the conservation law of angular momentum. The controller 500 can use the odometer and / or the gyro signal received from the odometer and / or the angular velocity sensor to locate the robot 100 in the work area 5 and / or on the occupied map 1700. In some embodiments, the controller 500 uses dead reckoning. Dead reckoning is the process of estimating the current position based on a previously determined position and advancing that position based on known or estimated speeds and paths over time. Know the location of the robot in work area 5 (eg, via odometry, gyroscope, etc.) and know the detected location of one or more objects 12 in work area 5 (via sensor system 400). ), The controller 500 can evaluate the relatively high confidence level of the location or motion of the object (in contrast, without odometry or gyroscope) on the occupied map 1700 and within the work area 5.

Odometri based on wheel motion can be electrically noisy. The controller 500 may receive image data of the environment or scene 10 around the robot 100 from the imaging sensor 450 in order to computerize the robot motion by visual odometry independently of the wheels based on the odometry of the drive system 200. it can. In visual odometry, it is necessary to determine the motion of the imaging sensor 450 using optical flow. The controller 500 may preferably use the calculated motion based on the imaging data of the imaging sensor 450 to correct the wheel-based odometry error, thus improving mapping and motion control. Visual odometry may be limited in the case of low texture or low light scene 10 if the imaging sensor 450 is unable to track features in the captured image.

Other details and features relating to odometry and imaging systems that can be combined with the details and features described herein are described in US Pat. No. 7,158,317 (this US patent specification is "depth-of". -field) ", which describes an imaging system) and US Pat. No. 7,115,849 (which describes a wavefront coding interference contrast imaging system), see these patent documents. The entire contents of these descriptions are incorporated herein by reference.

If the robot is unfamiliar with the building in which it works, it may be necessary to move the robot around for autonomous navigation or provide the robot with a map of the building (eg, the location of a room or corridor). For example, in a hospital, the robot may need to know the location of each patient's room, nursing station, and so on. In some embodiment, the robot 100 is, for example, FIG.<u style="single">A</u>It is good to receive the layout map 1810 shown in and train the robot 100 to learn the layout map 1810. For example, while guiding the robot 100 around the building, the robot 100 may record a specific location corresponding to the location on the layout map 1810. Robot 100 can display a layout map 1810 on the web pad 310, and when the user takes the robot 100 to a particular location, the user may want to tag that location on the layout map 1810 (eg). , Using the touch screen of the web pad 310 or other pointing device). The user can choose to enter the label of the tagged location, such as the room name or room number. During tagging, Le and example Figure in a state of applying a point Lee out map 1810<u style="single">18B</u>It is good to memorize the tag with the corresponding points applied to the robot map 1820 shown in.

Using the sensor system 400, the robot 100 can create a robot map 1820 when it is moving around. For example, the sensor system 400 can provide information about how far the robot 100 has moved and the direction of movement. Robot Map 1820 may include fixed obstacles in addition to the walls provided in Layout Map 1810. Robot 100 can perform autonomous navigation using Robot Map 1820. In Robot Map 1820, if the "wall" does not appear completely straight, for example due to the detection of packaging material placed along the wall of the corresponding corridor and / or the detection of furniture in various chambers. There is. In addition, there may be rotation and resolution differences between the layout map 1810 and the robot map 1820. After map training, if the user wants to send the robot 100 to a given location, the user can see the label / tag (eg, the location where the label or tag is displayed on the webpad 310). (Can be typed in a text box) or Robot 100 can display the layout map 1810 to the user on the web pad 310, and the user can select its location on the layout map 1810. If the user selects a tagged layout map location, Robot 100 can easily locate the location on Robot Map 1820 that corresponds to the selected location on Layout Map 1810, and then move forward and be selected. You can go to the place where you are.

If the selected location on layout map 1810 is not a tagged location, Robot 100 locates the corresponding location on robot map 1820. In some embodiment, the robot 100 uses the existing tagging location to computer-calculate the scale change size, origin mapping and rotation with the robot map 1820, and then uses the computer-calculated parameters to the robot. Determine the map location (eg, using affine transformations or coordinates).

The robot map 1820 does not have to have the same orientation and scale as the layout map 1810. Further, the layout map does not have to be on scale and may have various distortions depending on the map area. For example, layout maps 1810, created by scanning fire evacuation maps commonly found in hotels, offices and hospitals, are not typically made to scale and have different scales in different areas of the map. In some cases. Robot Map 1820 may have its own distortion. For example, locations on robot map 1820 may be computer calculated by counting wheel rotations as a measure of distance, not if the floor is slightly slippery or causes extra wheel rotations at corners. Due to the exact rotation calculations, the robot 100 may determine the inaccurate location of the mapped object.

The method of mapping a given location 1814 on the layout map 1810 to the corresponding location 1824 on the robot map 1820 is to use the existing tagged location 1812 in the area containing the layout map location (eg, within the threshold radius). It is good to include a step of computer computing the local distortion between the layout map 1810 and the robot map 1820. This method further includes the step of applying the distortion calculation to the layout map location 1814, the purpose of which is to find the corresponding robot map location 1824. The reverse procedure starts at a given location on the robot map 1820, and is performed, for example, if one wants to find a corresponding location on the layout map 1810 to ask the robot its current location. good.

FIG. 18C provides an exemplary flow diagram 1800 of an operation or step in which the robot 100 is operated to navigate the environment using the layout map 1810 and the robot map 1820. With reference to FIGS. 18B and 18C, the operation is the step of receiving the layout map 1810 corresponding to the environment of the robot 100 (1802c) and the step of moving the robot 100 in the environment to the layout map location 1812 on the layout map 1810 (1804c). Steps to record the robot map location 1822 on the robot map 1820 created by the robot 100 along with the environment (1806c), layout with the robot map 1820 using the recorded robot map location 1822 and the corresponding layout map location 1812. It has a step of finding the distortion to and from the map 1810 (1808c) and a step of applying the found distortion to the target layout map location 1814 to find the corresponding target robot map location 1824 (1810c), thus the robot is laid out. You can proceed to the selected location 1814 on map 1810. In some embodiment, the operation uses the existing tagging location to scale, origin mapping, and the robot corresponding to the step to find the rotation between the layout map and the robot map and the selected layout map location 1814. Includes steps to analyze and find map locations. The operation should include a step of analyzing the robot map location by applying it to the scale change size, origin mapping and rotation for which the affine transformation was obtained.

With reference to FIGS. 19A-19C, in some embodiment, this method is tagged with a tagged layout map location 1912 (also known as a recorded layout map location) 1912. Steps to perform a triangular division of the area within the bounding shape that contains the vertices so that all areas of the layout map 1810 are covered by at least one triangle 1910 located at the layout map location 1912 where the vertices are tagged. Including. This method finds the triangle 1910 containing the selected layout map location 1914 and the corresponding triangle 1920 (ie, the same tagged vertices) mapped in the layout map location 1810 and the corresponding triangle 1910 mapped in the robot map 1820. Includes further steps to find the scale, rotation, translation and skew between the robot map triangle with. This method involves applying the obtained scale, rotation, translation and skew to the selected layout map location 1914 to find the corresponding robot map location 1924.

FIG. 19C provides an exemplary flow diagram 1900 of an operation or step for determining the target robot map location 1924. The operation is to find the triangulation between layout maps that demarcate the target layout map location (1902), scale, rotate, and translate between the triangles mapped in the layout map and the corresponding triangles mapped in the robot map. And the step of finding the skew (1904) and applying the scale, rotation, translation and skew found to find the corresponding robot map location to the target layout map location (1906).

With reference to FIGS. 20A and 20B, in another embodiment, the method finds the distance of all tagging points 1912 in the layout map 1810 up to the selected layout map point 1914 and the layout map tagging point 1912. It has a step to find the map of 2012. This method further comprises the step of finding the centroid 2022 of all the tagged points 1922 on the robot map 1820. For each tagged layout map location 1912, this method is required to convert the vector 2014 extending from the layout map centroid 2012 to the selected layout location 1914 to the vector 2024 extending from the robot map centroid 2022 to the robot map location 1924. It has a step of requesting rotation and length scale change. Using this data, the method further comprises the steps of finding the average rotation and scale. For each tagged layout map location 1912, this method further comprises the step of finding the "ideal robot map coordinates" location 1924i by applying centroid transformation, average rotation, and average scale to the selected layout map location 1914. In addition, for each tagged layout map location 1912, this method is tagged by the steps to determine the distance from that layout map location 1912 to the selected layout map location 1914 and these distances, i.e., from the shortest distance to the longest distance. It has a step to classify the attached layout map location 1912. This method includes the step of finding an "influence factor" for each tagged layout map location 1912 using the inverse square of the distance between each tagged layout map location 1912 and the selected layout map location 1914. Next, for each tagged layout map location 1912, this method is based on the difference between the "ideal robot map coordinate" location 1924i and the robot map location 1924, which are proportionally distributed by using the influencing factors of the tagged layout map location 1912. Have a step to find a vector .. This method has a step of summing the proportionally distributed vectors and a step of adding the sum of such vectors to the "ideal robot map coordinates" location 1924i for the selected layout map location 1914. The result is the corresponding robot map location 1924 on the robot map 1820. In some embodiments, this method / algorithm includes only the nearest N tagged layout map locations 1912, rather than all tagged layout map locations 1912.

FIG. 20C provides an exemplary flow diagram 2000 of operations or steps for determining the target robot map location 1924 using layout map 1810 and robot map 1820. The operations are the step of finding the distance between all layout map locations and the target layout map location (2002), the step of finding the centroid of layout map locations (2004), and the centroids of all recorded robot map locations. Step to find (2006) and for each layout map location, ask for rotation and length scale change and convert the vector extending from the layout map centroid to the target layout location into a vector extending from the robot map centroid to the target robot map location (2006). including.

With reference to FIGS. 10A and 21A-21D, in some embodiment the robot 100 (eg, control system 510 shown in FIG. 22) has its local perceived space (eg, sensor field of view 405). It falls into three categories: obstacles (black) 2102, unknown areas (gray) 2104 and known free areas (white) 2106. Obstacle 2102 is located above ground G and below the height of Robot 100<u style="single">Ku</u>Lower than observed (detected) location and ground G<u style="single">Ku</u>Observed locations (eg holes, step-downs, etc.). The known free area 2106 corresponds to the area where the 3-D image sensor 450 can see the ground G. It is good to combine the data from all the sensors in the sensor system 400 into a discretized 3-D voxel grid. Next, it is better to analyze the 3-D grid and convert it to the 2-D grid 2100 by three local perceptual space classifications. FIG. 21A is a schematic diagram of an example of the local perceptual space of the robot 100 when in a stationary state. The information in the 3-D voxel lattice is persistent, but if it is not reinforced, it will collapse over time. When the robot 100 is in motion, the robot has a known free area 2106 that enters for sustainability.

The object detection object avoidance (ODOA) navigation scheme of the control system 500 may include either accepting or rejecting potential robot positions resulting from the command. Potential robot paths 2110 can be generated at many depth levels by various commands, resulting in robot positions at each level. FIG. 21B is an exemplary schematic of the local perceptual space of Robot 100 while in motion. ODOA behavior 600b (Fig. 22) can evaluate each predicted robot path 2110. When these evaluations are used by the action selection engine 580, favorable results and corresponding robot commands can be obtained. For example, for each robot position 2120 in robot path 2110, ODOA behavior 600b can perform methods for object detection and object avoidance, such methods being located within grid 2100 and of robot 100. Find the grid corresponding to this cell for each cell identified as a cell located in the bounding box around the corresponding location, the step to receive the classification of each cell, and each cell classified as an obstacle or unknown area. It has a step of performing a collision check and a step of performing a collision check by determining whether or not this grid portion is located in a collision circle around the location of the robot 100. If the grid points are located within the collision circle, this method further comprises performing a triangular test to see if the grid points are located within the collision triangle (eg, modeling the robot 100 as a triangle). Is good). If the grid points are located within the collision triangle, this method has a step of rejecting the grid points. If the robot positions are located inside the sensor system field of view of the paired grid locations, the "unknown" grid locations are ignored. This is because it is assumed that this will be known by the time the robot 100 reaches these grid points.

This method determines if an obstacle collision is within a robotic path region (eg, modeled as a triangle) between consecutive robot positions 2120s in the robot path 2110. During the transition from robot position 2120 to the next robot position, it is preferable to have a step to prevent robot collision.

FIG. 21C is a schematic representation of the robot 100's local perceptual space and the field of view 405 of the sensor system (control system 510 is a specific sensor for robot routing, such as first and second 3-D image sensors. Only 450a and 450b should be used). Utilizing the holonomic mobility of the drive system 200, the robot 100 utilizes the known sustainability of the ground G, which allows the robot 100 to move in a direction that the field of view 405 of the sensor system does not actually cover. .. For example, if the robot 100 is still sitting with the first and second 3-D image sensors 450a, 450b facing forward, the robot 100 can move sideways, but the control system 510 , Reject the proposed move. This is because Robot 100 does not know what is on its side as shown in the embodiment of FIG. 21C, and FIG. 21C shows an unknown classified area on the side of Robot 100. Is shown. If the robot 100 is moving forward with the first and second 3-D image sensors 450a, 450b facing forward, the next ground G of the robot 100 is classified as a known free area 2106. Is good. This is because both the first 3-D image sensor 450a and the second 3-D image sensor 450b can see the ground G as free when the robot 100 is moving forward, and the classification persists. This is because the sex is still unbroken (see, for example, Figure 21B). In such a situation, the robot 100 can move sideways.

Referring to FIG. 21D, in some embodiments, the ODOA behavior 600b allows the robot to move where the robot is going, given a number of possible trajectories in a state of nonholonomic mobility. You can select the orbit to see (but not now). For example, the robot 100 can predict the orientation of the field of view of a sensor from which the control system 510 can detect an object. Since the robot can rotate while translating, the robot can expand the sensor field of view 405 while advancing.

By understanding the field of view 405 of the sensor system 400 and what the sensor system sees at different positions, the robot 100 can select a motion trajectory that helps the robot see where it is going. .. For example, when turning a turn, the robot 100 can reject a tight turn orbit around the turn. This is because, as shown in Figure 21E, the robot 100 is not the sensor system field of view 405 at the parent robot position 2120, but the robot position 2 where the robot is currently unknown.<u style="single">1</u>Because the end can reach 20. In contrast, the robot 100 first orients in the desired direction of motion, then takes advantage of the holononomic mobility of the drive system 200 to move laterally as shown in Figure 21F, then around the corner. You can select a motion trajectory that moves straight.

Referring to FIG. 22, in some embodiment, the controller 500 executes a control system 510 including a control arbitration system 510a and a behavior system 510b that are in communication with each other. The control arbitration system 510a allows applications 520 to be dynamically added to or removed from control system 510, and each application 520 can control robot 100 without having to know about any other application 520. It will be easier to be able to. In other words, the control arbitration system 510a provides a simple priority control mechanism between application 520 and resource 530 of robot 100. The resource 530 may have a drive system 200, a sensor system 400 and / or any payload or controllable device in communication with the controller 500. The application 520 is better stored in the memory of the robot 100 or has a good communication relationship with the robot 100, thereby operating at the same time (eg, a processor) and at the same time controlling the robot 100. Application 520 can access behavior 600 of behavior system 510b. The independently deployed applications 520 can be dynamically combined at run time to share the robot resources 530 of the robot 100 (eg, drive system 200, arms, head, etc.). A low-level policy that dynamically shares robot resource 530 at run time is enforced within application 520. The policy determines which application 520 controls the robot resource 530 required by that application 520 (eg, the priority hierarchy of the application 520). Application 520 can be started and stopped dynamically, and completely independent of each other<u style="single">Be executed</u>.. Also, the control system 510 allows complex behaviors 600 that can be combined with each other to support each other.

The control arbitration system 510a has one or more resource controllers 540, a robot manager 550 and one or more control arbiters 560. These components do not have to be in a common process or computer, nor do they need to be started in any particular order. The components of the resource controller 540 interface with the control arbitration system 510a for application 520. There is an instance of this component for every application 520. Resource controller 540 extracts and summarizes the complexity of authentication, distributed resource control arbiters, command buffering, and so on. The robot manager 550 emphasizes application 520 prioritization by controlling which application 520 has exclusive control of any of the robot resources 530 at any particular point in time. Since this is the upper central coordinator, there is only one instance of Robot Manager 550 for each robot part. The robot manager 550 executes a priority policy having a linear priority order of the resource controller 540, and constantly monitors the resource control arbiter 560 that enables hardware control. The control arbiter 560 receives commands from all applications 520, issues a single command based on application priority, and publishes it for its associated resource 530. The control arbiter 560 also receives state feedback from its associated resource 530 and sends it back to application 520. The robot resource 530 may be a network of functional modules (eg, actuators, drive systems and groups thereof) with one or more hardware controllers. The command of the control arbiter 560 is specific to the resource 530 in performing a particular action.

The dynamics model 570, which can be run on the controller 500, is configured to computerize the center of gravity (CG), moment of inertia, and cross product of the various parts of the robot 100 to evaluate the current state of the robot. Is good. The dynamics model 570 can also model the shape, weight and / or moment of inertia of these components. In some embodiments, the dynamics model 570 is provided on the robot 100 and is in communication with the controller 500 to calculate the various centroids of the robot 100, the moment of inertia unit 470 (IMU) or a portion thereof ( For example, communicate with an accelerometer and / or a gyro). The dynamics model 570 can be used by the controller 500 along with other programs 520 or behavior 600 to determine the working envelope of the robot 100 and its components.

Each application 520 has an action selection engine 580 and a resource controller 540, one or more behaviors 600 are connected to the action selection engine 580, and one or more action models 590 are action selection engines. It is connected to the 580. The behavior system 510b provides predictive modeling, which allows the behavior 600 to be coordinated with respect to the robot's behavior by evaluating the possible outcomes of the robot's behavior. In some embodiments, the behavior 600 is a hierarchical state-full that combines sensory feedback from multiple sources with a priori limits and information with evaluation feedback on the robot's acceptable behavior. It is a plug-in component that provides an evaluation function. Since behavior 600 is pluggable to application 520 (eg, present inside or outside application 520), removing or adding behavior 600 without modifying application 520 or any other part of control system 510. Can be done. Each behavior 600 is a standalone policy. In order to make the behavior 600 powerful, it is possible to attach the outputs of many behaviors 600 together to the inputs of different behaviors so that they can have complex combination functions. Behavior 600 is designed to execute manageable parts of all jurisdictions of Robot 100.

The action selection engine 580 is a cooperative element of the control system 510, and when all the inputs of the behavior 600 are given, it searches for the optimum action and executes a rapid optimized action selection cycle (prediction / correction cycle). The action selection engine 580 has three stages: nomination, action selection search and completion. At the nomination stage, each behavior 600 is notified and the action selection cycle begins, which is given the cycle start time, current state and robot actuator space limits. Based on internal policies or external inputs, each behavior 600 determines whether it wants to participate in this action selection cycle. During this stage, a list of active behavioral prototypes will be generated, and these inputs will influence the choice of commands to be executed on the robot 100.

In the action selection search stage, the action selection engine 580 produces executable results from the available action space, also known as the action space. The action selection engine 580 uses the action model 590 to simulate the actions of each command in different time steps depending on the planned target time in the future, resulting in a set of executable commands (within limits) and response results. provide. The action selection engine 580 calculates a favorable result based on the result evaluation of the behavior 600, sends a corresponding command to the control arbitration system 510a, and informs the action model 590 of the command selected as feedback.

At the completion stage, the directives corresponding to the collaborative best scored results are combined with each other as global directives, and such global directives are executably provided to the resource controller 540 on the robot resource 530. The best results are provided as feedback to the active behavior 600 and will be used in future evaluation cycles.

A sensor signal received from the sensor system 400 allows an interaction with one or more behaviors 600 to perform an action. For example, using the control system 510, the controller 500 may move from the corresponding action space (eg, a group of possible actions or movements for that particular component) to each robot component (eg, action for a motor or actuator). Or move command) to ensure the coordinated movement of each robot component in an efficient way to avoid collisions with each robot component itself and with objects around the robot 100 known to the robot 100. Let me do it. The controller 500 can issue commands coordinated by a robot network, such as an EtherIO network.

The control system 510 may provide the adaptive speed / acceleration of the drive system 200 to maximize the stability of the robot 100 in different morphologies / positions as the robot 100 is moving around a given area. it can.

In some embodiment, the controller 500 issues a command to the drive system 200, which drives the robot 100 according to the directional and speed set values. One or more behaviors 600 evaluate the expected outcome of executable directives using the signals received from the sensor system 400, and one of these directives is made executable to handle obstacles. It should be chosen (alone or in combination with other commands as an overall robot command). For example, a signal from the proximity sensor 410 allows the control system 510 to change the commanded speed or orientation of the robot 100. For example, the control system 510 can issue a principle command as a result of a signal from the proximity sensor 410 due to the presence of a nearby wall. In another case, the collision signal from the contact sensor resulting from the encounter with the chair allows the control system 510 to issue a command to change direction. In other cases, the speed setting of the robot 100 cannot be reduced in response to the contact sensor, and / or the orientation setting of the robot 100 cannot be changed in response to the proximity sensor 410.

The behavior system 510b includes a mapping behavior 600a that gives rise to the occupancy map 1700 and / or the robot map 1820, a speed behavior 600c configured to adjust the speed setting of the robot 100 (eg, a behavior routine that can be executed on the processor) and It is preferable to have a directional behavior 600d configured to change the directional set value of the robot 100. The velocity and orientation behaviors 600c, 600d can be configured to run simultaneously or independently. For example, the velocity behavior 600c may be configured to poll one of the sensors (eg, a set of proximity sensors 410,420), and the orientation behavior 600d may be configured to poll another sensor (eg, a kinematic bump sensor). ) Should be configured to poll.

With reference to FIG. 23A, in some embodiment, the behavior system 510b includes a human tracking behavior 600e. While performing this behavior 600e, the robot 100 can detect, track, and follow the person 2300. Since the robot 100 can pan and tilt the head 160 using the neck 150, the robot 100 orients the second 3-D image sensor 450b and applies the corresponding field of view 452 to the person 2300. Can be maintained in a normal state. In addition, since the head 160 can move relatively faster than the base 120 (eg, using the drive system 200), the head 160 (and the associated second 3-D image sensor 450b) The person 2300 can be tracked more quickly than when the robot 100 is rotated in place. Robot 100 has a distance of 2300 people D_RYou can proceed towards the person 2300 to keep within (eg, corresponding to the sensor field of view). In some embodiments, the robot 100 turns forward towards the person / user 2300, tracking the person 2300. Robot 100 should use speed and / or waypoint commands to follow human 2300.

Referring to FIG. 23B, the human 2300 sees the second 3-D image sensor 450b as a result of a naive embodiment that follows the human.<u style="single">4</u>Once out of 52, the robot loses track of where the human 2300 is. This embodiment is when a person turns around a corner. To successfully avoid this problem, Robot 100 retains knowledge of the last known location of Person 2300 and its trajectory. Using this knowledge, the robot 100 can use waypoints (or a set of waypoints) to move towards the person 2300 along a corner. Furthermore, when the robot 100 detects a person 2300 moving around a corner, the robot 100 moves forward (holonomically) and / or moves the second 3-D image sensor 450b (for example, the head 160). By panning and / tilting), the field of view 452 of the second 3-D image sensor 450b can be directed to regain the visibility of the person 2300.

With reference to FIGS. 23B and 24A, the control system 510 can identify the person 2300 using the image data received from the second 3-D image sensor 450b (eg, by pattern recognition or image recognition). This allows the person 2300 to continue to follow. If the robot 100 encounters another person 2302, for example when the first person 2300 turns a corner, the robot 100 can identify that the second person 2302 is not the first person 2300. , And continue to follow the first person 2300. In some embodiment, the second 3-D image sensor 450b contains 3-D image data 2402 (eg, pixels in a 2-D array, each pixel containing depth information (eg, distance to the camera). ) Is provided in the segmentor 2404 for segmenting into objects or blobs 2406. For example, pixels are grouped into larger objects based on their proximity to adjacent pixels. These objects (or). Each of the blobs) is then received by a size filter 2408 and further analyzed. The size filter 2408 is either too small (eg, less than about 3 feet high) or too large to be a person (eg,). Process the object or blob 2406 to, for example, the correct size object or blob 2410 by rejecting the object (more than about 8 feet high). The shape filter 2412 receives the correct size object or blob 2410 and is specific. Eliminate objects that do not fill the shape. The shape filter 2412 allows you to see the expected width of where the midpoint of the head is expected to use the visuals of the camera 450b and the known distance to the object. The process converts a properly sized object or blob 2410 into human data 2414 (eg, an image or data representing it).

In some embodiments, Robot 100 can detect and track a large number of people 2300,2302 by maintaining a unique identifier for each person 2300,2302 detected. The human tracking behavior 600e reflects each person's trajectory individually, thereby allowing the robot 100 to track the person's knowledge that the robot 100 should track even if a primary obstruction is caused by another person or object. Can be maintained. Referring to FIG. 24B, in some embodiment examples, a multi-target tracker 2420 (eg, a routine that can be executed on a computer computing processor, eg controller 500), from shape filter 2412 to human data 2414 (eg, image or this). Data representing), gyro data 2416 (eg from IMU470) and odometry data 2418 (received from drive system 200, for example, providing location / speed data 2422, such data being received by human tracking behavior 600e). In some embodiment, the multi-target tracker 2420 uses a Kalman filter to track and reflect each person's motion behavior so that the robot 100 can see the user, for example, around a corner. Tracking can be performed beyond when moving with or when another person temporarily blocks direct visibility to that person.

Referring to FIG. 24C, in some embodiments, the human tracking behavior 600e is the tracking distance D between the robot 100 and the human 2300 while traveling._RFollow people by maintaining. The human tracking behavior 600e should be divided into two sub-components, namely the drive component 2430 and the pan-tilt component 2440. Drive component 2430 (eg, a follow-up distance routine that can be executed on a compute processor) receives human data 2414, human velocity data 2422, and waypoint data 2424 and follows distance D._R(This should be a range) can be found (eg, computer calculated). The drive component 2430 controls how the robot 100 attempts to achieve its purpose depending on the distance to the person 2300. If the robot 100 is within a threshold distance, it uses a direct speed command, which allows the robot 100 to turn towards the person 2300 or retreat from the person 2300 if the person 2300 is too close. Can be separated. If the person 2300 is farther than desired, it is better to use the waypoint command.

The pan-tilt component 2440 causes the neck 150 to pan and / or tilt to keep the field of view 452 of the second 3-D image sensor 450b in contact with the person 2300. The pan / tilt routine 2440 (eg, which can be run on a compute processor) receives human data 2414, gyro data 2416 and kinematic data 2426 (eg from the dynamics model 570 of control system 510) and is second. The 3-D image sensor 450b can determine the pan angle 2442 and tilt angle 2444 directed to keep the field of view 452 in contact with the person 2300. The movement of the base 120 with respect to the pan-tilt of the head 160 may be delayed, and the sensor information reaching the human tracking behavior 600e may also be delayed. This can be compensated for based on the gyro and odometry information 2416,2426, so that once the robot is spinning, the pan angle θ_RWill not overshoot so much.

Referring to FIG. 25A, in some embodiments, the human tracking behavior 600e allows the robot 100 to avoid obstacles 2502 and continue to follow human 2300. Robot 100 can use waypoints to follow human 2300, so ODOA (obstacle detection / obstacle avoidance) behavior even when a person is unable to cross an obstacle 600b<u style="single">To</u>It can be used to find a route to avoid obstacles. The ODOA behavior 600b (Fig. 22) can evaluate the predicted robot path (for example, it can make a reliable evaluation of the predicted robot path that does not cause a collision with the detected object). These assessments can be used by the action selection engine 580, which can determine favorable results and corresponding robot commands (eg, drive commands).

With reference to FIGS. 25B and 25C, in some embodiment, the control system 510 creates a local map 2500 of obstacles 2502 in the area near the robot 100. In a naive or simple system, the robot 100 may not be able to tell the difference between the actual obstacle 2502 and the person 2300 to follow. This usually prevents the robot 100 from moving in the direction of the person 2300. For the person 2300 appears to be an obstacle 2502 in that direction. The human tracking algorithm can continuously report the location of the following human 2300 to the ODOA behavior 600b. Therefore, the ODOA behavior 600b can then update the local map 2500 to remove the obstacle 2502 that previously corresponded to the person 2300 and optionally provide the location of the person 2300.

With reference to FIGS. 26A-26D, in some embodiment examples, the human follow-up behavior 600e evaluates the result of an executable command generated by the action selection engine 580 and has two purposes: 1) the robot 100 The purpose of keeping the person facing the person 2300 as shown in FIG. 26A (for example, the front drive direction F)<u style="single">Man</u>(Purpose of keeping towards 2300 / or panning / or tilting head 160 towards human 2300) and 2) Follow-up between robot 100 and human 2300 shown in Figure 26B Distance D_RAchieve the goal of maintaining (eg, about 2-3 meters). Robot 100 follows human 2300 distance D_RWhen located within the range of, due to the human tracking behavior 600e, the control system 510 has a velocity command (x, y, θ) as shown in FIG. 26C._z) Can be issued to achieve the above-mentioned purpose. If the human 2300 is too close to the robot 100, the robot 100 optionally recedes away from the human 2300, thereby providing a safe distance, eg, a follow-up distance D._R(And the distance that keeps the user within the sensor range (eg 0.5m or more)) can be maintained. The robot 100 should tilt the second 3-D image sensor 450b to respond to the human 2300 coming very close. This is because a person's head may still be in range even if the person's body is not in range. Robot 100 follows distance D_RWhen moving out of, the robot 100 uses the waypoint command to follow the distance D._RCan be obtained again. By using waypoint commands, the robot 100 can determine the optimal path for the robot 100, thus the ODOA behavior 600b can maintain a reasonable distance from nearby obstacles.

With reference to FIGS. 1-3 and 27, in some embodiment, the head 160 supports one or more parts of the interface module 300. The head 160 should have a dock 302 that deleasably accepts one or more computing tablets 310, also known as a web pad or tablet PC, each of which has a touch screen 312. good. The web pad 310 can be directed forward, backward or upward. In some embodiment, the web pad 310 has a touch screen, optional I / O (eg, buttons and / or connectors, such as micro USB, etc.), a processor and memory in communication with the processor. An example web pad 310 is Apple Inc. Apple iPad made by. In some embodiments, the web pad 310 functions as or assists the controller 500 and controls the robot 100. In some embodiments, the dock 302 is mounted firmly attached to the first computing tablet 310a (eg, a wired interface for data transmission at relatively high bandwidth, eg gigabit speeds) and the dock. It has a second computing tablet 310b that is detachably connected. The second web pad 310b can be attached to the first web pad 310a as shown in FIG. 27 or the second web pad 310b is the opposite of the head 160 to the first web pad 310a. It can be attached to the side facing side or the other side. In an additional embodiment, the head 160 supports a single web pad 310, which may be fixedly attached or detachably attached to it. The touch screen 312 can detect, monitor, and / or reproduce where the user touches the touch screen to receive user input and provide a touch-interactive graphical user interface. In some embodiments, the web pad 310 has a touch screen caller that allows the user to find it when it is removed from the robot 100.

In some embodiment, the robot 100 has a number of web pad docks 302 attached to one or more parts of the robot body 110. In the embodiment shown in FIG. 27, the robot 100 optionally has a web pad dock 302 attached to the legs 130 and / or the torso 140. This allows the user, for example, to dock the web pad 310 to the robot 100 at various heights to accommodate users of different heights and to capture images using the web pad 310's camera at different vantage points. Alternatively, a large number of web pads 310 can be attached to the robot 100.

The interface module 300 preferably has a camera 320 attached to the head 160 (see, eg, Figure 2), which can be used to capture footage from a high vantage point of the head 160. Can be (for example, for video conferencing). In the embodiment shown in FIG. 3, the camera 320 is attached to the neck 150. In some embodiments, the camera 320 is only activated when the web pads 310, 310a are removed or undocked from the head 160. When the web pads 310, 310a are attached or docked to the head 160 in the dock 302 (and optionally cover the camera 320), Robot 100 uses the camera of the web pad 310a to capture the footage. Can be done. In such cases, the camera 320 is preferably placed behind the docked web pad 310, which is in operation when the web pad 310 is removed or undocked from the head 160 and the web pad. When the 310 is attached to or docked with the head 160, it becomes inactive.

Robot 100 can be provided via an interface module 300 (eg, using a web pad 310, a camera 320, a microphone 330 and / or a speaker 340) for video conferencing (eg, at 24 fps). Video conferencing should be multi-party. Robot 100 can provide eye contact between both parties in a video conference by maneuvering the head 160 so that it faces the user. Further, the robot 100 preferably has a gaze angle of less than 5 ° (eg, an angle away from the axis perpendicular to the anterior face of the head 160). At least one 3-D image sensor 450 and / or camera 320 mounted on the robot 100 can capture a full-scale image including body language. Controller 500 can synchronize audio and video (eg, with a difference of less than 50 milliseconds). In the embodiment shown in FIGS. 28A-28E, the robot 100 adjusts the height of the web pad 310 and / camera 320 attached to the head 160 (by raising and lowering the torso 140). A video conference can be provided to people standing or sitting by / or by panning and / or tilting the head 160. The camera 320 can move within at least one degree of freedom separately from the web pad 310. In some embodiments, the camera 320 has an objective lens located at a distance of more than 3 feet from the ground but less than 10 percent of the webpad height from the apex of the display area of the webpad 310. .. Further, the robot 100 can zoom the camera 320 to obtain a magnified picture or image around the robot 100. The head 160 preferably has one or more speakers 340 so that audio is emitted from the head 160 near the web pad 310 displaying the video conference.

In some embodiments, the robot 100 can accept user input into the web pad 310 (eg, via a touch screen) as shown in FIG. 28E. In some embodiment, the web pad 310 is a display or monitor, and in other embodiment, the web pad 310 is a tablet computer. The web pad 310 preferably has an easy and intuitive control device that provides high interactivity, such as a touch screen. Webpad 310 can move with at least one degree of freedom 150 square inches (967.7 cm)²) It is preferable to have a monitor display 312 (for example, a touch screen) having a display area equal to or larger than that.

Robot 100 can provide EMR integration in some embodiments by providing video conferencing between doctors and patients and / or other doctors or nurses. Robot 100 should include pass-through consultation instructions. For example, the robot 100 may have a stethoscope configured to convey the listening to a video conferencing user (eg, a doctor). In another embodiment, the robot provides a connector that allows direct connection to Class II medical devices such as electronic stesoscopes, otoscopes (otoscopes) and ultrasound to send medical data to remote users (doctors). Have.

In the embodiment shown in FIG. 28B, the user is for remote control of Robot 100, video conferencing (eg, using the camera and microphone of Webpad 310) and / or use of a software application on Webpad 310. The web pad 310 can be removed from the web pad dock 302 attached to the head 160. Robot 100 is attached to the head 160 to obtain various vantage points for video conferencing, navigation, etc. while the web pad 310 is being removed from the web pad dock 302. It is good to have cameras 320a, 320b.

For interactive applications that are runnable on the computer 500 and / or in communication with the controller 500, it may be necessary to provide the robot 100 with more than one display. Numerous webpads 310 associated with Robot 100 include "FaceTime", Telestration, HD look at this-cam (eg, Webpad 310 with built-in camera). Can provide various combinations (in the case of), can act as a remote operator control unit (OCU) for remote control of the robot 100, and / or can provide a local user interface pad.

In some embodiment, the robot 100 is an intermediary security device 350, also called a bridge, that allows communication between the web pad 310 and the controller 500 (and / or other components of the robot 100) (Figure 27). Has. For example, the bridge 350 can convert the communication of the web pad 310 from a web pad communication protocol to a robot communication protocol (eg, Ethernet with gigabit capacity). The bridge 350 can authenticate that the web pad 310 is not fraudulent and allows communication conversion between the web pad 310 and the controller 500. In some embodiments, the bridge 350 has an authorization chip that authorizes / verifies communication traffic between the web pad 310 and the robot 100. The bridge 350 can inform the controller 500 when it has checked and authorized the web pad 310 that the bridge is trying to communicate with the robot 100. Further, after authorization, the bridge 350 informs the web pad 310 of the communication authorization. The bridge 350 can be attached to the neck 150 or head (or as shown in FIGS. 2 and 3) or anywhere else on the robot 100.

Session Initiation Protocol (SIP) is an IETF-specified signaling protocol that is widely used to control multimedia communication sessions, such as voice and video calls, over the Internet Protocol (IP). This protocol can be used to create, modify, or terminate a two-party (unicast) or multi-party (multicast) session containing one or several media streams. Modifications may include changing addresses or ports, inviting many participants, and adding or removing media streams. Examples of other viable applications include video conferencing, streaming multimedia distribution, instant messaging, presence information, file transmission, and the like. Voice over the Internet Protocol (Voice over IP, or VoIP) is part of a system of methodologies, communication protocols and transmission technologies for the delivery of voice communications and multimedia sessions over the Internet Protocol (IP) networks, such as the Internet. .. Other terms that are often found and often used synonymously with VoIP are IP telephony, Internet telephones, broadband voice (VoBB), broadband telephone technology and broadband telephones.

FIG. 29 provides an exemplary telephone technique that includes dialogue with the bridge 350 to initiate and perform communication via the robot 100. Phone A's SIP calls the SIP application server. SIP activates the VoIP dial function, which sends HTTP post requests to the VoIP web server. The HTTP post request should behave like a call back (callback function) phone function. The SIP application server sends a call to phone A to indicate that the call has started. The VoIP server initiates a call over the PSTN to the callback directory number included in the HTTP post request. The callback phone number ends with the SIP DID provider, which is configured to send a callback to the SIP application server. The SIP application server matches the incoming call with the original phone of phone A and replies both calls with an okay answer. Media sessions are phone A and SIP It is established with the DID provider. Phone A can hear artificial calls made by VoIP. Once VoIP confirms that it has responded to the callback leg, VoIP initiates a PSTN call to the destination, eg, via the bridge 350 to Robot 100. Robot 100 answers the call and the VoIP server bridges the media from the SIP DID provider to the media from Robot 100.

Referring again to FIG. 6, the interface module 300 is attached to a microphone 330 (eg, or a macrophone array) that receives audio input and a robot body 110, and one or more speakers 3 that send audio output.<u style="single">4</u>It is good to have 0. The microphone 330 and the speaker 340 can each communicate with the controller 500. In some embodiments, the interface module 300 has a basket 360 that is well configured to hold brochures, emergency information, household items and other items.

Mobile robots generally need to detect obstacles and hazards to safely navigate around the robot. This is especially important even if the robot has ever been running autonomously, except that human-operated robots also need such detection, because the operator is the robot's robot. This is because they do not always know the details of the surroundings and do not always pay attention to them. Contact detection (eg, a bumper that hits a switch and closes it) can be used as a kind of fail-safe mechanism, but sensors that detect objects from a distance usually need to improve their performance. become. Such sensors include laser range finders, infrared distance sensors, video cameras and depth cameras.

With reference to FIGS. 3-4C and 6, in some embodiments, the robot 100 has a large number of antennas. In the illustrated embodiment, the robot 100 has a first antenna 490a and a second antenna 490b, both of which are attached to the base 120 (provided that these antennas are optional to the robot 100 and others. Parts may be attached to, for example, legs 130, torso 140, neck 150 and / or head 160). By using a large number of antennas, sound signal reception and signal transmission become possible. The use of multiple antennas provides the robot 100 with multiple inputs and multiple outputs, or MIMO, which is the use of multiple antennas for transmitters and / receivers to improve communication performance. .. MIMO results in a significant increase in data throughput and link range without additional bandwidth or transmit power. MIMO is high<u style="single">Speckle</u>This is achieved by performance (higher number of bits per second per hertz of bandwidth) and link reliability or versatility (reduced fading). In view of these characteristics, MIMO is an important part of the latest wireless communication standards such as IEEE 802.11n (WiFi), 4G, 3GPP Long Term Evolution, WiMAX and HSPA +. In addition, Robot 100 can act as a WiFi bridge, hub or hotspot for other nearby electronics. The mobility and use of MIMO in Robot 100 allows the robot to be a relatively reliable WiFi bridge.

MIMO can be subdivided into three main categories: pre-coding, spatial multiplexing or SM and diversified coding diversity coding. Recording is a type of multi-stream beam shaping and is considered to be all spatial processing that occurs at the transmitter. In beam forming (single layer beam forming), the same signal is emitted from each of the transmitter antennas with the appropriate phase (and possibly gain) weighted so that the signal output is maximized at the receiver input. To. The advantage of beam shaping is that it increases the gain of the received signal and reduces the multipath fading effect by making the signals emitted from different antennas intensify each other. In the absence of scattering, beam formation results in a well-defined directional pattern. If the receiver has a large number of antennas, the transmitting beam formation cannot maximize the signal levels at all of the receiving antennas at the same time, so it is better to use recording with a large number of streams. .. Recording may require knowledge of Channel State Information (CSI) at the transmitter.

Spatial multiplexing requires a MIMO antenna configuration. Spatial multiplexing divides a high-rate signal into a number of low-rate streams, each stream being transmitted from a different transmitting antenna on the same frequency channel. If these signals reach the receiver antenna array with sufficiently different spatial signatures, the receiver can separate these streams into parallel (nearly parallel) channels. Spatial multiplexing is very powerful, increasing channel capacity with high signal-to-noise ratio (SNR).<u style="single">Na</u>It is a technology. The maximum number of spatial streams is limited by the reduction in the number of antennas at the transmitter or receiver. Spatial multiplexing can be used with or without knowledge of transmission channels. Spatial multiplexing can also be used for simultaneous transmission to multiple receivers, called spatial access multiple access. Good separability can be ensured by scheduling receivers with different spatial signatures.

Diversity coding techniques are best used when there is no knowledge of the channel at the transmitter. The diversity method sends a single stream (unlike many streams of spatial multiplexing), but is coded using a technique called spatiotemporal coding. The signal is emitted from each of the transmitting antennas with a fully or nearly orthogonal coding. Diversity coding utilizes independent fading on a large number of antenna links to increase signal diversity. Since there is no knowledge of channels, no beam shaping is done or there is no array gain from diversity coding. Spatial multiplexing can also be combined with programming if the channel is known at the transmitter, or with diversity coding if there is a trade-off in decoding reliability.

Some embodiment have a third antenna 490c and / or a fourth antenna 490d provided on the torso 140 and / or head 160, respectively (see, eg, FIG. 3). .. In such a case, the controller 500 can determine an antenna arrangement configuration that achieves a threshold signal level for robust communication (eg, raising and lowering the torso 140 and / or rotating and / or tilting the head 160). By letting, for example, by moving the antennas 490a-490d). For example, the controller 500 can raise the third and fourth antennas 490c, 490d by issuing a command to raise the height of the fuselage 140. Further, the controller 500 can issue a command to rotate and / or tilt the head 160 to further orient the fourth antenna 490d with respect to the other antennas 490a-490c.

FIG. 30 is a schematic representation of an exemplary dwelling 3010 equipped with an object detection system 3000. The object detection system 3000 is provided in various rooms 3020 (inside and / or outside), corridors 3030 and / or other parts of the dwelling 3010, and is a home monitoring, person monitoring and / or person emergency response system ( It has an imaging sensor 3070 for PERS), such as a 3-D speckle camera 1300 and / or a 3-D TOF camera 1500. The imaging sensor 3070 preferably has a field of view 3075 that covers the entire room 3020 (eg, scene 10) or a portion thereof. The object detection system 3000 can provide hands-free real-time behavior monitoring (eg, for the elderly), directing the caregiver's attention to the elderly looking for an independent living room. In the illustrated embodiment, the imaging sensor 3070 is located on one or more walls or in one or more corners of room 3020 where human activity can be most effectively monitored. .. For example, the imaging sensor 3070 may be located in a corner of a room or on a wall opposite the doorway, a sofa, a major seating area, and the like. A person's activity and rest patterns can be monitored, tracked, and memorized (eg, for comparison with future patterns) or relayed to a caregiver.

The object detection system 3000 preferably has a base station BS that is in communication with the imaging sensor 3070 (eg, by electrical connection or wirelessly). The base station BS preferably has a processor that performs image recognition and / or monitoring routines and a memory that stores sensor data, such as 3-D maps, at certain time intervals. In some embodiments, the object detection system 3000 has a mobile robot 100 in communication with the base station BS and / or the imaging sensor 3070.

In some embodiment, the image sensor 3070 forms a 3-D map of the corresponding room 3020 in which the imaging sensor 3070 is provided for object recognition and / or object tracking. Further, in the embodiment including the mobile robot 100 having at least one imaging sensor 450, the mobile robot 100 is the same room 3020 or 3-D of object 12 for comparison with the 3-D map of the imaging sensor 3070. D can be created. Base station BS can receive both 3-D maps, which should be from different vantage points or views, and compare the two 3-D maps to object 12 (eg, for example). Furniture, people, pets, etc.) and the location of the shield 16 can be recognized and analyzed.

In some embodiment, the object detection system 3000 is an imaging sensor 3070 to obtain depth perception of scene 10 (eg, 3-D map) to recognize gestures caused by a part or whole body of a person. Is used. The imaging sensor 3070 can identify position information about a body or part of the body, including depth position information about different parts of the body. The imaging sensor 3070 can create a depth image that includes the position information of the entire scene 10 for a person that includes the position information of the body part of interest. The imaging sensor 3070 and / or the base station BS (eg, having a processor that executes a command or routine) can segment the body part of interest from the background and other objects in the depth image, and one. The shape and position of the body part of interest (eg, statically) can be determined with a particular sensation. Dynamic determination can be performed as the body or body part moves within scene 10. In addition, the imaging sensor 3070 and / or base station BS can identify gestures that are dynamically generated by the body part over a given duration if the movement of the body part is of interest. Classify the identified body gestures, and events occur based on the classification. Additional details and features relating to gesture recognition that can be combined with the details and features described herein are found in U.S. Pat. No. 7,340,077 (Gesture Recognition System Using Depth Perceptive Sensors). Is cited by reference and the entire description thereof is incorporated herein by reference.

The imaging sensor 3070 can be used for human gesture recognition within each room 3020. For example, the imaging sensor 3070 can create a 3-D map of a person, which is a gesture or posture of the person (eg, sitting, sleeping, walking, etc.) It can be used to analyze or classify a fallen state, a pointing state, etc.). Imaging sensor 3070, base station BS and / robot 100 can use image data to form a 3-D map of a person and perform routines to identify and / or classify human gestures. Certain gestures, such as fallen gestures or waving gestures, can trigger an emergency trigger by the object detection system 3000. In response to an emergency event trigger, the base station BS can send emergency communications (eg, telephone calls, emails, etc.) for emergency assistance and / or the robot 100 can locate a person. It can identify and confirm gesture recognition and / or provide assistance (eg, sending a prescription and helping a person get out of the floor).

FIG. 31 provides an exemplary flow diagram 3100 of an operation or step in which the object detection system 3000 is activated. The operation is step (3102) of receiving image data of scene 10 (eg, room 3020) (eg from 3-D speckle camera 1300 and / or 3-D TOF camera 1500), target object in scene 10. Includes step (3104) to create 12 3-D maps. The operation further includes a step of classifying the target object 12 (eg, as a person) (3106) and a step of analyzing the state, posture or gesture of the target object 12 (3108). The operation includes a step (3110) that causes an object event (eg, an event specific to the analyzed state, posture and / or gesture) and optionally a step (3112) that responds to the object event.

In some embodiment, the step (3106) of classifying the target object 12 is the object type of the target object 12 (eg, person, dog, cat, chair, sofa, bed, etc.) and the type of living object. Includes a step of determining (eg, a person), a step of determining the state of the object 12, eg, whether the object 12 is alive, sleeping, injured, etc. (eg, detecting motion). ..

In some embodiment, the step (3108) of analyzing the state, posture and / or gesture of the target object 12 is whether the target object 12 is alive, sitting, sleeping or waving. Includes a step to determine if it is falling or not. The determination of state, posture and / or gesture can be performed separately, continuously, simultaneously or in combination thereof. For example, when detecting and analyzing a falling or waving gesture (eg, a falling and waving gesture), the operation is performed on the posture of the target object 12, such as the fallen posture (eg, on the floor). It may further include steps to determine (eg, alive, breathing, injured, impaired, unconscious, etc.) and condition (lying posture).

When a fallen target object 12 (eg, a person has fallen), for example, the operation recognizes the step (3106) of classifying the target object 12 as a person and, for example, a known feature memorable in memory. It is preferable to include a step (3108) of analyzing the state, posture and / or gesture of. Illustrated memorized traits are the gesture that a doll in a tattered kimono collapses or falls forward when unconscious, the gesture posture that falls forward, the left side if the left hip is weak, the use of a cane or walking aid. Despite being there, it includes things like falling down. Further, the operation may include a step of learning the state, posture and / or one end of the gesture of the specific target object 12, a step of filtering the sensor data and / or a step of executing a search algorithm on the sensor data.

In the case of a waving gesture, the operation may include a step of generating a waving event and an event in response to this event, and a step of sending the robot 100 to the target object 12 for assistance. For example, the robot 100 may have communication capabilities (eg, wireless, telephone, teleconference, email, etc.), drug dispenser or target object 12, eg, some other article that a person wants to approach. In addition, the robot 100 can take a person to the base station BS, which is installed on a large screen (on a TV), stable audio equipment (BS that does not generate motor noise and mechanical noise). It is preferable to have a television or display for a video conferencing session that utilizes a microphone array), stable AC power, stable cameras and wired broadband access.

In the case of lying and / or sitting posture, the operation is sent to the target object 12 to confirm the health event and the step to generate the event in response to this event, the posture of the robot 100, the vital signs, etc. It is good to include steps. In the case of a fallen gesture or fallen posture, the operation is a step to generate an emergency event and an event in response to this event, a step to send the robot 100 to the target object 12 for assistance, and an emergency for emergency assistance. It is good to include a step to put out a means of time transmission.

In some embodiments, the operation involves creating or updating the occupancy map 1700 and comparing the location of the current object with the location of the past object for object tracking. Gesture events should be generated for the moving object 12, and in response to this event the operation should include sending the robot 100 to the corresponding room 3020 to locate the object.

Robot 100 is an object detection system 30<u style="single">0</u>It is good to operate autonomously from 0 and maintain communication with this. Therefore, the robot 100 is an object detection system 30.<u style="single">0</u>It can receive 0 occurrence events and respond to these events. For example, instead of the base station BS issuing a command to the robot 100, for example, in order to confirm the classification of gestures, the robot 100 uses the object detection system 30.<u style="single">0</u>It is good to listen to the event that occurred for 0, and optionally deal with the event that occurred.

With reference to FIG. 32, in some embodiment it is better for a person to hang the pendant 3200, which is the object detection system 30.<u style="single">0</u>Communicating with 0 (eg, via base station BS and / or image sensor 3070). The pendant 3200 preferably has a motion sensor 3210 (eg, one or more accelerometers, a gyroscope, etc.) that detects the user's motion. Object detection system 30<u style="single">0</u>0 can track the movement of the pendant throughout the dwelling. For example, the pendant may include user movement data such as steps, sitting or lying down and / or current state (eg, standing, walking, etc.).<u style="single">Straight</u>It is possible to provide a sitting state, a lying state, etc.). The base station BS can receive the user motion data and generate an event accordingly.

For example, upon receiving the current state of the lying pendant 3200, the object detection system 30<u style="single">0</u>0 can generate a health status confirmation event and an event in response to this event, the robot 100 locates the person wearing the pendant 3200, and confirms the person's current health status. be able to. For example, upon locating a person, the robot 100 captures the person's image data, confirms chest movements as an indicator of whether the person is breathing, and uses infrared imaging to identify the person. It is possible to measure a person's body temperature and the like. Robot 100 can ask audible questions (eg, "How are you?") And wait for an audible response from that person. If the robot 100 does not detect a audible response, the robot 100 makes a communication request for emergency assistance in the detection system 30.<u style="single">0</u>It is good to generate an emergency event that can be received by 0 base station BS. Further, the image sensor 450 of the robot 100 may capture an image or a live image for communication with a third party (for example, a caregiver, an emergency response office, etc.).

In some embodiment, the pendant 3200 has a support button 3220 that the user can press to request help. The base station BS can receive the support request signal from the pendant 3200 and transmit the support request to the caregiver, the emergency response business operator, and the like. Further, in response to the reception of the support request signal, the base station BS can generate a support request event, and the robot 100 in communication with the base station BS can receive this and respond accordingly. it can. For example, Robot 100 locates a person wearing a pendant 3200, confirms the person's condition, and / or provides assistance (eg, providing medicine, providing video conferencing, telephone service). Can provide, help the person get up, etc.).

Referring to FIG. 33A, in some embodiment, the mobile robot system 3301 has a mobile robot 100 and a remote computer computing device 310r, eg, a web pad 310, with a touch screen 312 in communication with the robot 100. The remote computer computer 310r is a software application 3300 (eg, a tablet) that allows a remotely located user to visualize the environment or scene 10 around the robot 100 and to remotely control the robot 100. Useful UI component / application) is executed. Software application 3300 for example Apple iPad 2 and / or Motorola for Webpad 310 Hardware such as the PrimeSensor camera (available from PrimeSensor, Inc. 28, Tel Aviv 69710, Israel) or any suitable hardware can be used for the Xoom and image sensor 450. For example, with respect to software, the software application 3300 can use Apple iOS 4.x, Android 3.0 (also known as Honeycomb), OpenGL ES 2.0 or any other suitable operating system or program.

Referring to FIG. 33B, in some embodiment, the software application 3300 displays a user interface 3302 with map view 3304 (eg, provided on a remote computer computer 310r), and the user interface 3302 , Display a 2-D top-down layout map 1810 of scene 10 or area around robot 100. The user can select location 1814 on the layout map 1810 and send the robot 100 to the selected location 1814 (eg, by selecting the displayed button or shape). Robot 100 determines the corresponding target robot map location 1824 on robot map 1820 (eg, based on sensor data from sensor system 400), thus robot 100 determines the selected location 1814 on layout map 1810. You can proceed to (see also Figures 18A-18C). Robot 100 is a location selected by performing one or more behaviors, such as mapping behavior 600a, obstacle detection and obstacle avoidance behavior 600b, velocity behavior 600c and / or directional behavior 600d (FIG. 22). Proceed autonomously until 1814. Once the robot 100 reaches the selected location 1814, the user may want to see the scene 10 around the robot 100 and / or steer the robot 100 around its local scene 10. (For example, you may want to move the robot 100 remotely).

With reference to FIGS. 33C and 33D, software application 3300 can display 3-D map 3310 in map view 3304. The user can make the 3-D map 3310 appear by performing a specific hand gesture (eg, multi-fin gas wipe, tap, etc.) on the touch screen 312 of the remote computer computer 310r. In some embodiments, the user views sensor data from the robot 100's sensor system 400 or live camera view (eg, a first person's image from one or more cameras 320,450 attached to the robot 100). A 3-D map 3310 showing the 3-D scene 10a can be selected based on the view). The live camera view may include the anterior edge or anterior portion of the robot 100, so a remotely located user steers the robot without advancing to an obstacle located in the immediate vicinity of the anterior edge of the base 120. You will be able to.

In some embodiments, the 3-D scene 10a is preferably based on the point cloud 3320 captured by the 3-D imaging sensors 450, 450a, 450b. In the illustrated embodiment, the points 3322 of the point cloud 3320 are grouped together to form a three-dimensional image of the 3-D scene 10a. The software application 3300 can also place an animation or computer-generated model 100a of the robot 100 within the 3-D scene 10a and show the position and / or kinematic state of the robot 100. The robot model 100a should be an accurate description of the actual robot 100 and should be based on the computer-aided design (CAD) model of the robot 100. In the embodiment shown in Figure 33C, the 3-D map 3110 floats on the 2-D layout map 1810 and provides a quick view of the actual scene 10 around the robot 100 at its current location. .. In the embodiment shown in FIG. 33D, the user can extend the 3-D map 3310 to full screen and near full screen view. In addition, the software application 3300 can display a control portion 3306 for operating the robot 100 (eg, for driving the robot 100 and / or changing its pose). The user can see the 3-D map 3310 from different perspectives or eye positions 3350 (Fig. 33G) with respect to the robot 100 (eg, one or more hands on the touch screen 312 with the 3-D map 3310). By rotating with a gesture). Figures 33C-33G provide 3-D maps 3310 with different views.

With reference to FIGS. 33A-33D again, the user can command the robot and drive the robot 100 using the 2-D layout map 1810 and / or the 3-D map 3310 (eg, the destination of the robot). By identifying). The 2-D layout map 1810 allows the user to move the robot 100 around a large area, but may not be possible for the user to do so. For example, a 2-D layout map typically shows the location of rooms, walls, corridors, etc., but cannot show furniture such as hospital beds or conference room tables. The remote user commands the robot 100 and the robot autonomously advances into the room and just stops by the side of the hospital bed or commands the robot 100 at a predetermined spot in the room. Allows it to be stopped, but once the robot 100 is in the room, the user may need to make the next move, for example, to a particular spot at the bedside. In the illustrated embodiment, the software application 3300 is from a spatial volume 3330 located adjacent to the robot 100 (eg, including a floor plane 3332 in the direction of movement of the robot 100) to a volume point cloud 3320 (eg, from an imaging sensor 450). A 3-D map 3310 in the form of a 3D depth image based on or in the form of a 3D depth image as a volume point cloud 3320 based on (reflected light from speckle emission of light) is provided in map view 3304.

In some embodiment, the imaging sensors 450, 450a, 450b attached to the robot 100 are volume point group imaging devices. The imaging sensor 450 should be supported above the drive system 200 at a height of about 1 foot or more above the ground and on the floor plane 33 in the direction of movement of the robot 100.<u style="single">3</u>Space volume 3330 including 2 to point cloud 33<u style="single">2</u>It is good to be sent to get 0. The controller 500 can receive the point cloud signal from the 3-D imaging sensor 450 and can issue a drive command to the drive system 200 based on the point cloud signal received at least partially. At least one 3-D image sensor 450,450a, 450b can emit light to the scene 10 around the robot 100, and can capture the image of the scene 10 along, for example, the driving direction F of the robot 100. it can. The image should include at least one of (a) three-dimensional depth image, (b) active lighting image and (c) ambient lighting image. Illustrated point cloud 33<u style="single">2</u>0 is an example of a three-dimensional depth image of the room scene 10a perceived by the robot 100. 3-D map view 33<u style="single">04</u>May include at least one of (a) three-dimensional depth image, (b) active lighting image and (c) ambient lighting image. In the illustrated embodiment, the software application 3300 has a spatial volume 33 visible to the 3-D image sensors 450, 450a, 450b in scene 10.<u style="single">3</u>Display 3-D map 3310 as a 3D depth image of 0. Some or all points 3322 of the point cloud 3320 should be colored with the colors from the collogate RGB camera 320. In some embodiments, the point cloud 3320 has 320 x 240 points (76,), each with x, y, z coordinates and RGB colors.<u style="single">8</u>00 points) is included.

When interacting with people from a remote location, the user depends on what he sees and hears from the audio / video supply from Robot 100. The typical field of view of a video camera is limited (eg, 60 °) and may not be suitable for providing "situational awareness". Robot 100 can use multiple cameras 320,450,450a, 450b to obtain a wide combined field of view and / or scan one or more of cameras 320,450,450a, 450b to effectively scan the corresponding field of view 452. Can be expanded to. Further, the robot 100 can execute the scene update routine by rotating one or more cameras 320 or the imaging sensor 450 to obtain a combined field of view 452 including the entire 360 ° around the robot 100.

In some embodiment, the robot 100 periodically or when one or more events occur, the robot 100 moves the imaging sensors 450, 450b, 450c to a third dimension 360 ° around the vertical center axis Z. Execute a scene update routine that captures the original depth image data 360. The robot 100 can rotate 360 ° in a nonholonomic manner at a fixed position, rotate the head 160 360 °, and the like. In some embodiments, with respect to a field of view 452 of about 60 °, the robot 100 places the image sensors 450, 450b, 450c in the 0 ° first position, the 60 ° second position and the 120 ° third position mutual. Move in each forward drive direction F. The software application 3300 can provide the 3-D scene 10a of the actual scene 10 based on the captured 3-D depth image data.

FIG. 34 is an exemplary schematic of Architecture 3400 for Software Application 3300. Software application 3300 includes robot data model 3420 (eg graphics data file or programming object), point group data model 3430 (eg graphics data file or programming object) based on imaging sensors from imaging sensors 450, 450a, 450b and Includes an operating system hand gesture application program interface (API) 3440 with touch recognition and an application engine 3410 in communication. Application engine 3410 is rendering engine 3<u style="single">4</u>Communicating with 50, this rendering engine communicates with OpenGL ES 3460. The Open GL ES3460 can communicate with the central processing unit (CPU) 3470 and / or the graphics processing unit (GPU) 3480 of the remote computer computing unit 310r.

The application engine 3410 provides a user interface 3302 for rendering by the rendering engine 3450. With respect to the 3-D map 3310, the application engine 3410 provides the robot model 100a using the robot data model 3420 (eg, a matrix mass for manipulating the robot model 100a in 3-D space) and both are single. A 3-D scene 10a can be provided using the point cloud data model 3430 (eg, faceted projection matrix) for rendering within the view of. The application engine 3410 can use the hand gesture API 3440 to determine touch gestures (eg, touch location coordinates, gesture path, gesture duration, etc.) on the touch screen 312 of the remote computer computer 310r.

The application engine 3410 uses the robot data model 34 to determine the attitude of the robot model 100a.<u style="single">2</u>You can access 0. Robot data model 34<u style="single">2</u>0 is the four degrees of freedom of the robot 100 (eg, panning / tilting the head 160, torso height H_TAnd the rotation of the robot 100) can be explained. Robot data model 34<u style="single">2</u>0 allows the robot model 100a to be manipulated graphically with four degrees of freedom, for example, the posture of the robot model 100a can be moved or changed within the 3-D map 3310. In addition, the application engine 3410 is used in ambient occlusion, or 3-D computer graphics, to add realism to the Phong reflection model by taking into account the attenuation of light due to shading. Shading methods can be implemented. Environmental projection seeks to approximate how light radiates in the real world, especially from objects that are usually considered non-reflective surfaces. Unlike the local method, like Phong shading, ambient shading is a global method that means that the illumination at each point is a function of other geometric shapes in the scene. The soft appearance achieved by ambient shielding alone is similar to how an object looks on a cloudy day. In some embodiments, the application engine 3410 is a robot data model 34.<u style="single">2</u>A low-pass filter is applied to 0 to remove the low-reflected polygons on the robot model 100a.

In some embodiment, the application engine 3410 has the following parameters: robot model coordinates (x, y, z), point cloud coordinates (x, y, z), eye position 3350 (x, y, z). ), Eye target 3352 (ie, line-of-sight information), light source coordinates (x, y, z) and / or viewport size (ie, the display area of the touch screen 312), one or more of the rendering engine 3450 Tell to. The rendering engine 3450 can communicate with OpenGL to display graphics on the touch screen 312. In some embodiments, the rendering engine 3450 performs a rendering routine and / or CPU3470 and / or GPU3480 to calculate rotation or translation of robot model 100a, changes in eye position 3350 and / or eye target 3352, etc. Communicate with. The robot model 100a should have 15 rendered parts that can be touched and manipulated separately by the user.

Seeing Figure 33D again, Robot 100 is capable of autonomous navigation, but once a remote user begins a dialogue with people, a distant user needs full control of Robot 100. In some cases. In addition to typical robot movements such as translation and rotation, the remote user has a robot shoulder height H_T(Fig. 2) and the pan / tilt position of the head 160 can be adjusted. A simple slider or dial on the user interface (UI) 3302 allows adjustment of the robot's kinematic state, while traditional UI widgets provide a good visualization of the robot's current kinematic state. Can't bring. The user interface 3302 can display a directional pad 3308 (D-pad) for driving the robot 100 in the control portion 3306. For example, the directional pad 3308 may have left and right action buttons or controls 3308a, 3308b corresponding to the nonholonomic XY drive command. The directional pad 3308 may be identical or similar to a video game console game pad, game controller or remote control unit for television. The combinatorial gestures on the left and right buttons or controls 3308a, 3308b can be analyzed with combinatorial motion commands such as up and left as diagonal motion directions.

In some embodiments, the software application can provide a virtual floating and invisible D-pad 3308 and / or mouse look, so what if the user's left thumb hits the touch screen 312. Even so, the D-Pad 3308 floats and / or the mouse look floats in any case when the user's right thumb hits the touch screen 312. Both or one of these features can be reset when the user lifts his or her thumb.

Also referring to FIGS. 33B-33D, the user interface 3302 (eg, view) displayed by the software application 3300 is entered by the user, for example, by gesture recognition on the touch screen 312 of the web pad 310 in communication with the robot 100. To receive. Illustrative gestures include tapping (tapping any number of times), pinching in and pinching out (to zoom the view), panning or dragging, swiping (in any direction), and rotating. (Move the fingers in opposite directions) and / or long press (also called "touch and hold"), but are not limited to these. The software application 3300 can recognize gestures on the touch screen 312 and can determine the corresponding commanded motion for the robot 100 (eg, operate or command the robot model 100a). sometimes). It is good to transmit the robot command from the web pad 310 to the robot 100 in the actual space, and then the point cloud 3 as an option.<u style="single">32</u>At 0, it can be reproduced in the mode view of a third person (for example, indicating that the robot 100 is moving according to a command) and back-propagated in the indicated state. For example, the application engine 3410 using the touch gesture API 3440 can recognize the received touch gesture and respond accordingly by, for example, generating a touch event that can be handled by the touch routine.

The software application 3300 receives a touch event and / or performs a touch recognition function (using the touch gesture API 3440), thereby causing its location (or multi-finger touch location), path, duration and / or other. You can check the touch recognition data. The touch location on the touch screen 312 can be projected directly onto the corresponding location on the 2-D layout map 1810, but the 3-D map<u style="single">33</u>For 10, the software application projects the 2-D touch gesture along the truncated octahedron projection as shown in Figure 35. To determine the touch location within the 3-D scene 10a displayed on the 2-D touch screen 312, the software application 3300 applies the touch location 3510 on the touch screen 312 to the touch location 3510 along the frustum projection 3500. It can be projected onto the corresponding intersection 3520 in 3-D scene 10a.

The robot 100 can be guided in a third party mode in various ways. In one technique, the operator is in the displayed 3-D scene 10a.<u style="single">As a touch location 3510</u>Select a floor location (eg, select a location on the touch screen 312 of the web pad 310 using a pointing device or communicating with the robot 100). Software application 3300 selected<u style="single">As a touch location 3510</u>Project the floor application onto the 3-D scene 10a along the truncated octahedron 3500 to find the corresponding selected scene point 3520 in the 3-D scene 10a. The software application 3300 can communicate to the scene point 3520 (eg, as actual spatial coordinates) to the controller 500, which issues a drive command to identify the robot 100 along the identified robot path 2110 (FIG. 21B). You can move to the selected location by avoiding obstacle 12.

The software application 3300 and / or controller 500 is preferably running one or more physical simulators (after the collision has occurred in the simulated model) for post-collision detection. In post-collision detection, software application 3300 and / or controller 500 advances the physical simulator by one time step, and then whether object 12 intersects or is believed to intersect object 12. Check if it is within the distance. In each simulation step, a list of all intersecting objects 12 is made and the positions and trajectories of these objects are somewhat "fixed" to describe the collision. In post-simulation, collisions are detected after this actually happens in the simulation. The collision detection algorithm does not need to know the myriad of physical variables, a simple list of physical objects is often sent to the algorithm, and the program returns to the list of intersecting objects. The collision detection algorithm does not need to understand friction, elastic collisions, inelastic collisions and deformable bodies. After identifying the collision-free robot path 2110 in the simulator, the robot 100 can proceed through scene 10 in real space. See Figure 33E. Software application 3300 received destination location L₂And destination location L₂Current location of robot 100 up to L₁From destination location L₂It is possible to use the projection path in the virtual space of the scene up to. The point cloud 3320 and / or the known floor plane 3332 can be a virtual surface for collision detection. Robot path 3350 in virtual space Current location of robot 100 L₁Destination location selected from L₂Can be projected up to, and finally the robot path hits the virtual surface. The software application can analyze the drive command for the robot path 3350 to the virtual plane.

Referring to FIGS. 33D and 35, in order to operate the robot model 100a on the touch screen 312, the user touches the torso 140 of the robot model 100a, moves it vertically, and releases it to the torso shoulder height. H_TCan be changed. Software application 3300 touch<u style="single">place</u>The 3510 can be projected onto the screen 312 along the frustum projection 3500, whereby the corresponding touch points 3520 within the 3-D scene 10a can be determined, such touch points in this embodiment. It intersects the body 140 of the robot model 100a. Software application 3300 touch gesture API3<u style="single">4</u>Use 40 to determine the characteristics of the touch gesture received, thereby sending a command in this case torso shoulder height H_TRespond accordingly by changing. The user can touch the other part of the body 140 (or robot model 100a) of the robot model 100a on the touch screen 312, move it horizontally, and release it to rotate the robot 100. The user can touch the front bumper 124a (FIG. 4A) of the robot model 100a, hold the front bumper 124a, and move the robot backwards. To move the robot 100 forward, the user touches and holds one of the rear bumpers 124b, 124c (Fig. 4A) of the robot model 100a and moves the robot forward in scene view 3304. Is good. A quick tap on one of the three bumpers 124a-124c allows the robot to move a short distance (eg 10 cm). The user can pan the head 160 of the robot 100 by touching the head 160 of the robot model 100a, moving it horizontally, and then releasing the head 160. Similarly, in order to tilt the head 160, the user should touch the head 160 of the robot model 100a and move it vertically.

In some embodiments, the user can make multiple adjustments simultaneously by synthetic or combined movements (eg, panning and tilting the head 160 at the same time). To glue the local waypoint command, the user should touch the base 120 of the robot model 100a, move it in any direction, and release it. The release location can be projected onto the ground plane 3332. Other recognizable gestures include quickly tapping and / or touching and moving the head 160 of the robot model 100a to update the 3-D scene 10a based on new sensor data (in any direction). To set the head pan / tilt direction by releasing the head 160 of the robot model 100a. The release location can be projected onto a point in 3-D space that is sufficiently close to the effective point in the 3-D scene 10a (ie, the point in the point cloud 3320).

With reference to FIG. 36A, the software application 3300 can project a third perspective view onto the shoulder view (eg, third party shooter view) of model 100a of robot 100 in the 3-D scene 10a. The eye position 3350 should have an initial default position behind the robot model 100a and, optionally, be lifted for perspective. The robot model 100a should be shown in the current posture of the robot 100 and the current orientation of the robot 100 with respect to the visible spatial volume 3330 around the robot. In addition, the robot model 100a and 3-D map 3310 are preferably shown from the operator's point of view and / or other point of view of the robot 100, examples of which are shown in FIGS. 33E-33G, 36A and 36B. Has been done.

See Figures 36A-36C, the Freelook Gesture Directive (also known as the Mouse Look) rotates the 3-D map 3310 or moves the view perspective of the robot model 100a and 3-D scene 10a in a different way. Provides functionality (eg, through gesture recognition). In addition, the freelook command allows the user to rotate the robot model 100a nonholonomically and / or pan or tilt the cameras 320,450. For scene viewing, the user can touch the robot model 100a, move it in any direction, and release it into open space to change the eye position 3350 in the 3-D scene 10a. The effect of this is to leave the eye target position 3352 invariant so that the robot model in 3-D scene 10a<u style="single">100a</u>The eye target position 3352 can be set to the robot start position by default. In the embodiment shown in FIGS. 36A and 36B, the user rotates the 3-D map 3310 to visually recognize the robot model 100a and the 3-D scene 10a from the side with respect to the robot model 100a. The robot model 100a can move away from its starting position within the 3-D scene 10a. Furthermore, the eye target position 3352 does not have to follow the robot 100. By tapping the body 140 of the robot model 100a twice, the eye target position 3352 can be changed to the current position of the robot.

Referring to FIG. 36C, the software application 3300 should project the line of sight 3354 from the eye position 3350 to the eye target 3352 in order to define the viewing area of the 3-D scene 10a. In the illustrated embodiment, the software application 3300 uses the virtual position of the touch screen 312 with respect to the actual scene 10 and then initially places the line-of-sight projection image 3354 in the eye position 3350 (eg, the default position on the user's shoulders). (Placed) can be placed via screen point 3356 on the touch screen 312 to determine the eye target position 3352 within the 3-D scene 10a. The software application 3300 can determine the size of the 3-D scene 10a rendered using the received display area of the touch screen 312. The initial eye position 3350 should be located 5 meters behind the robot 100 and elevated along the line of sight 3350 at a 45 ° angle ψ above the ground. Robot model 10<u style="single">0</u>The initial location of a should be centered within the 3-D scene 10a, and the initial virtual location of screen 312 is the vector between the eye position 3350 and the initial eye target 3352 on the robot 100 (eg, line of sight). It should be located along 3354).

To change the eye position 3350 and / or the line of sight 3354, the user can use the immovable part of the 3-D map 3310 (eg, robot model 10).<u style="single">0</u>It's better to select (part of 3-D scene 10a) instead of a, move the touch location along screen 312, and release. The software application 3300 should rotate the eye position 3350 by the corresponding amount with respect to the eye target 3352 or screen location 3356 (ie, touch location).

In some embodiment, the software application 3300 uses the eye position 3350 and the selected screen location 3356 to move the line of sight 3354 from the eye position 3350 through the screen location 3356 with the 3-D screen 10a as the eye target 3352. It is good to project at the intersection of. A collection of eye targets 3352 can be used to obtain drive commands to guide the robot 100 along a predetermined path along the analyzed eye target 3352. In an additional embodiment, the software application 3300 seeks the first eye target 3352 and the last eye target 3352, respectively, a first selected screen location 3356 and a last selected screen location 3356 (eg, first touch and first touch and). Release point) is determined. If the first eye target 3352 coincides with the base 120 of the robot model 100a and the last eye target 3352 is located away from the current location of the robot model 100a, the software application 3300 issues a drive command to end the robot 100. It is better to drive to the location corresponding to the eye target 3352. In addition, the user can allow the robot 100 to autonomously advance to a destination location selected from its current location.

Kinematic state manipulation may be possible with the software application 3300. For example, the software application 3300 can calculate the position change and / or kinematic state of the robot model 100a (using, for example, the robot data model 3420) and issue appropriate commands to robot model the robot 100. Can be synchronized to 100a. The software application 3300 communicates with the robot controller 500 and sends commands to adjust the position and kinematic state of the robot 100 based on changes in the software application 3300 and to update the robot model 100a accordingly. Receive feedback on changes in the position and state of the robot. In other words, the robot's position and kinematic state can be synchronized between the software application 3300 and the robot 100. For example, the user can see the height H of the torso of the robot model 100a._TThe robot model 100a on the touch screen 312 can be operated by rotating the head 160. After recognizing the model operation, the software application 3300 can send the robot position and kinematic state data to the robot controller 500, which decides the appropriate command to change the posture of the robot 100 and / or the software. Application 3300 determines commands, updates the robot's posture, and sends these commands to robot controller 500 for execution. Further, when the user physically operates the robot 100 by changing the posture and / or position (location) of the robot 100, the robot 100 sends the corresponding data to the software application 3300, and the software application is a robot model. Update its position and its location in the 2-D layout map 1810 and 3-D map 3310. To provide the current and accurate 3-D map 3310, the robot 100 should execute the scene update routine after one or more robot-robot model synchronizations. The software application 3300 can control how often the scene update routine is executed.

In some embodiments, the software application 3300 displays a ghost image of the modified robot model 100a and then in real time with the actual robot 100 from the starting position and / or attitude to the modified position and / or attitude. A series of images of the moving robot model 100a are displayed in sequence, and finally the state after the last change is achieved.

With reference to FIGS. 37A and 37B, in some embodiment, the software application 3300 analyzes point group 3320 and unrecognizes mass 3324, such as a non-reflective object (eg, a black object), a light source (eg, eg, , Windows), etc., identify each of the unrecognizable masses 3324 (Fig. 37A) using the sensor system 400, and then pull the recognizable image 3710 (Fig. 37B) onto the corresponding unrecognizable mass 3324. Or overlay. The software application 3300 can communicate with the camera 320 viewing scene 10 or receive RGB image data from this camera 320, and RGB image data corresponding to the unrecognizable mass 3324 for use as a recognizable image 3710. A part of can be trimmed. In another embodiment, software application 3300 can use a general image stored for use as a recognizable image 3710. Therefore, it is better to render the 3-D map 3310 as a combination of one or more recognizable images 3710 overlaid on or combined with the point cloud 3320 and the point cloud 3320. In some embodiments, the recognizable image 3710 is partially transparent, so that the user can partially see the corresponding unrecognized mass 3324.

Referring to FIG. 38, in some embodiment, the software application 3300 communicates with the robot 100 via a wireless network or cloud computing network 3810 via a remote computer computing device 310r. Communication should go through firewall 3820 for security. In some embodiments, the remote computer computer 310r establishes a virtual private network (VPN) communication or hypertext transfer protocol (http) tunnel with the robot 100.

The cloud computing network 3810 can provide internet-based computing, which allows shared servers to provide resources, software and data to various computers and other devices on demand. For example, the cloud computing network 3810 may be a cloud computing service that includes at least one server computing device, which cloud computing service is on a service abstraction layer and a server virtual machine instantiated on it. It is good to include a hypertext transfer protocol wrapper for. The server computing device should be configured to parse HTTP requests and send HTTP responses. Cloud computing can be described as a technology that uses the Internet and a central remote server to maintain data and applications. Cloud computing allows users to access and use applications via Internet access without configuration and access personal files on any computer. Cloud computing enables relatively efficient computer processing by centralizing storage, memory, processing and bandwidth. The cloud computing network 3810 can provide scaleable on-demand computing power, storage and bandwidth, while mitigating robot hardware requirements (eg, free CPU and memory). By ensuring). By connecting the robot to the cloud computing network 3810, it is possible to automatically collect data on the robot operation and usage history without the robot 100 having to return to the base station. In addition, continuous data collection over time can generate large amounts of data that can be hunted for marketing, product development and support.

Various embodiment of the systems and techniques described herein can be realized with digital electronic circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software and / or combinations thereof. .. These various implementations can include implementations in the state of one or more computer programs that are executable and / or interpretable on a programmable system and are such programmable. The system is combined to receive data and instructions from the storage system, at least one input device and at least one output device, and send the data and instructions to the storage system, at least one input device and at least one output device. Or it includes at least one programmable processor, which is good for general purposes.

These computer programs (also called programs, software, software applications or code) include machine instructions for programmable processors and can be implemented in high-level procedures and / or object-oriented programming languages and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" are used to provide machine instructions and / or data to a programmable processor that includes a machine-readable medium that receives machine instructions as machine-readable signals. It means any computer program product, device and / or device used (eg, magnetic disk, optical disk, memory, programmable logic device (PLD)). The term "machine readable signal" means any signal used to provide machine instructions and / or data to a programmable processor.

The content of the invention and the embodiment of the functional operations described herein are digital electronic circuits or computer software, firmware or structures disclosed herein and equivalent examples of these structures or one or two of them. It can be embodied in hardware that includes more than one combination. Embodiments of the content of the invention described herein are on one or more computer program products, i.e., a computer-readable medium that is executable by or controls the operation of the data processing device. It can be embodied as one or more modules of computer program instructions coded in. The computer-readable medium may be a machine-readable storage device, a machine-readable storage board, a memory device, a component of an object that sends a machine-readable propagation signal, or a combination of one or more of these. The term "data processor" includes all devices, devices and machines for processing data, such as programmable processors, computers or multiple processors or computers. The device, in addition to the hardware, is the code that gives rise to the execution environment for the computer program in question, such as the processor firmware, protocol stack, database management system, operating system, or a combination of one or more of these. It is good to include the code that makes up the. Propagation signals are artificially generated signals, such as machine-generated electrical signals, optical signals or electromagnetic signals, that are generated to encode information for transmission to a suitable receiver device.

Computer programs (also called programs, software, software applications, scripts or code) are writable in any form of programming language, including compiled or interpreted languages, which can be written as stand-alone programs or modules, components, subroutines or It is available in any form included as another unit suitable for use in a computer processing environment. Computer programs do not always support files in the file system. A program may be another program or data (eg, a file that stores one or more modules, subprograms, or parts of code) in a single file or multiple coordinated files dedicated to the program in question. It can be stored in a part of a file that holds one or more scripts stored in a make-up language sentence. Computer programs are available to run on a number of computers that are located on one computer or site or distributed across multiple locations and are connected to each other by a communication network.

The process and logic flow described herein is one or more programmable to execute one or more computer programs to perform a function by working on the input data and producing an output. It can be implemented by the processor. Processes and logic flows can also be implemented by dedicated purpose logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits), and devices can also be embodied as these.

Suitable processors for executing computer programs include, for example, both general purpose microprocessors and dedicated purpose processors and any one or more processors of any type of digital computer. Generally, the processor receives instructions and data from read-only storage or read-write storage. An essential element of a computer is a processor that executes instructions and one or more storage devices that store instructions and data. In general, a computer includes or receives data from one or more mass storage devices that store data, such as magnetic disks, magneto-optical disks or optical disks, and / or transmits data to them. Operated to do. However, the computer need not have such a device. In addition, computers can be embedded in other devices, such as mobile phones, personal digital assistants (PDAs), mobile audio players, and Global Positioning System (GPS) receivers, to name a few. Computer-readable media suitable for storing computer program instructions and data include non-volatile memory, media and storage devices of all forms, such as semiconductor storage devices such as EPROM. Examples include EEPROM and flash memory devices, magnetic disks such as internal hard disks or removable disks, opto-magnetic disks, CDROMs and DVD-ROM disks. The processor and memory are preferably captured by dedicated purpose logic circuits or incorporated therein.

Examples of embodiment of the contents of the present invention described herein include, for example, a back-end component as a data server or a middleware component, such as an application server, or a front-end computer, eg, a user described herein by a user. Any combination of back-end, middleware or front-end components that includes or has one or more such back-end, middleware or front-end components that include a client computer with a graphical user interface or web browser that allows interaction with examples of the content of the invention. It can be embodied in the including computing system. The components of such a system can be connected to each other by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LAN) and wide area networks (WAN), such as the Internet.

The computing system may include clients and servers. Clients and servers are generally separated from each other and typically interact over a communication network. The client-server relationship arises from computer programs that run on their respective computers and have a client-server relationship with each other. The present specification has many details, which should not be mediated by the scope of the invention or the claims of the invention, but rather are specific to a particular embodiment of the invention. It should be mediated as an explanation of the features. Certain features described herein in relation to separate embodiment can also be embodied in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment can also be embodied separately or in any suitable subcombination state in a number of embodiment. Further, the features described above as being used in a particular combination, and in some cases initially claimed in that state, may include one or more features from the claimed combination. , In some cases, the combination that can be removed from the combination and claimed is a sub-combination or<u style="single">sub</u>It can be changed to a modified example of the combination.

Similarly, operations or steps are described in the drawings in a particular order, which means that such operations or steps are performed in the specific order or sequential order shown or that all illustrated operations or steps are desired. It should not be understood as a prerequisite that it should be carried out to achieve the purpose of. In certain situations, multiple tasking and concurrency may be advantageous. Moreover, the separation of the various system components in the embodiments described above should not be understood to require such separation in any embodiment, and the program components and systems described above are generally single software products. It should be understood that they can be integrated with each other or packaged into a large number of software products.

Many concrete examples have been explained. Nevertheless, it will be appreciated that various modifications can be made without departing from the spirit and scope of the invention. Therefore, there are other embodiment within the scope of the invention described in the claims. For example, the acts described in the claims can be carried out in a different order, and even in this case, the desired result can be achieved.