Mobile robot system

[Cross-reference of related applications] This U.S. patent application is U.S. Provisional Application No. 61 / 428,717 filed December 30, 2010, U.S. Provisional Application No. 61 / 428,734 filed December 30, 2010, December 30, 2010. Claims priority under U.S. Patent Act Article 119 (e) over U.S. Provisional Application No. 61 / 428,759 filed in and U.S. Provisional Application No. 61 / 429,863 filed on January 5, 2011. .. These prior disclosures are deemed to be part of the disclosures of the present application and are incorporated herein by reference in their entirety.

The present disclosure relates to a mobile robot system that incorporates cloud computing.

Robots are generally electromechanical machines guided by computers or electronic programming. Movable robots have the ability to move around in their environment and are not fixed in one physical position. Examples of mobile robots commonly used today are automated guided vehicles (AGVs) or automated guided vehicles (AGVs). AGVs are generally mobile robots that follow markers or wires in the floor or use visual systems or lasers for navigation. Mobile robots can be found in industrial, military, and security environments. They have also emerged as consumer products that perform certain tasks such as recreational or vacuum cleaner cleaning and home support.

One aspect of the present disclosure is to communicate with a mobile robot having a controller that executes a control system for controlling the movement of the robot, a cloud computing service communicating with the robot controller, and a cloud computing service. Provides robotic systems, including remote computing devices. Remote computing devices communicate with robots through cloud computing services.

The embodiment of the present disclosure may include one or more of the following features: In some embodiments, the remote computing device runs an application for generating a layout map of the robot operating environment. The remote computing device may use a cloud computing service to store the layout map in an external cloud storage device. In some embodiments, the robot controller accesses the layout map through a cloud computing service to issue drive commands to the robot's drive system.

The remote computing device may execute an application (eg, a software program or routine) that provides remote control of the robot. For example, an application can drive a robot, change the posture of the robot, view a video from the robot's camera, and operate the robot's camera (eg, move the camera, and / or camera). You may provide control for at least one of (taking a snapshot or photo using).

In some embodiments, the remote computing device runs an application that provides video conferencing between the user of the computing device and a third party within the field of view of the robot's camera. The remote computing device may run an application to schedule the use of the robot. The remote computing device may also run applications for monitoring the use and operation of the robot. The remote computing device may optionally include a tablet computer with a touch screen.

Another aspect of the disclosure is a mobile robot having a controller that executes a control system to control the movement of the robot, a computing device communicating with the controller, and a cloud communicating with the computing device. It provides a robot system that includes a computing service and a portal that communicates with the cloud computing service.

The embodiment of the present disclosure may include one or more of the following features. In some embodiments, the portal comprises a web-based portal that provides access to the content. The portal may receive robot information from the robot through a cloud computing service. The robot may also receive user information from the portal through a cloud computing service.

In some embodiments, the computing device includes a touch screen (such as on a tablet computer). The computing device may run a different operating system than the controller's operating system. For example, the controller may run an operating system for robot control, while the computing device may run an enterprise operating system. In some embodiments, the computing device runs at least one application that collects robot information from the robot and sends the robot information to a cloud computing service.

The robot may include a base that defines a vertical central axis and supports the controller, and a holonomic drive system supported by the base. The drive system has first, second, and third drive wheels, each separated in a triangle around the vertical central axis and having a drive direction perpendicular to the radial axis with respect to the vertical central axis, respectively. The robot may also include expandable legs extending upward from the base and a torso supported by the legs. The operation of the legs causes a change in the height of the torso. The computing device can be detachably supported above the torso. In some embodiments, the robot comprises a neck supported by the torso and a head supported by the neck. The neck may be able to rotate and tilt its head with respect to the body. The head may detachably support the computing device.

Another aspect of the disclosure is between a mobile robot, a computing device communicating with the controller, and a controller and a computing device, having a controller that performs a control system to control the movement of the robot. It provides a robot system including an intermediary security device that controls communication, a cloud computing service that communicates with a computing device, and a portal that communicates with the cloud computing service.

In some embodiments, the intermediary security device translates communication between the computing device communication protocol of the computing device and the robot communication protocol of the robot. In addition, the intermediary security device may include an authorization chip for approving communication traffic between computing devices within the robot.

The computing device may communicate wirelessly with the robot controller. In some embodiments, the computing device is releasably attachable to the robot. An example computing device includes a tablet computer.

The portal may be a web-based portal that provides access to content (eg, news, weather, robot information, user information, etc.). In some embodiments, the portal receives robot information from the robot through a cloud computing service. In a further embodiment, the robot receives user information from the portal through a cloud computing service. Computing devices may use cloud computing services to access cloud storage devices. The computing device may run at least one application that collects robot information from the robot and sends the robot information to a cloud computing service.

One aspect of the present disclosure is to receive a layout map corresponding to the environment of the robot, to move the robot in the environment to the layout map position on the layout map, to correspond to the environment, and to be generated by the robot. Recording the robot map position on the robot map and using the recorded robot map position and the corresponding layout map position to determine the distortion between the robot map and the layout map, and the corresponding goal. Provided is a method of operating a mobile robot, including applying the determined distortion to a target layout map position so as to determine the robot map position.

The embodiment of the present disclosure may include one or more of the following features. In some embodiments, the method involves receiving a layout map from a cloud computing service. The method may include generating a layout map on an application running on a computing device and using a cloud computing service to store the layout map on a remote cloud storage device.

In some embodiments, the method uses existing layout map positions and recorded robot map positions to determine the scaling size, start point mapping, and rotation between the layout map and the robot map. Includes resolving the robot map position corresponding to the target layout map position. The method may further include applying the affine transformation to the determined scaling size, start point mapping, and rotation to resolve the target robot map position.

In some embodiments, the method involves determining a triangulation between layout map positions that indicate the boundaries of the target layout map position. The method also determines the scale, rotation, translation, and skew between the triangle mapped in the layout map and the corresponding triangle mapped in the robot map, and the corresponding target robot. It may include applying the determined scale, rotation, translation, and skew to the target layout map position so as to determine the map position.

The method, in some embodiments, determines the distance between all layout map positions and the target layout map position, determines the center of gravity of the layout map position, and all recorded robot maps. Determining the centroid of a position and for each layout map position, rotate and scale the length so that the vector extending from the layout map centroid to the target layout position is transformed into a vector extending from the robot map centroid to the target robot map position. Including to judge.

The method may include using a robot's sensor system to generate a robot map. In some implementations, the method radiates light onto the scene of the environment, receives reflections of the radiated light from the surface of the scene, determines the distance between each reflecting surface, and 3 of the scene. Includes building a dimensional depth map. The method may include radiating a speckle pattern of light onto the scene and receiving reflection of the speckle pattern from the scene. In some embodiments, the method comprises storing a reference image of a speckle pattern such that it is reflected from a reference object in the scene, the reference image being captured at various different distances from the reference object. .. The method further captures at least one target image with a speckle pattern that is reflected from the target object in the scene and references the at least one target image to determine the distance of the reflective surface of the target object. It may include comparing with an image. In some embodiments, the method determines the primary speckle pattern on the target object and at least one of the respective cross-correlation and non-correlation between the primary speckle pattern and the speckle pattern of the reference image. Includes calculating one. The method may include maneuvering the robot with respect to the target object based on the determined distance of the reflective surface of the target object.

In some embodiments, the method includes determining the flight time between the emission of light and the reception of reflected light and determining the distance to the reflecting surface of the scene. The method may include radiating light onto the scene with intermittent pulses. The method may also include changing the frequency of the synchrotron radiation pulse.

In yet another aspect, the robot system includes a mobile robot having a controller that executes a control system for controlling the movement of the robot and a sensor system communicating with the controller. Robot systems also include cloud computing services that communicate with robot controllers. The cloud computing service receives data from the controller, processes the data, and returns the processed result to the controller.

In some embodiments, the cloud computing service stores the received data in the cloud storage device at least temporarily, and optionally discards the stored data after processing the data. In some embodiments, the robot is communicating with a camera and / or a controller capable of capturing an image of the scene around the robot and / or the volume around the robot. It includes a volume point cloud imaging device capable of acquiring a point cloud from space. The volume space may include the floor surface in the moving direction of the robot. The controller transmits the image data to the cloud computing service.

The data may include raw sensor data and / or data with relevant information from the sensor system. In some embodiments, the data includes image data having at least one of accelerometer data tracking, mileage measurement data, and a time stamp.

The cloud computing service may receive image data of the scene around the robot from the controller and process the image data into a 3-D map and / or model of the scene. In addition, the cloud computing service may provide the controller with a 2-D height map and / or model, where the cloud computing service calculates the 2-D height map from the 3-D map. In some embodiments, the cloud computing service periodically receives image data, accumulates up to a threshold image dataset, and then processes the received image data.

In some embodiments, the controller communicates with the controller and optionally cloud-computes data wirelessly through a portable computing device (eg, a tablet computer) that can be detachably attached to the robot. Communicate to service. The controller may buffer the data and periodically send the data to the cloud computing service.

The sensor system may include at least one of a camera, a 3-D imaging sensor, a sonar sensor, an ultrasonic sensor, LIDAR, LADAR, an optical sensor, and an infrared sensor.

In another aspect of the present disclosure, the method of operating the mobile robot is to move the robot around the scene, receive sensor data indicating the scene, process the received sensor data, and process the processing result into the robot. Includes transmitting sensor data to cloud computing services. The method further includes moving the robot in the scene based on the received processing result.

In some embodiments, the method comprises radiating light onto the scene around the robot and capturing an image of the scene along the driving direction of the robot. The image includes at least one of (a) a three-dimensional depth image, (b) an active illumination image, and (c) an ambient illumination image. The sensor data includes images and the processing results include a map or model of the scene.

This method emits a speckle pattern of light on the scene, receives reflection of the speckle pattern from an object in the scene, and a reference image of the speckle pattern that is reflected from a reference object in the scene. May include storing in the cloud storage device of the cloud computing service. The reference image is captured at various different distances from the reference object. The method also includes capturing at least one target image of a speckle pattern that is reflected from a target object in the scene and transmitting at least one target image to a cloud computing service. The cloud computing service compares at least one of the target images with the reference image in order to determine the distance of the reflective surface of the target object. In some embodiments, the method determines the primary speckle pattern on the target object and at least one of the respective cross-correlation and non-correlation between the primary speckle pattern and the speckle pattern of the reference image. Includes calculating one.

The cloud computing service may store the sensor data received in the cloud storage device at least temporarily, and optionally discard the stored sensor data after processing the data. The sensor data may include image data with associated sensor system data, which may include at least one of accelerometer data tracking, mileage measurement data, and time stamps.

In some embodiments, the cloud computing service receives image data from the robot and processes the image data into a 3-D map and / or model of the scene. Cloud computing services may provide robots with 2-D height maps and / or models. The cloud computing service calculates a 2-D height map from a 3-D map.

The method may include periodically transmitting sensor data to the cloud computing service, which processes the image data received after accumulating the threshold sensor dataset. In some embodiments, the method communicates with the robot and optionally cloud sensor data wirelessly through a portable computing device (eg, a tablet computer) that is removable and attachable to the robot. Includes communicating to computing services.

In another aspect, the method of navigating a mobile robot is to capture a streaming sequence of high-density images of the scene around the robot along the trajectory of the robot's motion at real-time capture speed, and to high-density annotations. Includes associating with at least some of the images. The method also includes sending high density images and annotations to a remote server at a transmission rate slower than the real-time capture rate, and receiving the dataset from the remote server after a processing time interval. The dataset derives from and represents at least a portion of the high density image sequence and the corresponding annotation, but excludes the raw image data of the high density image sequence. The method includes moving the robot relative to the scene based on the received dataset.

The method may include sending high density images and annotations to a local server and buffer, and then sending high density images and annotations to a remote server at a transmission rate slower than the real-time capture rate. Local servers and buffers may be within a relatively short range of the robot (eg, within 20-100 feet or wireless communication range).

In some embodiments, the annotation is a time stamp, such as an absolute time reference, corresponding to at least some of the high density images, and at least of mileage measurement data, accelerometer data, tilt data, and angular velocity data. Includes posture-related sensor data, which may include one. Annotations can be associated with high-density images that reflect dangerous events captured in time intervals for a robot's hazard response (eg, avoiding cliffs, escaping confinement, etc.). In a further embodiment, associating an annotation may include associating a keyframe identifier with a subset of high density images. The keyframe identifier can enable identification of high density images based on the characteristics of the keyframe identifier (eg, flags, types, groups, etc.).

Annotations may include a set of low density 3-D points derived from the structural and motion restoration of features tracked between high density images in a streaming sequence of high density images. The low density set of 3-D points can be derived from the volume point imaging device on the robot. In addition, the annotation may include camera parameters such as the camera's orientation with respect to the individual 3-D points of the low density set of 3-D points. The crossable and non-crossable area markings of the scene can be annotations to the high density image.

The dataset contains one or more texture maps, such as 2-D height maps, and / or terrain maps that represent features in the high-density image of the scene, extracted from the high-density image. It may be. The dataset may include trained classifiers for classifying features within new high-density images that capture the scene.

In yet another aspect, the method of abstracting the mobile robot environment data comprises receiving a sequence of high density images of the robot environment from the mobile robot at a receiving rate. High-density images are captured along the trajectory of the mobile robot's motion at real-time capture speeds. The reception speed is slower than the real-time acquisition speed. The method further receives annotations associated with at least some of the high density images in a sequence of high density images and high density data in at least some of the high density images. Includes dispatching batch processing tasks to reduce to datasets that represent at least a portion of a sequence of images. The method also includes transmitting the dataset to a mobile robot. The dataset excludes raw image data from a sequence of high density images.

In some embodiments, the batch processing task processes a sequence of high-density images into a high-density 3-D model of the robotic environment, and processes a high-density 3-D model into a 2-D position and Includes making a terrain model for a coordinate system at least one height above the floor. In some embodiments, the terrain model is for a coordinate system of 2-D positions and multiple occupied and unoccupied height boundaries from the floor. For example, a terrain model that allows a robot to determine if a room with a table can pass under the table when it provides data indicating the upper and lower bound heights of the associated table surface. Is.

Batch processing tasks accumulate high-density image sequences for multiple robotic environments (for example, so that the cloud can build classifiers to identify desired features in any environment). May include that. Therefore, the batch processing task may include training multiple classifiers on a sequence of high density images and / or one or more classifiers on a sequence of high density images. For example, a batch processing task associates annotations that reflect hazards with high-density images captured within a time interval for a mobile robot's hazard response, and, for example, a dataset of classifier model parameters. As provided, it may include training a classifier for risk-related high-density images using the relevant hazard annotations and the corresponding high-density images as training data. The classifier may include at least one support vector machine that builds at least one hyperplane for classification, and the model parameters are trained hyperplanes that can classify the dataset into risk-related classifications. Define. The model parameters may include sufficient parameters that define the kernel of the support vector machine, as well as soft margin parameters.

In some embodiments, the batch processing task involves instantiating multiple scalable virtual processes that are proportional to the scale of the dense image sequence being processed. At least some of the virtual processes are released after transmitting the dataset to the robot. Similarly, a batch processing task may include instantiating multiple scalable virtual memories proportional to the scale of the stored high density image sequence. At least some of the virtual memory is released after the data set is transmitted to the robot. In addition, batch processing tasks include distributing multiple scalable virtual servers according to their geographical proximity to mobile robots and / or network traffic from multiple mobile robots. Good.

Details of one or more embodiments of the present disclosure will be described in the accompanying drawings and in the description below. Other aspects, features, and advantages will become apparent from this description and drawings, as well as claims.

<figref num="1">It is a perspective view of the movable human interface robot as an example.</figref><figref num="2">It is the schematic of the movable human interface robot as an example.</figref><figref num="3">It is an elevation perspective view of the movable human interface robot as an example.</figref><figref num="4A">It is a front perspective view of the base which is an example of a movable human interface robot.</figref><figref num="4B">It is a rear perspective view of the base shown in FIG. 4A.</figref><figref num="4C">It is a top view of the base shown in FIG. 4A.</figref><figref num="5A">It is a front schematic of the base which becomes an example of a movable human interface robot.</figref><figref num="5B">It is a top view of the base part which becomes an example of a movable human interface robot.</figref><figref num="5C">It is a front view of the holonomic wheel which is an example for a movable human interface robot.</figref><figref num="5D">It is a side view of the wheel shown in FIG. 5C.</figref><figref num="6A">It is a front perspective view of the body part which is an example of a movable human interface robot.</figref><figref num="6B">It is a front perspective view of the body part which is an example which has the touch sensing ability of a movable human interface robot.</figref><figref num="6C">It is the bottom perspective view of the body part shown in FIG. 6B.</figref><figref num="7">It is a front perspective view of the neck part which is an example of a movable human interface robot.</figref><figref num="8A">It is the schematic of the circuit which becomes an example of a movable human interface robot.</figref><figref num="8B">It is the schematic of the circuit which becomes an example of a movable human interface robot.</figref><figref num="8C">It is the schematic of the circuit which becomes an example of a movable human interface robot.</figref><figref num="8D">It is the schematic of the circuit which becomes an example of a movable human interface robot.</figref><figref num="8E">It is the schematic of the circuit which becomes an example of a movable human interface robot.</figref><figref num="8F">It is the schematic of the circuit which becomes an example of a movable human interface robot.</figref><figref num="8G">It is the schematic of the circuit which becomes an example of a movable human interface robot.</figref><figref num="9">FIG. 5 is a perspective view of an exemplary mobile human interface robot with a removable web pad.</figref><figref num="10A">It is a perspective view of a person interacting with an example movable human interface robot.</figref><figref num="10B">It is a perspective view of a person interacting with an example movable human interface robot.</figref><figref num="10C">It is a perspective view of a person interacting with an example movable human interface robot.</figref><figref num="10D">It is a perspective view of a person interacting with an example movable human interface robot.</figref><figref num="10E">It is a perspective view of a person interacting with an example movable human interface robot.</figref><figref num="11A">It is the schematic of the movable human interface robot as an example.</figref><figref num="11B">FIG. 5 is a perspective view of an exemplary mobile human interface robot having a plurality of sensors directed to the ground.</figref><figref num="12A">It is the schematic of the image pickup sensor which is an example which senses an object in a scene.</figref><figref num="12B">It is the schematic of the arrangement which becomes an example of the operation for operating an image sensor.</figref><figref num="12C">It is a schematic diagram of a three-dimensional (3D) speckle camera which is an example of detecting an object in a scene.</figref><figref num="12D">It is the schematic of the arrangement which becomes an example of the operation for operating a 3D speckle camera.</figref><figref num="12E">It is a schematic diagram of a 3D flight time (TOF) camera which is an example of detecting an object in a scene.</figref><figref num="12F">It is the schematic of the arrangement which becomes an example of the operation for operating a 3D TOF camera.</figref><figref num="13">It is the schematic of the control system which is an example executed by the controller of the movable human interface robot.</figref><figref num="14">It is a perspective view of an example movable human interface robot that receives a human touch command.</figref><figref num="15">A schematic telephone communication diagram is provided as an example for starting and performing communication with a movable human interface robot.</figref><figref num="16A">A schematic diagram of an example robot system architecture is provided.</figref><figref num="16B">A schematic diagram of an example robot system architecture is provided.</figref><figref num="16C">A schematic diagram of an example robot system architecture is provided.</figref><figref num="16D">A schematic diagram of an example robot system architecture is provided.</figref><figref num="16E">A schematic diagram of an example robot system architecture is provided.</figref><figref num="16F">Provided is an example arrangement of movements for a method of navigating a mobile robot.</figref><figref num="16G">Movable Robots Provide an example arrangement of operations for a method of abstracting environmental data.</figref><figref num="16H">A schematic diagram of an example robot system architecture is provided.</figref><figref num="17A">It is a schematic diagram of an example occupation map.</figref><figref num="17B">It is a schematic diagram of a movable robot which has a view of a scene in a work area.</figref><figref num="18A">It is a schematic diagram of an example layout map.</figref><figref num="18B">It is a schematic diagram of the robot map which is an example corresponding to the layout map shown in FIG. 18A.</figref><figref num="18C">Layout maps and robot maps are used to provide exemplary arrangements for manipulating mobile robots to navigate around the environment.</figref><figref num="19A">It is a schematic diagram of the layout map which is an example with a triangulation type layout point.</figref><figref num="19B">It is a schematic diagram of the robot map which is an example corresponding to the layout map shown in FIG. 19A.</figref><figref num="19C">A layout map and a robot map are used to provide an example arrangement of actions for determining a target robot map position.</figref><figref num="20A">It is the schematic of the layout map which is an example with the center of gravity of a close layout point.</figref><figref num="20B">It is a schematic diagram of the robot map which is an example corresponding to the layout map shown in FIG. 20A.</figref><figref num="20C">A layout map and a robot map are used to provide an example arrangement of actions for determining a target robot map position.</figref><figref num="21A">A schematic diagram is provided that provides an example of a local perceptual space while a mobile human interface robot is stationary.</figref><figref num="21B">A schematic diagram is provided that provides an example of a local perceptual space while a mobile human interface robot is moving.</figref><figref num="21C">A schematic diagram of a local perceptual space while a mobile human interface robot is stationary is provided.</figref><figref num="21D">A schematic diagram is provided that provides an example of a local perceptual space while a mobile human interface robot is moving.</figref><figref num="21E">An exemplary schematic is provided for a mobile human interface robot with a corresponding sensor field of view that moves closer around a corner.</figref><figref num="21F">An exemplary schematic of a mobile human interface robot with a corresponding sensor field of view that moves widely around a corner is provided.</figref>

Similar reference symbols in various drawings refer to similar elements.

Mobile robots can interact with or interact with people to provide a number of services, from home support to commercial support and more. In the example of home support, mobile robots include medication plan maintenance, mobility support, communication support (eg video conferencing, telephone communication, internet access, etc.), home or field monitoring (indoor and / or outdoor), person monitoring, It can support the daily lives of the elderly, including, but not limited to, providing personalized emergency response systems (PERS). In commercial support, mobile robots can provide video conferencing (eg, in hospital situations), sales floor dedicated terminals, interactive information / marketing terminals, and the like.

With reference to FIGS. 1 and 2, in some embodiments, the mobile robot 100 includes a robot body 110 (or chassis) that defines a forward drive direction F. The robot 100 also includes a drive system 200, an interface module 300, and a sensor system 400, each of which is supported by the robot body 110 and communicates with a controller 500 that coordinates the movement and movement of the robot 100. The power source 105 (eg, batteries (s) can be carried by the robot body 110 and telecommunications with each of these components and deliver power to them as needed. For example, the controller 500 may include a computer capable of over 1000 MIPS (1 million instructions / second), and the power supply 1058 provides sufficient battery to power the computer for more than 3 hours.

In the embodiments shown, the robot body 110 includes a base 120, at least one leg 130 extending upward from the base 120, and a body 140 supported by at least one leg 130. The base 120 may support at least a portion of the drive system 200. The robot body 110 also includes a neck 150 supported by a body 140. The neck 150 supports the head 160, which supports at least a portion of the interface module 300. The base 120 has a low center of gravity CG of the base 120 in order to maintain mechanical stability._BAnd low overall center of gravity CG of robot 100_RIncludes enough weight to maintain (eg, by supporting power supply 105 (battery)).

With reference to FIGS. 3 and 4A-4C, in some embodiments, the base 120 defines a triangularly symmetrical shape (eg, a triangular shape in the top view). For example, the base 120 supports a base body 124 having first, second, and third base body portions 124a, 124b, 124c corresponding to each leg of the triangular base 120 (see, eg, FIG. 4A). The base chassis 122 may be included. Each base body portion 124a, 124b, 124c can be movably supported by the base chassis 122 so that it can move independently of the base chassis 122 in response to contact with an object. The triangular symmetric shape of the base 120 enables 360 ° collision detection around the robot 100. Each base body portion 124a, 124b, 124c may have associated contact sensors (eg, capacitance sensors, reed switches, etc.) that detect movement of the corresponding base body portions 124a, 124b, 124c with respect to the base chassis 122.

In some embodiments, the drive system 200 provides omnidirectional and / or holonomic motion control for the robot 100. As used herein, the term "omnidirectional" refers to the ability to move in virtually any plane direction, i.e., left-right (lateral), forward / backward, and rotational movement. These directions are generally referred to herein as x, y, and θz, respectively. Moreover, the term "holonomic" is substantially consistent with the use of the term in the literature and refers to the ability to move in the plane with three degrees of freedom, namely two translations and one rotation. Therefore, the holonomic robot has the ability to move in the plane at a speed consisting of virtually any ratio of the three plane velocities (front / rear, lateral, and rotation), and virtually continues the three ratios. Has the ability to change.

Robot 100 can use wheel mobility to operate in a human environment (eg, an environment typically designed for bipedal pedestrians). In some embodiments, the drive system 200 is evenly spaced (eg, 120 degrees apart) about the vertical axis Z from the first, second, and third drive wheels 210a, 210b, 210c (ie, 120 degrees apart). , Triangular symmetry), however, other arrangements are possible as well. With reference to FIGS. 5A and 5B, the drive wheels 210a, 210b, 210c have a laterally bowed rolling surface (ie, rolling direction D) that can facilitate the maneuverability of the nonholonomic drive system 200._RA contour that traverses or curves in a direction perpendicular to it) may be defined. Each drive wheel 210a, 210b, 210c can drive the drive wheels 210a, 210b, 210c in the forward and / or reverse directions independently of the other drive motors 220a, 220b, 220c, respectively. It is connected to 220b and 220c. Each drive motor 220a-c can have its own encoder 212 (FIG. 8C) that provides wheel rotation feedback to the controller 500. In some embodiments, each drive wheel 210a, 210b, 210c is mounted on or near one of the three points of an equilateral triangle with respect to the bisector of the corner of the end of each triangle. Has right-angled drive directions (forward and reverse). The drive of the triangular symmetric holonomic base 120 with forward drive direction F autonomously escapes from confinement or scatter, then rotates and / or translates to drive along forward drive direction F after the escape is resolved. This allows the robot 100 to shift in a non-forward drive direction.

With reference to FIGS. 5C and 5D, in some embodiments, each drive wheel 210 is in the direction of rotation D of the drive wheel 210, respectively._RAnd the direction of rotation perpendicular to D_rIncludes inner and outer rows 232, 234 of rollers 230. Rows 232, 234 of the rollers 230 can be alternated (eg, so that one roller 230 in the inner row 232 is equally positioned between two adjacent rollers 230 in the outer row 234). The roller 230 provides infinite sliding perpendicular to the drive direction of the drive wheels 210. As the rollers 230 together define the circular or substantially circular perimeter of the drive wheels 210, the rollers 230 rotate in their direction D._rDefines an arched (eg, convex) outer surface 235 perpendicular to. The outer shape of the roller 230 affects the overall outer shape of the drive wheels 210. For example, the rollers 230 may define a bow-shaped outer roller surface 235, both of which define the corrugated rotating surface of the drive wheels 210 (eg, as a tread for static friction). However, by configuring the rollers 230 to have contours that define the circular overall rotating surface of the drive wheels 210, the robot 100 moves smoothly on a plane instead of vibrating vertically on the treads of the wheels. Make it possible. When approaching an object at an angle, to climb alternating rows 232, 234 of rollers 230 (of radius r) to an object as high as, or nearly as high as, the wheel radius R of the drive wheels 210. Can be used as a tread.

In the embodiment shown in FIGS. 3 to 5B, the first drive wheel 210a is arranged as a tip drive wheel along the forward drive direction F, and the remaining two drive wheels 210b and 210c follow from behind. .. In this arrangement, in order to move forward, the controller 500 slides the first drive wheels 210a along the forward drive direction F, while the second and third drive wheels 210b, 210c forward at equivalent speeds. A drive command may be issued to drive in the turning direction. Further, this drive wheel arrangement allows the robot 100 to stop suddenly (eg, generate a sudden negative acceleration with respect to the forward drive direction F). This is due to the natural dynamic instability of the 3-wheel design. If the forward drive direction F is along the bisector of the angle between the two forward drive wheels, the sudden stop produces torque that can cause the robot 100 to tip over and of its two "front" wheels. Pivot on. Instead, forward movement with one drive wheel 210a inevitably supports or prevents the robot 100 from tipping forward when a rapid stop is required. However, when accelerating from a stop, the controller 500 is the CG of its entire center of gravity of the robot 100._RThe moment of inertia I from may be considered.

In some embodiments of the drive system 200, each drive wheel 210a, 210b, 210 has a rolling direction D that is radially aligned with the vertical axis Z orthogonal to the X and Y axes of the robot 100._RHave. The first drive wheel 210a can be arranged as a tip drive wheel along the forward drive direction F, and the remaining two drive wheels 210b, 210c follow rearward. In this arrangement, in order to move forward, the controller 500 drives the first drive wheel 210a in the forward rotation direction, and the second and third drive wheels 210b and 210c have the same speed as the first drive wheel 210a. You may issue a drive command to drive in the opposite direction with.

In another embodiment, the drive system 200 is positioned so that the bisector of the angle between the two drive wheels 210a, 210b is aligned with the forward drive direction F of the robot 100, first and second. It can be arranged so as to have the second drive wheels 210a and 210b. In this arrangement, in order to move forward, the controller 500 drives the first and second drive wheels 210a, 210b at the same speed in the forward rotation direction, while driving the third drive wheel 210c in the opposite direction. A drive command may be issued to cause or keep the engine idling and drag it to the rear of the first and second drive wheels 210a, 210b. To turn left or right while moving forward, controller 500 may issue commands to drive the corresponding first or second drive wheels 210a, 210b at relatively faster / slower speeds. Arrangements for other drive systems 200 can be used as well. Drive wheels 210a, 210b, 210c may delineate cylindrical, circular, elliptical, or polygonal contours.

With reference to FIGS. 1 to 3 again, the base 120 supports at least one leg 130 extending upward from the base 120 in the Z direction. The legs (s) 130 may be configured to have variable heights for raising and lowering the torso 140 relative to the base 120. In some embodiments, each leg 130 includes first and second leg portions 132, 134 that move relative to each other (eg, nested stretch, linear, and / or angular movement). In the embodiment shown, the second leg portion 134 is said to have continuously smaller diameter extrusions that nestly extend and contract in and out of each other and out of the relatively larger base extrusions. Rather, it nests and stretches over the first leg portion 132, thus imposing other components along the second leg portion 134, and potentially the second leg portion. Allows movement with 134 in relative proximity to base 120. The leg 130 may include an actuator assembly 136 (FIG. 8C) for moving the second leg 134 relative to the first leg 132. Actuator assembly 136 may include motor driver 138a, which is communicating with lift motor 138b and encoder 138c, which provides position feedback to controller 500.

Generally, the nested telescopic arrangement is the center of gravity CG of the entire leg 130._LIn order to keep it as low as possible, the base 120 includes a continuously smaller diameter extrusion that nests up and out of the relatively larger extrusion. In addition, more robust and / or larger components can be placed on the bottom to accommodate the greater torque that the base 120 receives when the leg 130 is fully extended. However, this approach presents two problems. First, when relatively smaller components are placed on top of the leg 130, any rain, dust, or other microparticles tend to flow or roll down the extrusion, between the extrusions. It gets into the space of the extrusion and therefore clogs the nesting of the extrusion. This poses a very difficult sealing problem while still trying to maintain full mobility / joint movement of the leg 130. Second, it may be desirable to mount an load or accessory on the robot 100. One common place to mount accessories is at the top of the torso 140. If the second leg portion 134 nests out of the first leg portion, the accessories and components are the entire second leg portion 134 if they need to move with the torso 140. It can only be mounted above. Otherwise, any component mounted on the second leg portion 134 would limit the nested telescopic movement of the leg portion 130.

By nesting the second leg portion 134 over the first leg portion 132, the second leg portion 134 can be moved perpendicular to the base 120 for additional loading attachments. Provide points. This type of arrangement allows water or airborne particles to enter the space between the leg portions 132 and 134 and on the outside of the respective leg portions 132, 134 (eg, extrusion) of the body 140. Let it run down. This makes it much easier to seal any joint of the leg 130. In addition, the payload / accessory mounting features of the torso 140 and / or the second leg 134 are always exposed and available, regardless of how extended the leg 130 is.

With reference to FIGS. 3 and 6A, the legs (s) 130 support the body 140, which may have steps 142 extending above and above the base 120. In the embodiments shown, the body 140 has a downward facing surface or bottom surface 144 (eg, base facing) and an opposite upward facing surface or top surface 146 that form at least a portion of the step 142. And with a side 148 extending in between. The torso 140 has a central portion 141 supported by the legs (s) 130 and a peripheral free portion 143 that extends laterally beyond the range of the legs (s) 130. Various shapes or geometric shapes, such as circular or elliptical shapes, may be defined, thus providing an overhang that defines the downward facing surface 144. In some embodiments, the torso 140 is a polygonal or other complex shape that defines a step that provides an overhang that extends beyond the legs (s) 130 over the base 120. Is defined.

Robot 100 may include one or more accessory ports 170 (eg, mechanical and / or electrical interconnection points) to receive the payload. The accessory port 170 may be positioned so that the receiving payload does not block or block the sensor of the sensor system 400 (eg, above the bottom 144 and / or top 146 of the torso 140). it can. In some embodiments, as shown in FIG. 6A, the torso 140 receives the load in, for example, the basket 360 and is on the anterior portion 147 of the torso 140 or other part of the robot body 110. One or more accessory ports 170 are included on the rear portion 149 of the body 140 so as not to block the sensor.

The outer surface of the body 140 may be sensitive to touch or touch by the user so as to receive touch commands from the user. For example, when the user touches the top surface 146 of the torso 140, the robot 100 has a torso height H relative to the floor._TBy lowering (eg, the height H of the legs 130 supporting the torso 140)_LRespond (by reducing). Similarly, when the user touches the bottom surface 144 of the torso 140, the robot 100 raises the torso 140 relative to the floor (eg, the height H of the legs 130 supporting the torso 140)._LRespond (by increasing). Further, upon receiving a user touch on the anterior, posterior, right or left portion of the side surface 148 of the body 140, the robot 100 receives the corresponding directions of the received touch commands (eg, posterior, anterior, left, and respectively). Respond by moving to (right). The outer surface of the body 140 may include a capacitance sensor communicating with the controller 500 to detect user contact.

With reference to FIGS. 6B and 6C, in some embodiments, the body 140 has a body body having a top panel 145t, a bottom panel 145b, a front panel 145f, a back panel 145b, a right panel 145r, and a left panel 145l. Including 145. Each panel 145t, 145b, 145f, 145r, 145r, 145l may move independently of the other panels. In addition, each panel 145t, 145b, 145f, 145r, 145r, 145l detects motion and / or contact with the respective panel, associated motion and / or contact sensors 147t, 147b, communicating with controller 500, It may have 147f, 147r, 147r, 147l.

With reference to FIGS. 1-3 and 7 again, the torso 140 supports the neck 150, which provides the rotation and tilt of the head 160 with respect to the torso 140. In the embodiment shown, the neck 150 includes a rotating portion 152 and an inclined portion 154. The rotating portion 152 has a range of angular movement θ from about 90 ° to about 360 °._R(For example, centered on the Z axis) may be provided. Other ranges are possible as well. Further, in some embodiments, the rotating portion 152 maintains an infinite number of rotations of the head 150 relative to the body 140, while maintaining electrical communication between the head 160 and the rest of the robot 100. Includes electrical connectors or contacts that allow continuous 360 ° rotation of the. The tilted portion 154 is an identical or similar electrical connector that allows rotation of the head 160 with respect to the torso 140, while maintaining electrical communication between the head 160 and the rest of the robot 100. May include contact. The rotating portion 152 may include a rotating portion motor 152m connected to or fitted to a ring 153 (eg, a toothed ring rack). The inclined portion 154 has its head at an angle θ with respect to the body portion 140 independently of the rotating portion 152._TMay be moved to (eg, centered on the Y axis). In some embodiments, the tilted portion 154 tilts the head 160 at an angle θ of ± 90 ° with respect to the Z axis._TIncludes tilt motor 155 to move to. Other ranges such as ± 45 ° are possible as well. The robot 100 is configured such that the legs (s) 130, body 140, neck 150, and head 160 remain within the perimeter of the base 120 to maintain stable mobility of the robot 100. You may. In the example schematic shown in FIG. 8F, the neck 150 includes a rotation-tilt assembly 151 that includes a rotation portion 152 and an inclination portion 154 along with corresponding motor drivers 156a, 156b, and encoders 158a, 158b.

Head 160 may be sensitive to user contact or touch so that it receives touch commands from the user. For example, when the user pulls the head 160 forward, the head 160 tilts forward with passive resistance and then holds its position. Further, if the user pushes / pulls the head 160 vertically downward, the body 140 may be lowered to lower the head 160 (via reducing the length of the legs 130). Head 160 and / or neck 150 may include strain gauges and / or contact sensors 165 (FIG. 7) that sense user contact or operation.

8A-8G provide circuit diagrams that are examples of Robot 100. 8A-8C may accommodate proximity sensors such as sonar proximity sensor 410 and step proximity sensor 420, contact sensor 430, laser scanner 440, sonar scanner 460, and drive system 200, base 120. A circuit diagram that serves as an example of the above is provided. The base 120 may also accommodate the controller 500, the power supply 105, and the leg actuator assembly 136. The torso 140 includes a microcontroller 140c, a microphone (s) 330, a speaker (s) 340, a scanning 3-D image sensor 450a, and a controller 500 that receives and responds to user contact or touch. (For example, by moving the body 140 with respect to the base 120, rotating and / or tilting the neck 150, and / or issuing commands to the drive system 200 in response). , The body touch sensor system 480 may be accommodated. The neck 150 may include a rotating motor 152 having a corresponding motor driver 156a and an encoder 158a and a tilting motor 154 having a corresponding motor driver 156b and an encoder 158b. Good. The head 160 may accommodate one or more web pads 310 and a camera 320.

With reference to FIGS. 1-3 and 9, in some embodiments, the head 160 supports one or more parts of the interface module 300. The head 160 may include a dock 302 for releasably accepting one or more computing tablets 310, also called a web pad or tablet PC, each capable of having a touch screen 312. The web pad 310 may be oriented forward, backward, or upward. In some embodiments, the webpad 310 includes a touch screen, optional inputs and outputs (eg, connectors such as buttons and / or micro USB), a processor, and memory communicating with the processor. An example web pad 310 is Apple, Including iPad by Inc. In some embodiments, the webpad and 10 either function as controller 500 or assist in controlling controller 500 and robot 100. In some embodiments, the dock 302 is fixedly attached to a first computing tablet 310a (eg, a wired interface for data transfer at relatively high bandwidths such as gigabyte speeds) and removed from it. Includes a second compute tablet 310b that can be installed. The second web pad 310b may be received over the first web pad 310a, as shown in FIG. 9, or the second web pad 310b heads relative to the first web pad 310a. It may be received on the opposite side or the opposite side of part 160. In a further embodiment, the head 160 supports a single web pad 310, which can either be fixed to it or detachably attached. The touch screen 312 may detect, monitor, and / or reproduce points of user touch on it to receive user input and provide a touch-interactive graphical user interface. In some embodiments, the web pad 310 includes a touch screen call that allows the user to know when it has been removed from the robot 100.

In some embodiments, the robot 100 includes a plurality of web pad docks 302 over one or more parts of the robot body 110. In the embodiment shown in FIG. 9, the robot 100 includes a web pad dock 302 optionally located on the legs 130 and / or the torso 140. This allows the user to adapt to users of different heights, capture video using the webpad 310's camera from different perspectives, and / or accept multiple webpads 310 on the robot 100. Allows the web pad 310 to be docked at different heights on the robot 100.

The interface module 300 may include a camera 320 placed on the head 160 that can be used to capture video from a high viewpoint of the head 160 (eg, for video conferencing) (eg, for video conferencing). See Figure 2). In the embodiment shown in FIG. 3, the camera 320 is placed on the neck 150. In some embodiments, the camera 320 is operated only when the web pads 310, 310a are removed from the head 160 or out of the dock. When the web pads 310, 310a are mounted or docked in the dock 302 (and optionally over the camera 320) on the head 160, the robot 100 uses the web pads 310a to capture video. You may use the camera of. In such cases, the camera 320 may be placed behind the docked web pad 310 and becomes active when the web pad 310 is removed from or removed from the head 160 and is activated. The web pad 310 becomes inactive when mounted or docked on the head 160.

Robot 100 can provide video conferencing through interface module 300 (eg, at 24 fps) (eg, using webpad 310, camera 320, microphone 320, and / or speaker 340). Video conferencing can be multi-person. Robot 100 can provide eye contact between both parties to the video conference by maneuvering the head 160 to face the user. Also, the robot 100 can have a gaze angle of less than 5 degrees (eg, an angle away from an axis perpendicular to the anterior surface of the head 160). At least one 3-D image sensor 450 and / or camera 320 on the robot 100 can capture full-scale images such as body language. Controller 500 can synchronize audio and video (eg, with a difference of less than 50 ms).

In the embodiment shown in FIGS. 10A-10E, the robot 100 adjusts the height of the head 160 and / or the web pad 310 on the camera 320 (by raising or lowering the torso 140) and / Alternatively, by rotating and / or tilting the head 160, a video conference can be provided to a standing or sitting person. The camera 320 may be movable within at least one degree of freedom, independent of the web pad 310. In some embodiments, the camera 320 is more than 3 feet above the ground, but is positioned from the top of the display area of the web pad 310 no more than 10 percent of the height of the web pad. It has an objective lens. In addition, the robot 100 can zoom the camera 320 to take a close-up photo or video of the robot 100's surroundings. The head 160 may include one or more speakers 340 so that the head 160 emits sound in the vicinity of the web pad 310 displaying the video conference.

In some embodiments, the robot 100 can receive user input into the web pad 310 (eg, via a touch screen), as shown in FIG. 10E. In some embodiments, the web pad 310 is a display or monitor, while in other embodiments, the web pad 310 is a tablet computer. The web pad 310 can have easy and intuitive controls such as a touch screen that provide high interactivity. The web pad 310 may have a monitor display 312 (eg, a touch screen) having a display area of 150 square inches or more that can be moved with at least one degree of freedom.

Robot 100 can provide EMR integration in some embodiments by providing video conferencing between doctors and patients and / or other doctors or nurses. The robot 100 may include a transit diagnostic instrument. For example, robot 100 may include a stethoscope configured to convey auscultation to a video conferencing user (eg, a doctor). In another embodiment, the robot allows direct connection to Class II medical devices such as electronic stethoscopes, otoscopes, and ultrasound to transmit medical data to remote users (doctors), connector 170. including.

In the embodiment shown in FIG. 10B, the user is for remote control of the robot 100, video conferencing (eg, using the camera and microphone of the webpad 310), and / or the use of software applications on the webpad 310. The web pad 310 may be removed from the web pad dock 302 on the head 160. Robot 100 has first and second cameras 320a, on the head 160, to acquire different viewpoints for video conferencing, navigation, etc. while the web pad 310 is detached from the web pad dock 302. 320b may be included.

An interactive application running on controller 500 and / or communicating with controller 500 may require more than one display on robot 100. Multiple webpads 310 associated with Robot 100 can provide different combinations of "FaceTime", Telestration, HD appearance (eg for Webpad 310 with built-in camera) to remotely control Robot 100. It can act as a remote operator control unit (OCU) and / or provide a local user interface pad.

Referring again to FIG. 6A, the interface module 300 includes a microphone 330 (eg, or a microphone array) for receiving voice input and one or more speakers 340 arranged on the robot body 110 for delivering voice output. And may be included. The microphone 330 and the speaker 340 may each communicate with the controller 500. In some embodiments, the interface module 300 includes a basket 360, which may be configured to hold brochures, emergency information, household items, and other items.

With reference to FIGS. 1 to 4C, 11A and 11B, in order to achieve reliable and robust autonomous movement, the sensor system 400 is intelligent about the actions that the robot 100 takes in its environment. It may include several different types of sensors that can be used in conjunction with each other to generate sufficient perception of the robot's environment to be able to make objective decisions. The sensor system 400 may include one or more types of sensors, including obstacle detection obstacle avoidance (ODOA) sensors, communication sensors, navigation sensors, etc., supported by the robot body 110. For example, these sensors include proximity sensors, contact sensors, 3D (3D) imaging / depth map sensors, cameras (eg visible and / or infrared cameras), sonars, radars, lidar (Light Detection And Ranging, remote). LADAR (Laser Detection and), which measures the properties of scattered light, may require optical remote sensing to discover the distance and / or other information of the target. Ranging), etc., but not limited to these. In some embodiments, the sensor system 400 includes a range-finding sonar sensor 410 (eg, nine around the base 120), a proximity step detector 420, a contact sensor 430, a laser scanner 440, and one or more. Includes 3-D imaging / depth sensor 450 and imaging sonar 460.

There are some challenges associated with placing the sensor on the basic skeleton of the robot. First, the sensors need to be positioned so that they have a maximum coverage area of interest around the robot 100. Second, the sensors may have to be placed so that the robot 100 itself blocks the sensors in an absolutely minimal manner, essentially the sensors, they are "obscured" by the robot itself. It cannot be fixed like this. Third, the placement and mounting of the sensor should not interfere with the rest of the industrial design of the basic backbone. From an aesthetic point of view, a robot with a discreetly mounted sensor can be considered more "attractive" than one that is not. From a practical point of view, the sensor should be mounted so as not to interfere with normal robot operation (such as being caught on an obstacle).

In some embodiments, the sensor system 400 communicates with the controller 500 and is located within one or more areas or parts of the robot 100 to detect any obstacles near or invading. Includes a set of proximity sensors 410, 420 or an array thereof, which is installed (eg, located on or near the base body portions 124a, 124b, 124c of the robot body 110). Proximity sensors 410, 420 provide a signal to controller 500 when an object is within a given range of robot 100: convergent infrared (IR) emitter-sensor element, sonar sensor, ultrasonic sensor, and / or It may be an imaging sensor (eg, a 3D depth map image sensor).

In the embodiments shown in FIGS. 4A-4C, the robot 100 is a sonar-type proximity that is located around the base body 120 (eg, substantially equidistant) and with an upward view. Includes an array of sensors 410. The first, second, and third sonar proximity sensors 410a, 410b, 410c are located on or near the first (front) base body portion 124a and are the outermost in the radial direction of the first base body 124a. At the edge 125a of the sonar proximity sensor, there is at least one of the sonar proximity sensors. The fourth, fifth, and sixth sonar proximity sensors 410d, 410e, and 410f are located on or near the second (right) base body portion 124b and are the outermost in the radial direction of the second base body 124b. It has at least one of the sonar proximity sensors near the edge 125b of the. The seventh, eighth, and ninth sonar proximity sensors 410g, 410h, 410i are located on or near the third (right) base body portion 124c and are the outermost in the radial direction of the third base body 124c. It has at least one of the sonar proximity sensors near the edge 125c of the. This configuration provides at least three detection areas.

In some embodiments, a set of sonar proximity sensors 410 (eg, 410a-410i) placed around the base body 120 is arranged to face upwards (eg, substantially in the Z direction) and is optional. Depending on the selection, it is angled outward away from the Z axis, thus creating a detection curtain 412 around the robot 100. Each sonar proximity sensor 410a-410i guides the sonar emission upwards, or at least not towards other parts of the robot body 110 (eg, not detecting movement of the robot body 110 with respect to itself), a shroud. Alternatively, it may have a release guide 414. The release guide 414 may define a shell or semi-shell shape. In the embodiments shown, the base body 120 extends laterally beyond the legs 130 and the sonar proximity sensors 410 (eg, 410a-410i) are on the base body 120 around the legs 130 (eg). , Substantially along the perimeter of the base body 120). In addition, the upward-facing sonar proximity sensors 410 are spaced apart to give rise to a continuous or substantially continuous sonar detection curtain 412 around the leg 130. The sonar detection curtain 412 can be used to detect obstacles with elevated lateral protrusions such as table tops, shelves, etc.

The looking-up sonar proximity sensor 410 provides the ability to see nearby objects in a horizontal plane, such as the top of a table. Due to their aspect ratio, these objects may be missed by other sensors in the sensor system, such as the laser scanner 440 or the imaging sensor 450, which can cause problems for the robot 100. A top-looking sonar proximity sensor 410, located around the periphery of the base 120, provides a means for viewing or detecting these types of objects / obstacles. In addition, the sonar proximity sensor 410 is placed at a slight outward angle around the maximum width point around the base so that it is not blocked or blocked by the body 140 or head 160 of the robot 100. And therefore does not result in false positives by sensing parts of the robot 100 itself. In some embodiments, the sonar proximity sensor 410 is arranged (upward and outward) to leave a volume around the body 140 outside the field of view of the sonar proximity sensor 410, and thus the basket 360, etc. Freely accept the payload or accessories to be loaded. The sonar proximity sensor 410 can be retracted into the base body 124 to provide visual concealment and eliminate external features that get caught or collide with obstacles.

The sensor system 400 has one or more sonar proximity sensors 410 (eg, rear proximity sensor 410j) oriented backwards (eg, opposite the forward drive direction F) to detect obstacles while retracting. ) May be included. The rear sonar proximity sensor 410j may include an emission guide 414 that directs its sonar detection field 412. In addition, the rear sonar proximity sensor 410j can be used to measure the distance between the robot 100 and a detected object in the field of view of the rear sonar proximity sensor 4l0j (eg, "backward alarm". As). In some embodiments, the rear sonar proximity sensor 410j is recessed and mounted within the base body 120 so as not to introduce any visual or functional irregularities to the housing configuration.

With reference to FIGS. 3 and 4B, in some embodiments, the robot 100 allows the drive wheels 210a, 210b, 210c to detect a step before it encounters a step (eg, a staircase). Includes a step proximity sensor 420 located near or around the drive wheels 210a, 210b, 210c. For example, the step proximity sensor 420 may be located at or near each of the outermost edges 125a to 125a in the radial direction of the base body 124a to c, or in between. In some cases, step sensing may include infrared emitters 422 and infrared detectors 424 that are angled towards each other so that they have overlapping radiation and detection fields, and thus detection areas, where the floor should be expected to be. Achieved using infrared (IR) proximity or real-range sensing, which is used. IR proximity sensing may have a relatively narrow field of view, reliability may depend on the surface albedo, and distance accuracy may vary between surfaces. As a result, a plurality of separate sensors can be placed around the perimeter of the robot 100 in order to properly detect the step from the plurality of points on the robot 100. Moreover, IR proximity-based sensors typically cannot distinguish between steps and safety events such as immediately after the robot 100 has climbed a threshold.

The step proximity sensor 420 can detect when the robot 100 encounters a low edge of the floor, such as when it encounters a set of stairs. The controller 500 (which executes the control system) may execute an action of causing the robot 100 to take an action such as changing the moving direction when the edge portion is detected. In some embodiments, the sensor system 400 includes one or more secondary step sensors (eg, other sensors configured for step sensing and optionally other types of sensing). The step detection proximity sensor 420 provides data to distinguish between actual steps and safety events (such as overcoming a threshold) so as to provide early detection of steps, and their field of view is the robot body. It can be arranged so as to be positioned lower on the outside so as to include at least a part of 110 and a region away from the robot body 110. In some embodiments, the controller 500 is the edge of a supporting work surface (eg, the floor), an increase in the distance past the edge of the work surface, and / or the distance between the robot body 110 and the work surface. Run a step detection routine that identifies and detects the increase. This embodiment: 1) early detection of potential steps (which may allow faster movement speeds in unknown environments), 2) whether or not the step event is really unsafe, or safely. Autonomous mobility by having the controller 500 receive step image information from the step detection proximity sensor 420 to know if it is possible to cross (eg, climb and cross a threshold). It enables increased reliability and 3) reduced false positives of steps (eg, by using edge detection vs. separate IR proximity sensors with multiple narrow fields of view). Additional sensors can be used, which are arranged as "derail" sensors, for redundancy and to detect situations where distance sensing cameras are unable to reliably detect certain types of steps.

The sill and step detection allows the robot 100 to efficiently plan either to cross a sill that can be climbed or to avoid a step that is too high. This can be the same for indiscriminate objects on the work surface that the Robot 100 can or cannot safely cross. Knowing the height of obstacles or thresholds that Robot 100 determines can be climbed is for Robot 100 to maximize smoothness and minimize any instability due to sudden acceleration. In addition, it allows for proper deceleration when deemed necessary to enable a smooth transition. In some implementations, sill and step detection is at the height of the object above the work surface, in addition to geometric shape recognition (eg, distinguishing between sill or small chunks such as electrical cables vs socks). Based on. The threshold may be recognized by edge detection. The controller 500 receives the imaging data from the step detection proximity sensor 420 (or another imaging sensor on the robot 100), executes the edge detection routine, and issues a drive command based on the result of the edge detection routine. May be good. Controller 500 may use pattern recognition as well to identify objects. The sill detection allows the robot 100 to change its orientation with respect to the sill so as to maximize its ability to climb steps smoothly.

Proximity sensors 410, 420 may function alone or, as an alternative, in combination with one or more contact sensors 430 (eg, raised switches) for redundancy. For example, one or more contact or ridge sensors 430 on the robot body 110 can detect whether the robot 100 has physically encountered an obstacle. Such sensors may use physical properties such as capacitance or physical movement within the robot 100 to determine when an obstacle is encountered. In some embodiments, the respective base body portions 124a, 124b, 124c of the base 120 detect the movement of the corresponding base body portions 124a, 124b, 124c with respect to the base chassis 122 (see, eg, FIG. 4A), the associated contact. It has a sensor 430 (eg, capacitance sensor, reed switch, etc.). For example, each base body 124a-c may move radially with respect to the Z axis of the base chassis 122 to provide three-way collision detection.

With reference to FIGS. 1 to 4C, 11A, and 11B again, in some embodiments, the sensor system 400 is mounted on the front portion of the robot body 110 and communicates with the controller 500, a laser. Includes scanner 440. In the embodiments shown, the laser scanner 440 is mounted on a first base body 124a or above the base body 120 facing forward (eg, to have a maximum imaging coverage along the robot drive direction F). It is mounted (eg, has a view along the forward drive direction F). In addition, the placement of the laser scanner on or near the anterior tip of the base 120 of the triangle states that the outer angle of the robot base (eg, 300 degrees) is greater than the field of view 442 (eg, about 285 degrees) of the laser scanner 440. This means that the base 120 prevents the laser scanner 440 from blocking or blocking the detection field of view 442. The laser scanner 440 is designed to minimize any part of the laser scanner from sticking out past the base body 124 (eg, for aesthetics and to minimize getting caught on obstacles). ), It can be mounted as retracted as possible in the base body 124 without obstructing its field of view.

The laser scanner 440 scans the surrounding area of the robot 100, and the controller 500 uses the signal received from the laser scanner 440 to create an environment map or object map of the scanned area. Controller 500 may use object maps for navigation, obstacle detection, and obstacle avoidance. In addition, controller 500 may use sensing inputs from other sensors in sensor system 400 to create object maps and / or to navigate.

In some embodiments, the laser scanner 440 has a laser that quickly scans an area in one dimension as the "main" scan line and a phase difference or similar technique for assigning depth to each generated pixel in the line. It is a scanning lidar that can be used with a flight time imaging element that uses (returns a two-dimensional depth line in the scanning plane). To generate a 3D map, lidar can perform an "auxiliary" scan in a second direction (eg, by "up and down" the scanner). This mechanical scanning technique, if uncomplicated, allows flight time calculations for a complete 2-D matrix of pixels to provide the depth of each pixel, or even a series of depths of each pixel. It can be complemented by techniques (using coded illuminators or illuminating lasers), techniques using semiconductor stacks, techniques such as "Flash" LIDAR / LADAR and "Swiss Ranger" type focal plane imaging element sensors.

The sensor system 400 may include one or more three-dimensional (3-D) image sensors 450 communicating with the controller 500. If the 3-D image sensor 450 has a limited field of view, the controller 500 or the sensor system 400 may use the 3-D image sensor 450a, so that the 3-D image sensor 450a produces a relatively wider field of view in order to perform robust ODOA. It can be operated to scan left and right. With reference to FIGS. 1-3 and 10B, in some embodiments, the robot 100 is mounted on the front portion of the robot body 110 with a view along the forward drive direction F (eg, the robot). Includes a scanning 3-D image sensor 450a (so that it has a maximum imaging coverage along the drive direction F). The scanning 3-D image sensor 450a can be used primarily for obstacle detection / obstacle avoidance (ODOA). In the embodiments shown, the scanning 3-D image sensor 450a is mounted on the body 140, for example, to prevent the user from coming into contact with the scanning 3-D image sensor 450a, as shown in FIG. Mounted on the bottom of the portion 142 or on the bottom surface 144 and retracted into the body portion 140 (eg, coplanar with or past the bottom surface 144). The scanning 3-D image sensor 450 has a downward view 452 within the front of the robot 100 for obstacle detection and obstacle avoidance (ODOA) (eg, by base 120 or other part of robot body 110). It can be arranged so as to aim substantially downward and away from the robot body 110 (with obstruction). Scanning on or near the front edge of the torso 140 When the 3-D image sensor 450a is placed, the field of view of the 3-D image sensor 450 (eg, about 285 degrees) is the torso relative to the 3-D image sensor 450. It can be made smaller than the outer surface angle of 140 (eg, 300 degrees), thus preventing the body 140 from blocking or blocking the detection field of view 452 of the scanning 3-D image sensor 450a. In addition, scanning 3-D image sensor 450a (and associated actuators) is possible It can be embedded and mounted within the torso 140 as long as it does not block its view (eg, for aesthetics and to minimize catching on obstacles). The distracting scanning motion of the scanning 3-D image sensor 450a results in a less unpleasant dialogue experience that is invisible to the user. Unlike protruding sensors or features, the retracted scanning 3-D image sensor 450a has virtually no moving parts extending beyond the perimeter of the torso 140, especially when moving or scanning, so it is environmentally friendly. Tends to have no unintended interactions (getting caught in people, obstacles, etc.).

In some embodiments, the sensor system 400 includes an additional 3-D image sensor 450 located on the base body 120, legs 130, and neck 150 and / or head 160. In the embodiment shown in FIG. 1, the robot 100 includes a 3-D image sensor 450 on a base body 120, a body 140, and a head 160. In the embodiment shown in FIG. 2, the robot 100 includes a 3-D image sensor 450 on a base body 120, a body 140, and a head 160. In the embodiment shown in FIG. 11A, the robot 100 includes a 3-D image sensor 450 on the legs 130, body 140, and neck 150. Other configurations are possible as well. One 3-D image sensor 450 (eg, above the neck 150 and above the head 160) can be used for human recognition, gesture recognition, and / or video conferencing, while another 3-D image sensor. 450 (eg, on base 120 and / or leg 130) can be used for navigation and / or obstacle detection and obstacle avoidance.

A forward-facing 3-D image sensor 450, located on the neck 150 and / or head 160, can be used for human recognition, face recognition, and / or gesture recognition of people around the robot 100. For example, using the signal input from the 3-D image sensor 450 on the head 160, the controller 500 creates a 3D map of the user's face seen / captured, and the created 3D map of the person. The user may be recognized by comparing it with a known 3-D image of the face and determining a match with one of the known 3-D face images. Face recognition may be used to authenticate the user as a legitimate user of Robot 100. In addition, one or more of the 3-D image sensors 450 are determined by the robot 100 to determine the viewer's gestures, based on the determined gestures (s) (eg, manual feed, hand gesture, and / or hand signal). , Can be used to react arbitrarily. For example, the controller 500 may issue a drive command in response to a recognizable manual feed in a particular direction.

The 3-D image sensor 450 can generate the following types of data: (i) depth maps, (ii) reflectance-based intensity images, and / or (iii) standard intensity images. The 3-D image sensor 450 may acquire such data by image pattern matching, flight time and / or phase delay shift measurement of light emitted from the source and reflected from the target.

In some embodiments, the inference or control software that can be run on a processor (eg, robot controller 500) uses a combination of algorithms that are run using the various data types generated by the sensor system 400. To do. The inference software processes the data collected from the sensor system 400 and outputs the data, for example, to make a navigation decision as to whether the robot 100 can move without colliding with an obstacle. By accumulating imaging data over time of the robot's siege, the inference software selects the perceived image (s) to improve the depth measurement of the 3-D image sensor 450. Efficient methods can be applied to the segments. This may include using appropriate time and space averaging techniques.

The reliability of performing collision-free robotic movements is (i) the confidence level built by high-level inference over time, and (ii) three main types of data for analysis, (a) depth images. , (B) Active Illumination Images, and (c) Ambient Illumination Images may be based on depth perception sensors. An algorithm that recognizes different types of data can be executed for each of the images acquired by the depth perception imaging sensor 450. Aggregated data can improve confidence levels compared to systems that use only one type of data.

The 3-D image sensor 450 may acquire an image containing depth and brightness data (eg, a sensor field of view portion of a room or work area) from a scene 100 around the robot containing one or more objects. .. The controller 500 may be configured to determine the occupation data of the object based on the reflected light captured from the scene. Further, in some embodiments, the controller 500 issues a drive command to the drive system 200 to bypass obstacles (ie, objects in the scene), at least in part, based on the occupation data. The 3-D image sensor 450 repeatedly captures the scene depth image to make a real-time decision by the controller 500 to navigate the robot 100 around the scene without colliding with any object in the scene. May be good. For example, the speed or frequency at which depth image data is acquired by the 3-D image sensor 450 may be controlled by the shutter speed of the 3-D image sensor 450. In addition, the controller 500 may receive an event trigger (eg, from another sensor component of the sensor system 400, such as proximity sensors 410, 420, which notifies the controller 500 of nearby objects or dangers). The controller 500 can increase the frequency with which the 3-D image sensor 450 captures the depth image in response to the event trigger, and the occupation information is acquired.

Referring to FIG. 12A, in some embodiments, the 3-D imaging sensor 450 includes a light source 1172 that radiates light into a scene 10 such as the surrounding area (eg, a room) of the robot 100. Further, the image pickup sensor 450 captures the reflected light from the scene 10 (for example, as a scene depth image) including the reflected light of the light emitted from the light source 1172, and an array of the image pickup unit 1174 (for example, the light sensing pixel 1174p). ) May also be included. In some embodiments, the imaging sensor 450 includes a light source lens 1176 and / or a detector lens 1178 to manipulate (eg, spec ring or focus) the emitted light and the received reflected light, respectively. .. The robot controller 500 or the sensor controller (not shown) communicating with the robot controller 500 is the object 12 in the scene 10 based on the image pattern matching and / or flight time characteristics of the reflected light captured by the imaging unit 1174. An optical signal is received from the imaging unit 1174 (for example, pixel 1174p) to determine the depth information.

FIG. 12B provides an arrangement 1200 that is an example of an operation for operating the image sensor 450. With additional reference to FIG. 12A, the actions are to radiate light to the scene 10 around the robot 100 (1202) and to reflect the radiated light from the scene 10 onto the imaging unit (eg, an array of light-sensing pixels). Including receiving (1204). The operation is that the controller 500 receives a photodetection signal from the imaging unit (1206) and uses the image data obtained from the photodetection signal to detect one or more features of the object 12 in the scene 10. (1208) further includes tracking the location of the detected feature (s) of the object 12 in the scene 10 using the image depth data obtained from the photodetection signal (1210). The operation is to emit light (1202), receive light reflection (1204), receive a light detection signal to increase the resolution of the image data or image depth data and / or to provide a confidence level. It may include repeating the actions of doing (1206), detecting an object feature (s) (1208), and tracking the position of an object feature (s) (12010) (1212). ..

Repeating the operation (1212) can be performed at relatively slow speeds (eg, slow frame speeds) due to the relatively high resolution , at intermediate speeds, or at high speeds with relatively low resolutions. The frequency of repeating movements (1212) may be adjustable by the robot controller 500. In some embodiments, the controller 500 may repeat the operation (1212) more or less frequently upon receiving the event trigger. For example, a perceived item in a scene increases the frequency of repeated movements (1212) to detect a potentially high object 12 (eg, a doorway, sill, or step) in scene 10. Can trigger an event to cause. In additional embodiments, the elapsed time event between the detected objects 12 can be inactive for a period of time (eg, inactive until caused by another event), which can reduce the frequency of repeated movements (1212). Become). In some embodiments, the action of detecting one or more features of object 12 in scene 10 (1208) is a relatively frequent repetitive action (1208) to increase the rate at which image depth data is acquired. Trigger a feature detection event that results in 1212). The relatively high acquisition speed of image depth data can enable relatively more reliable feature tracking within the scene.

The action also includes outputting navigation data (1214) to bypass the object 12 in the scene 10. In some embodiments, the controller 500 uses the output navigation data to issue a drive command to the drive system 200 to move the robot 100 to avoid collision with the object 12.

In some embodiments, the sensor system 400 detects a plurality of objects 12 in the scene 10 around the robot 100, and the controller 500 tracks the position of each of the detected objects 12. Controller 500 may create an occupation map of an object 12 in an area around the robot 100, such as a bounded area of a room. The controller 500 may use the image depth data of the sensor system 400 to match the scene 10 with a portion of the occupation map and update the occupation map at the location of the tracked object 12.

Referring to FIG. 12C, in some embodiments, the 3-D image sensor 450 includes a three-dimensional (3D) speckle camera 1300 that allows image mapping through speckle uncorrelation. The speckle camera 1300 emits a speckle pattern to the scene 10 (as a target area), a speckle radiator 1310 (for example, infrared, ultraviolet, and / or visible light) and an object 12 in the scene 10. Includes an imaging unit 1320 that captures images of speckle patterns on the surface.

The speckle radiating unit 1310 may include a light source 1312, such as a laser, that radiates a light beam onto the diffuser 1314 and on the reflector 1316 for reflection and thus projection as a speckle pattern onto the scene 10. The imaging unit 1320 may include an objective optical element 1322 that forms an image on an image sensor 1324 having an array of photodetectors 1326 such as a CCD or CMOS-based image sensor. The optical axes of the speckle radiating section 1310 and the imaging section 1320 are shown to be on the same straight line, but in the non-correlated mode, for example, the optical axes of the speckle radiating section 1310 and the imaging section 1320 are also non-correlated. It may be on the same straight line, while in the cross-correlation mode, for example, the imaging axis may deviate from the radiation axis.

The speckle radiating section 1310 radiates a speckle pattern to the scene 10, and the imaging section 1320 has a wide range of different object distances Z from the speckle radiating section 1310._nThe reference image of the speckle pattern in the scene 10 is captured in (for example, the Z axis can be defined by the optical axis of the imaging unit 1320). In the example shown, Z₁, Z₂, Z₃The reference image of the projected speckle pattern is captured on a continuous plane at each distance, which is different from the origin, such as the reference position labeled as. The distance ΔΖ between reference images may be set to a threshold distance (eg, 5 mm) or may be adjustable by controller 500 (eg, in response to a triggered event). The speckle camera 1300 allows the speckle pattern to be uncorrelated with the distance from the speckle radiating section 1310 and provides a reference image to be captured in order to perform distance measurement of the object 12 captured in the subsequent image. Store and index each radiation distance. For example, the reference distance Z where ΔΖ is adjacent₁, Z₂, Z₃Position Z, assuming it is approximately equal to the distance between, ..._AZ speckle pattern on object 12₂It can be correlated with the reference image of the speckle pattern captured in. On the other hand, for example, the speckle pattern on the object 12 of ZB is Z.₃Can be correlated with the reference image of. These correlation measurements provide an approximate distance of the object 12 from the origin. To map the object 12 in three dimensions, the speckle camera 1300 or controller 500 receiving information from the speckle camera 1300 can use local cross-correlation with the reference image that resulted in the closest match.

For example, other details and features in 3D image mapping using speckle ranging via triangulation or speckle cross-correlation using uncorrelation that may be combined with those described herein. Can be found in PCT Patent Application No. PCT / IL2006 / 000335, which is incorporated herein by reference in its entirety.

FIG. 12D provides an example arrangement 1400 for operating the speckle camera 1300. The operation includes radiating a speckle pattern to the scene 10 (1402) and capturing a reference image (eg, of the reference object 12) at a different distance from the speckle radiating section 1310 (1404). The operation further includes radiating a speckle pattern to the target object 12 in the scene 10 (1406) and capturing a target image of the speckle pattern on the object 12 (1408). The action is to compare the target image (of the speckled object) with a different reference image to identify the reference pattern that most strongly correlates with the speckle pattern on the target object 12 (1410) and within the scene 10. Further including determining the estimated distance range of the target object 12 (1412). This may include determining the primary speckle pattern on the object 12 and finding a reference image having the speckle pattern that most strongly correlates with the primary speckle pattern on the object 12. The distance range can be determined from the corresponding distance of the reference image.

The movement is optional, eg, 3D on the surface of the object 12 by local cross-correlation between the speckle pattern on the object 12 and the identified reference pattern to determine the position of the object 12 in the scene. Includes building a map (1414). This was identified as determining the primary speckle pattern on the object 12 to obtain a three-dimensional (3D) map of the object and as the primary speckle pattern on multiple regions of the object 12 in the target image. It may include finding each offset to and from the primary speckle pattern in the reference image. The use of solid state components for 3D mapping of scenes provides a relatively inexpensive solution for robot navigation systems.

Typically, at least some of the different distances are axially apart longer than the axial length of the primary speckle pattern at each distance. Comparing the target image with the reference image has the respective cross-correlation between the target image and at least some of the reference images, and the maximum cross-correlation with the target image. It may include selecting a reference image.

The movement may include repeating movements 1402 to 1412 or 1406 to 1412 (1416) and optionally movement 1414 to track the movement of the object 12 in the scene 10 (eg, continuously). Good. For example, the speckle camera 1300 may capture a continuous target image while the object 12 is moving for comparison with a reference image.

Other details and features in 3D image mapping using speckle ranging, which may be combined with those described herein, are incorporated herein by reference in their entirety, U.S.A. Patent No. 7,433,024, US Patent Application Publication No. 2008/0106746 with the name "Depth Mapping Using Projected Patterns", US Patent Application Publication No. 2010/0118123 with the name "Depth Mapping Using Projected Patterns", and the name "Modeling Of" Humanoid Forms From Depth Maps US Patent Application Publication No. 2010/0034457, Name "Depth Mapping Using Multi-Beam Illumination" US Patent Application Publication No. 2010/0020078, Named "Optical Designs For Zero Order" US Patent Application Publication No. 2009/0185274 for "Reduction", US Patent Application Publication No. 2009/0096783 for "Three-Dimensional Sensing Using Speckle Patterns", US Patent Application Publication for "Depth Mapping Using Projected Patterns" It can be found in 2008/0240502, and in US Patent Application Publication No. 2008/0106746, whose name is "Depth-Varying Light Fields For Three Dimensional Sensing."

Referring to FIG. 12E, in some embodiments, the 3-D imaging sensor 450 includes a 3D flight time (TOF) camera 1500 for acquiring depth image data. 3D The TOF camera 1500 communicates with the light source 1510, the complementary metal oxide semiconductor (CMOS) sensor 1520 (or charge-coupled device (CCD)), the lens 1530, and the light source 1510 and the CMOS sensor 1520. Includes control logic or camera controller 1540 (and / or robot controller 500) with. Light source 1510 may be a laser or light emitting diode (LED) with an intensity modulated by a periodic radio frequency signal. In some embodiments, the light source 1510 comprises a focusing lens 1512. The CMOS sensor 1520 may include an array of pixel detectors 1522, or other arrangements of pixel detectors 1522, on which each pixel detector 1522 can detect the intensity and phase of photon energy colliding on it. Good. In some embodiments, each pixel detector 1522 has a dedicated detector circuit 1524 for processing the detected charge output of the associated pixel detector 1522. The lens 1530 focuses the light reflected from the scene 10 containing one or more objects of interest 12 on the CMOS sensor 1520. The camera controller 1540 provides a series of operations that format the pixel data acquired by the CMOS sensor 1520 into depth maps and brightness images. In some embodiments, the 3D TOF camera 1500 also has an input / output (IO) 1550 (eg, communicating with the robot controller 500), memory 1560, and / or camera controller 1540 and / or pixel detector. It also includes a clock 1570 communicating with the 1522 (eg, detector circuit 1524).

Figure 12F provides an example arrangement 1600f for operating the 3D TOF camera 1500. The operation is to radiate a light pulse (eg, infrared, ultraviolet, and / or visible light) into scene 10 (1602f) and start time measurement of the flight time of the light pulse (eg, clock pulse of clock 1570). (By counting) (1604f) and including. The action involves receiving a reflection of light emitted from one or more surfaces of the object 12 in the scene 10 (1606f). The reflection is at a different distance Z from the light source 1510_nThen, it may be reflected from the surface of the object 12. The reflection is received on the pixel detector 1522 of the CMOS sensor 1520 through the lens 1530. The operation involves receiving the flight time of each light pulse reflection received on each corresponding pixel detector 1522 of the CMOS sensor 1520 (1608f). During the round-trip time-of-flight (TOF) of the optical pulse, the counter in the detector circuit 1523 of each pixel detector 1522 accumulates the clock pulse. The accumulation of a larger number of clock pulses represents a longer TOF and, therefore, a larger distance between the light reflection point on the imaged object 12 and the light source 1510. The operation further includes determining the distance between the reflecting surfaces of each received light pulse reflecting object 12 (1610f) and, optionally, constructing a three-dimensional object surface (1612f). In some embodiments, the motion includes repeating motions 1602f to 1610f (1614f) and, optionally, 1612f to track the movement of the object 12 within the scene 10.

Other details and features in 3D flight time imaging that may be combined with those described herein are incorporated herein by reference in their entirety, as referred to as "CMOS Compatible 3-". US Pat. No. 6,323,942 for "D Image Sensor", US Pat. No. 6,515,740 for "Methods for CMOS-Compatible Three-Dimensional Image Sensing Using Quantum Efficiency Modulation", and "Method and System to Enhance Dynamic Range Conversion Usable with" It can be found in PCT Patent Application No. PCT / US02 / 16621 of "CMOS Three-Dimensional Imaging".

In some embodiments, the 3-D imaging sensor 450 provides three types of information: (1) depth information (eg, from each pixel detector 1522 on the CMOS sensor 1520 to the corresponding position on scene 12). , (2) Ambient light intensity at each pixel detector position, and (3) Active illumination intensity at each pixel detector position. Depth information makes it possible to ensure that the position of the detected object 12 is tracked over time, especially with respect to the robot deployment site from the proximity of the object. Active illumination intensity and ambient light intensity are different types of brightness images. The active illumination intensity is captured from the reflection of active light (such as that provided by light source 1510) reflected from the target object 12. The ambient light image is that of the ambient light reflected from the target object 12. Both images both provide additional robustness, especially in poor lighting conditions (eg, too dark or excessive ambient lighting).

Image classification and classification algorithms may be used to classify and detect the position of the object 12 in the scene 10. The information provided by these algorithms, as well as the distance measurement information obtained from the imaging sensor 450, can be used by the robot controller 500 or other processing resources. The imaging sensor 450 is tuned based on the principle of flight time, including techniques for modulating the sensitivity of the photodiode to filter ambient light, and more specifically, reflected from scene 10. It can operate based on the detectable phase delay in the light pattern.

Robot 100 includes 1) mapping, location &amp; navigation, 2) object detection &amp; object avoidance (ODOA), 3) object search (eg for finding people), 4) gesture recognition (eg for companion robots). ), 5) Person &amp; Face Detection, 6) Person Tracking, 7) Monitoring of Object Manipulation by Robot 100, and Other Suitable Applications for Robot 100 Autonomous Movement, Using Imaging Sensor 450 May be good.

In some embodiments, at least one of the 3-D image sensors 450 is positioned above the ground on the robot 100 at a height of more than 1 foot or 2 feet. A volume point cloud imaging device (speckle or flight time) oriented so that the point cloud can be obtained (via the omnidirectional drive system 200) from the volume of space including the floor in the direction of movement of the robot. It may be a camera, etc.). In the embodiments shown in FIGS. 1 and 3, the first 3-D image sensor 450a is placed at a height greater than 1 or 2 feet above the ground on the base 120 (or approximately 1 or 2 feet above the ground). Aim along the forward drive direction F (eg, obstacle detection and obstacles) to capture an image of the volume including the floor (eg, volume point cloud) while positioning and driving (height). Can be (for avoidance). Mounted on the head 160 (eg, at a height of more than about 3 or 4 feet above the ground) so that skeletal recognition and demarcation points can be obtained from the spatial volume adjacent to the robot 100. A second 3-D image sensor 450b is shown. Controller 500 may run skeleton / digital recognition software to analyze the captured volume point cloud data.

Proper sensing of the object 12 using the imaging sensor 450, regardless of ambient light conditions, can be important. In many environments, lighting conditions range from direct sunlight to bright fluorescent lighting to hazy shadows and can result in significant changes in the surface texture and basic reflectance of object 12. Lighting can vary within a given position and from scene 10 to scene 10 as well. In some embodiments, the imaging sensor 450 is used to identify and resolve a person and an object 12 in all situations with a relatively small effect from ambient light conditions (eg, ambient light exclusion). Can be used.

In some implementations, the VGA resolution of the image sensor 450 is 640 pixels horizontal x 480 pixels vertical, however, other resolutions such as 320 x 240 (for example, for short range sensors) are possible as well. is there.

The imaging sensor 450 may include a pulsed laser and a camera iris that acts as a passband filter in the time domain to see the object 12 only within a specific range. The variable iris of the imaging sensor 450 can be used to detect objects 12a at different distances. In addition, for outdoor applications, higher power laser pulses can be emitted.

In some embodiments, the robot includes a sonar scanner 460 for acoustic imaging of the area surrounding the robot 100. In the embodiments shown in FIGS. 1 and 3, the sonar scanner 460 is placed on the front portion of the base body 120.

With reference to FIGS. 1, 3B, and 11B, in some embodiments, the robot 100 has a laser scanner or laser rangefinder 440 for redundant sensing, and a sonar proximity sensor 410j facing backwards for safety. Both of these are oriented parallel to the ground G. The robot 100 may include first and second 3-D image sensors 450a, 450b (depth cameras) to provide a strong sense of the environment surrounding the robot 100. The first 3-D image sensor 450a is mounted on the body 140 so as to point downward at a fixed angle to the ground G. By angling the first 3-D image sensor 450a downward, the robot 100 receives the high density sensor coverage in the area immediately or adjacent to the robot 100 that is associated with the short-term forward movement of the robot 100. .. The backward-facing sonar 410j provides object detection as the robot moves backwards. When backward movement is typical for the robot 100, the robot 100 includes a third 3D image sensor 450 pointing downwards and backwards to provide a high density sensor coverage range immediately after or adjacent to the robot 100. But it may be.

The second 3-D image sensor 450b is mounted on the head 160, which can be rotated and tilted via the neck 150. The second 3-D image sensor 450b can be useful for remote drive because it allows a human operator to see where the robot 100 is moving. The neck 150 allows the operator to tilt and / or rotate the second 3-D image sensor 450b to see both near and far objects. Rotating the second 3-D image sensor 450b increases the associated horizontal field of view. During high-speed movement, the robot 100 increases the overall or compound field of view of both the 3-D image sensors 450a, 450b, and gives the robot 100 sufficient time to avoid obstacles (more). The second 3-D image sensor 450b may be tilted slightly downwards (because high speed generally means less time to react to obstacles). At lower speeds, Robot 100 tilts the second 3-D image sensor 450b upwards or approximately parallel to ground G to track the individual that Robot 100 is intended to follow. You may let me. Further, while driving at a relatively low speed, the robot 100 can rotate the second 3-D image sensor 450b to increase its field of view around the robot 100. The first 3-D image sensor 450a can remain fixed (eg, moved relative to the base 120) while the robot is being driven to expand the robot's perceptual range. Absent).

In some embodiments, at least one of the 3-D image sensors 450 is at least 1 foot (about 30.5 cm) or 2 feet (about 61 cm) above the ground (or above the ground). To be positioned on the robot 100 (at a height of about 1 or 2 feet) and to be able to obtain a point cloud from the volume of space including the floor in the direction of movement of the robot (via the omnidirectional drive system 200). It can be a volume point cloud imaging device (such as a speckle or flight time camera) oriented towards. In the embodiments shown in FIGS. 1 and 3, the first 3-D image sensor 450a is positioned on base 120 at a height of at least 1 or 2 feet above the ground (eg, obstacle detection and obstacles). Aiming can be done along the forward driving direction F to capture an image of the volume including the floor (eg, volume point cloud) while driving (for object avoidance). So that it is possible to obtain skeletal recognition and demarcation points from the volume of space adjacent to Robot 100 (eg, at a height higher than about 3 feet above the ground (about 91 cm or 4 feet)). A second 3-D image sensor 450b mounted on the head 160 is shown. Controller 500 runs skeletal / digital recognition software to analyze the captured volume point group data. You may.

With reference to FIGS. 2 and 4A-4C again, the sensor system 400 is the total center of gravity CG of the robot 100._RIt may include an inertial measurement unit (IMU) 470 communicating with the controller 500 to measure and monitor the moment of inertia of the robot 100 with respect to.

Controller 500 may monitor any deviation in feedback from the IMU 470 from the threshold signal corresponding to normal uninterrupted operation. For example, if the robot begins to tilt away from its upright position, it may have been "tackled", otherwise disturbed, or someone suddenly added a heavy load. In these situations, it may be necessary to take emergency actions (including, but not limited to, avoidance maneuvers, recalibration, and / or issuance of audio / visual alerts) to ensure safe operation of Robot 100. ..

Since the robot 100 can operate in a human environment, it may interact with humans and operate in a space designed for humans (and robot constraints are not considered). Robot 100 can limit its drive speed and acceleration when in a crowded, constrained, or highly dynamic environment such as a cocktail party or a crowded hospital. However, while it is safe to drive the robot 100 at relatively high speeds, such as in a long empty corridor, one may encounter situations where someone can still suddenly slow down as they cross the robot's path of motion.

When accelerating from a stop, the controller 500 has its entire center of gravity CG to prevent the robot from tipping over._RThe moment of inertia of the robot 100 from may be considered. Controller 500 may use a model of its attitude, including its current moment of inertia. When the load is supported, the controller 500 has a total center of gravity CG_RThe load effect on the robot may be measured to monitor the moment of inertia of the robot's movement. For example, the body 140 and / or the neck 150 may include a strain gauge that measures strain. If this is not possible, the controller 500 applies a test torque command to the drive wheels 210 and uses the IMU470 to measure the actual linear and angular acceleration of the robot to determine the safety limits experimentally. May be good.

During sudden deceleration, the commanded load on the second and third drive wheels 210b, 210c (rear wheels) is reduced, while the first drive wheels 210a (front wheels) slide in the forward drive direction. , Support the robot 100. If the loads on the second and third drive wheels 210b, 210c (rear wheels) are asymmetric, the robot 100 may "swing", which reduces dynamic stability. An IMU470 (eg, a gyro) can be used to detect this sway and command the second and third drive wheels 210b, 210c to reorient the robot 100.

With reference to FIGS. 3-4C and 6A, in some embodiments, the robot 100 includes multiple antennas. In the embodiments shown, the robot 100 is both (although the antenna may be placed on any other part of the robot 100 such as legs 130, body 140, neck 150, and / or head 160). It includes a first antenna 490a and a second antenna 490b, both of which are located on the base 120. The use of multiple antennas provides robust signal reception and transmission. The use of multiple antennas provides robot 100 with multiple inputs and multiple outputs or MIMO, which is the use of multiple antennas for transmitters and / or receivers to improve communication performance. MIMO provides a significant increase in data throughput and link range without additional bandwidth or transmission power. It achieves this with higher spectral efficiency (more bits per second per hertz of bandwidth) and link certainty or versatility (reduced fading). Due to these characteristics, MIMO is an IEEE. It is an important part of modern wireless communication standards such as 802.11n (Wifi), 4G, 3GPP Long Term Evolution, WiMAX, and HSPA +. In addition, Robot 100 can act as a Wi-Fi bridge, hub, or hotspot for other nearby electronic devices. The mobility of Robot 100 and the use of MIMO can allow the robot to become a relatively very reliable Wi-Fi bridge.

MIMO can be subdivided into three main categories: precoding, spatial multiplexing or SM, and diversity coding. Precoding is a type of multi-stream beam formation and is considered to be all spatial processing that occurs at the transmitter. In (single layer) beam formation, the same signal is emitted from each of the transmission antennas, with appropriate phase (and sometimes gain) weighting, so that signal power is maximized at the receiver input. The advantage of beam formation is to increase the received signal gain and reduce the multipath fading effect by constructively adding signals originating from different antennas. In the absence of scattering, beam formation can result in a clear directional pattern. When the receiver has multiple antennas, transmission beam formation cannot maximize signal levels at all of the receiving antennas at the same time, and precoding with multiple streams can be used. Precoding may require knowledge of Channel State Information (CSI) in the transmitter.

Spatial multiplexing requires a MIMO antenna configuration. Spatial multiplexing divides a high-speed signal into multiple low-speed streams, each stream being transmitted from different transmission antennas on the same frequency channel. If these signals arrive at a receiver array with sufficiently different spatial evidence, the receiver can separate these streams into (nearly) parallel channels. Spatial multiplexing is a very powerful technique for increasing channel capacitance at higher signal-to-noise ratios (SNRs). The maximum number of spatial streams is limited by the smaller number of antennas in the transmitter or receiver. Spatial multiplexing can be used with or without knowledge of transmission channels. Spatial multiplexing can also be used for simultaneous transmission, known as spatial split multiple access. Good separability can be ensured by scheduling receivers with different spatial traces.

Diversity coding techniques can be used when the transmitter has no knowledge of channels. In the diversity method, a single stream is transmitted (unlike multiple streams in spatial multiplexing), but the signal is encoded using a technique called spatiotemporal coding. The signal is transmitted from the transmission antenna using full or near-orthogonal coding. Diversity coding utilizes independent fading on multiple antenna links to enhance signal diversity. There is no beam formation or array gain from diversity coding due to lack of channel knowledge. Spatial multiplexing can also be combined with precoding when the channel is known in the transmitter, or with diversity coding when the certainty of decoding is a trade-off.

In some embodiments, the robot 100 includes a third antenna 490c and / or a fourth antenna 490d, respectively, and a torso 140 and / or head 160 (see, eg, FIG. 3). In such cases, the controller 500 moves the antenna 490a-d (eg, by raising or lowering the body 140 and / or rotating and / or tilting the head 160, etc.). ) It is possible to determine the antenna arrangement that achieves the threshold signal level for robust communication. For example, the controller 500 can issue commands to raise the third and fourth antennas 490c, 490d by raising the height of the torso 140. The controller 500 can also issue commands to rotate the head 160 and / or to further orient the fourth antenna 490d relative to the other antennas 490a-c.

Referring to FIG. 13, in some embodiments, the controller 500 executes a control system 510, including a control arbitration system 510a communicating with each other and a behavior system 510b. The control arbitration system 510a allows the application 520 to be dynamically added to and removed from the control system 510, and the application 520 has a robot 100 for each without having to know about any other application 520. Promote to be able to control. In other words, the control arbitration system 510a provides a simple priority control mechanism between application 520 and resource 530 of robot 100. Resource 530 includes any on-board or controllable device communicating with drive system 200, sensor system 400, and / or controller 500.

The application 520 can run simultaneously on the robot 100 (eg, a processor) and can be stored or communicated in the memory of the robot 100 so as to control the robot 100 at the same time. Application 520 may access behavior 600 of behavior system 510b. The independently deployed application 520 dynamically at run time and shares the robot resources 530 of the robot 100 (eg, drive system 200, arms (s), heads (s), etc.). Can be combined as follows. A low-level policy is implemented to dynamically share the robot resource 530 between applications 520 at run time. The policy determines which application 520 has control of the robot resource 530 required by that application 520 (eg, the priority hierarchy among the applications 520). Application 520 can be dynamically started and stopped, and can run completely independently of each other. The control system 510 also allows for complex behaviors 600 that can be combined together to support each other.

The control arbitration system 510a includes one or more resource controllers 540, a robot manager 550, and one or more control arbiters 560. These components do not have to be in a common process or computer and do not have to start in any particular order. The resource controller 540 component provides application 520 with an interface to the control arbitration system 510a. There is an instance of this component for every application 520. The resource controller 540 removes and encapsulates the complexity of authentication, distributed resource control arbiters, command buffering, and so on. The robot manager 550 coordinates the prioritization of the application 520 by controlling which application 520 has exclusive control of any of the robot resources 530 at any particular time. Since this is the central coordinator of information, there is only one instance of Robot Manager 550 per robot. The robot manager 550 tracks and records the resource control arbiter 560, which implements a priority policy and provides hardware control, with a linearly prioritized order of the resource controller 540. The control arbiter 560 receives commands from each application 520, generates a single command based on application priority, and publishes it to its associated resource 530. The control arbiter 560 also receives state feedback from its associated resource 530 and returns it to application 520. The robot resource 530 may be a network of functional modules (eg, actuators, drive systems, and groups thereof) having one or more hardware controllers. The command of the control arbiter 560 is specific to the resource 530 that performs a particular operation.

Configure a kinetic model 570 that can be run on controller 500 to calculate the cross product of the center of gravity (CG), moment of inertia, and inertia of various parts of the robot 100 to assess the current state of the robot. be able to. The kinetic model 570 may also model the shape, weight, and / or moment of inertia of these components. In some embodiments, the kinetic model 570 is located on the robot 100 and communicates with the controller 500 to calculate the various centroids of the robot 100, one (eg, accelerometer and /). Or communicate with the moment of inertia unit 470 (IMU) of the gyro) or its part. The controller 500 can use the kinetic model 570, along with other programs 520 or behavior 600, to determine the operating limits of the robot 100 and its components.

Each application 520 has an action selection engine 580, a resource controller 540, one or more behavior 600 connected to the action selection engine 580, and one or more action models 590 connected to the action selection engine 580. .. The behavior system 510b provides predictive modeling, allowing the behavior 600 to collaboratively determine the robot's motion by evaluating the expected outcome of the robot's motion. In some embodiments, the behavior 600 is a plug that provides a hierarchical state complete evaluation function that connects sensor feedback from multiple sources, along with a priori limits and information, to evaluation feedback in the robot's permissible movements. It is an in-component. Behavior 600 is pluggable into application 520 (eg, inside or outside application 520), so it can be removed and added without modifying any other part of application 520 or control system 510. Can be done. Each behavior 600 is an independent policy. In order to make the behavior 600 more powerful, it is possible to attach the outputs of the plurality of behaviors 600 together to different inputs so that they can have complex combination functions. Behavior 600 is intended to provide a manageable portion of the overall perception of Robot 100.

The motion selection engine 580 is a cooperative element of the control system 510, and executes a high-speed optimum motion selection cycle (prediction / correction cycle) that searches for the optimum motion by considering the inputs of all the behaviors 600. The motion selection engine 580 has three stages: nomination, motion selection search, and completion. At the nomination stage, each behavior 600 is notified that a motion selection cycle has begun and is provided with a cycle start time, current state, and robot actuator space limits. Based on internal policies or external inputs, each behavior 600 determines whether it wants to be involved in this behavior selection cycle. During this stage, a list of active behavior primitives is generated, and this input influences the choice of commands to be executed on the robot 100.

In the motion selection search stage, the motion selection engine 580 produces feasible results from the available motion space, also referred to as the motion space. The motion selection engine 580 provides a pool of feasible commands (within limits) and a corresponding result as a result of simulating the behavior of each command at different time steps in the future target period. To use. The motion selection engine 580 calculates a preferable result based on the result evaluation of the behavior 600, sends a command corresponding to the control arbitration system 510a, and notifies the motion model 590 of the command selected as feedback.

At the completion stage, the commands corresponding to the highest collaborative score results are combined together as general commands presented to the resource controller 540 for execution on the robot resource 530. Optimal results are provided for Active Behavior 600 as feedback to be used in future evaluation cycles.

The sensor signal received from the sensor system 400 can generate an interaction with one or more behaviors 600 to perform an operation. For example, using control system 510, controller 500 makes coordinated movement of each robot component to avoid collisions with itself and any object around robot 100 known to robot 100. To achieve efficiency, the movement (or movement command) of each robot component (eg, motor or actuator) is a set of possible movements or movements of the corresponding movement space (eg, component of that particular element). ) To select from. As described in US Patent Application No. 61 / 305,069, filed February 16, 2010, which is incorporated herein by reference in its entirety, the Controller 500 is a robot such as the Ether IO Network. Coordinated commands can be issued on the network.

The control system 510 may provide the adaptive speed / acceleration of the drive system 200 to maximize the stability of the robot 100 in different configurations / positions as the robot 100 maneuvers around the area ( For example, through one or more behaviors 600).

In some embodiments, the controller 500 issues commands to the drive system 200 that propels the robot 100 according to the direction of travel and speed settings. One or more behaviors 600 can be selected to be executed by one to deal with obstacles (alone or in combination with other commands as general robot commands), predicting feasible commands. The signal received from the sensor system 400 may be used for evaluation. For example, the signal from the proximity sensor 410 may cause the control system 510 to change the commanded speed or direction of travel of the robot 100. For example, a signal from the proximity sensor 410 by a nearby wall can result in the control system 510 issuing a deceleration command. In another case, the collision signal from the contact sensor (s) due to the encounter with the chair may cause the control system 510 to issue a command to change the direction of travel. In other cases, the speed setting of the robot 100 may not be reduced in response to the contact sensor, and / or the direction of travel setting of the robot 100 may not be changed in response to the proximity sensor 410.

The behavior system 510b comprises a speed behavior 600 (eg, a behavior routine that can be executed on a processor) configured to adjust the speed setting of the robot 100 and a progression configured to change the travel direction setting of the robot 100. Orientation behavior 600 and may be included. The velocity and direction of travel behavior 600 may be configured to perform simultaneously and independently of each other. For example, the velocity behavior 600 may be configured to poll one of the sensors (eg, a set (s) of proximity sensors 410, 420), and the traveling direction behavior 600 may be configured as another sensor (eg, a set of proximity sensors 410, 420). For example, it may be configured to poll the dynamic uplift sensor).

With reference to FIGS. 13 and 14, the behavior system 510b is configured to respond to user 15 touching the torso 140 for remote control (eg, guiding the robot 100), torso touch remote. Operational behavior 600a (eg, behavior routines that can be executed on the processor) may be included. The torso touch remote control behavior 600a can be activated when the sensor system 400 detects that the torso has been in contact (eg, human contact) for at least a threshold period (eg, 0.25 seconds). .. For example, exercise and / or contact associated with the corresponding top panel 145t, bottom panel 145b, front panel 145f, back panel 145b, right panel 145r, and left panel 145l of the body body 145 communicating with the controller 500. Sensors 147t, 147b, 147f, 147r, 147r, 147l can detect motion and / or contact with their respective panels, as shown in Figures 6B and 6C. Once activated, the torso touch remote control behavior 600a receives the direction of contact force (eg, as sensed and calculated from the elliptical position of the touch) and local X / Y coordinates (using holononomic mobility). Issue a speed command to the drive system 200 at. Obstacle detection and obstacle avoidance behavior may be turned off while the body touch remote control behavior 600a is active. When the sensed touch position, force, or direction changes, the body touch remote control behavior 600a changes the speed command to correspond to the sensed contact force direction. The torso touch remote control behavior 600a may execute a stop routine when the sensor system 400 no longer senses contact with the robot 100 for a threshold period (eg, 2 seconds). The stop routine is about 0. When the sensor system 400 no longer senses contact with the robot 100 (eg, with the torso 140). The drive system 200 may stop driving after 5 seconds. The torso touch remote control behavior 600a can provide a delay in stopping the robot 100 so as to allow movement of the touch point without having to wait for a trigger period.

The body touch remote control behavior 600a may issue a support drive command to the drive system 200 that allows the user to press the robot 100 while receiving drive support from the drive system 200 (eg, the robot on its own). A partial speed command) that cannot move 100, but can help the user move the robot 100).

The torso touch remote control behavior 600a is a touch sensor system 480 (eg, buttons, capacitance sensors, etc.), of which a portion thereof may be placed on the torso 140 (and elsewhere on the robot 100, such as the head 160). A sensor signal may be received from a contact sensor or the like). The torso touch remote control behavior 600a is 3 to 5 feet above the ground G to position at least a portion of the touch sensor system 480 at a height accessible to the typical user. H_TThe body 140 may be positioned with.

In some embodiments, the body touch remote control behavior 600a recognizes the robot 100 and the user touch for placing a particular posture. For example, when the user 15 pushes down on the torso 140, the sensor system 400 detects a downward force on the torso 140 and sends a corresponding signal to the controller 500. The torso touch remote control behavior 600a receives a downward force instruction to the torso 140 and tells the control system 510 that the height of the leg 130 is H._LReduced, thereby increasing the height H of the torso 140_TIssue a command to reduce. Similarly, when the user 15 pushes / pulls up the torso 140, the torso touch remote control behavior 600a receives an upward force instruction from the sensor system 400 to the torso 140 and the leg to the control system 510. Part 130 length H_LIncreased, thereby increasing the height H of the torso 140_TIssuance a command to increase.

When the user 15 pushes, pulls, and / or rotates the head 160, the body touch remote control behavior 600a gives the user action instructions from the sensor system 400 (eg, strain gauge on the neck 150 / It may be received (from the motion / contact sensor 165) and respond by having the control system 510 move the head 160 accordingly and then issue a command to hold the posture.

In some embodiments, the robot 100 provides passive resistance and / or active support to user manipulation of the robot 100. Legs to user operation of robot 100 so that, for example, motors 138b, 152, 154 provide operation feedback as well as assistance for moving relatively heavy components such as elevation of body 140. Activate passive resistance and / or active support for 130 and neck 150. This allows the user to move various robot components without having to bear the entire weight of the corresponding components.

The behavior system 510b may include a tap attention behavior 600b (eg, a behavior routine that can be executed on a processor) that is configured to focus the attention of the robot 100 towards the user. The tap attention behavior 600b is when the sensor system 400 detects that the torso 140 (or any other part of the robot 100) has been in contact (eg, human contact) for at least a threshold period (eg, 0.25 seconds). In addition, it may be activated. Further, the tap attention behavior 600b may only become active when the body touch remote control behavior 600a is inactive. For example, a perceived touch on the torso 140 for 0.2 seconds would not trigger the torso touch remote control behavior 600a, but would trigger the tap attention behavior 600b. The tap caution behavior 600b may use the contact position on the torso 140 to tilt and / or rotate the head 160 to see the user (via the actuation of the neck 150). When the head 160 reaches a position looking in the direction of the touch position, the stop criterion of behavior 600b can be reached.

In some embodiments, the behavior system 510b is configured to stop the drive system 200 from driving (eg, stop the robot 100), the tap stop behavior 600c (eg, can be performed on a processor). Behavior routine) is included. The tap stop behavior 600c causes the sensor system 400 to detect that the body 140 has been touched (eg, human contact) and issues a zero speed drive command to the drive system 200 to cancel any previous drive command. May be activated when you do. If the robot is driven and the user wants to stop it, the user can tap the body 140 (or any other part of the robot 100) or the touch sensor. In some embodiments, the tap stop behavior 600c can only be activated if higher priority behaviors such as the torso touch remote control behavior 600a and the tap attention behavior 600b are not active. The tap stop behavior 600c may terminate when the sensor system 400 no longer detects a touch on the torso 140 (or elsewhere on the robot 100).

In some embodiments, the Robot 100 is an intermediary security device 350, also known as a bridge, to allow communication between the Webpad 310 and the Controller 500 (and / or other components of the Robot 100). 9) is included. For example, the bridge 350 may convert the communication of the web pad 310 from the web pad communication protocol to a robot communication protocol (eg, Ethernet® with gigabyte capacity). The bridge 350 may authenticate the web pad 310 and provide a communication conversion between the web pad 310 and the controller 500. In some embodiments, the bridge 350 includes an approval chip 352 that approves / validates any communication traffic between the web pad 310 and the robot 100. The bridge 350 may notify the controller 500 when it checks the approved web pad 310 in an attempt to communicate with the robot 100. After approval, the bridge 350 may notify the web pad 310 of the communication approval. The bridge 350 may be located on the neck 150 or head (as shown in FIGS. 2 and 3), or elsewhere on the robot 100.

Session Initiation Protocol (SIP) is an IETF-defined signaling protocol that is widely used to control multimedia communication sessions such as voice and video calls over the Internet Protocol (IP). The protocol can be used to generate, modify, and terminate two-party (unicast) or multiple joint (multicast) sessions that include one or more media streams. Modifications can involve changing addresses or ports, inviting more participants, and deleting or adding media streams. Other viable application examples include video conferencing, streaming multimedia distribution, instant messaging, presence information, file transfer, and the like. Voice over Internet Protocol (Voice over IP, VoIP) is part of a set of methods, communication protocols, and transmission technologies for delivering voice communications and multimedia sessions over Internet Protocols (IPs) such as the Internet. Other terms that are frequently encountered and often used as synonyms with VoIP are IP telephony, Internet telephony, voice over broadband (VoBB), broadband telephony, and broadband telephones.

FIG. 15 provides an example of telephone communication, including a dialogue with the bridge 350 to initiate and perform communication through the robot 100. Phone A's SIP calls the SIP application server. SIP calls the VoIP dial function to send an HTTP Post request to the VoIP web server. The HTTP Post request may behave like a callback function. The SIP application server sends a ring to phone A to indicate that the call has begun. The VoIP server initiates a call over the PSTN to the callback number contained in the HTTP Post request. The callback number terminates on the SIP DID provider, which is configured to send the callback to the SIP application server. The SIP application server matches the incoming call with the outgoing call on phone A and answers both calls with an OK answer. Media session with phone A and SIP Established with the DID provider. Phone A may hear the artificial ringing tone generated by VoIP. Once VoIP verifies that the callback process has been answered, it initiates a PSTN call (via the bridge 350) to a destination such as Robot 100. The robot 100 responds to the call, and the VoIP server bridges the media from the SIP DID provider with the media from the robot 100.

Figures 16A-16D show the robot 100 (or a portion of the controller 500 or drive system 200, etc.), the computing device 310 (detachable or fixed to the head 160), and the cloud 1620 (cloud computing). To provide an example schematic of robotic system architectures 1600, 1600a-d, which may include (for) and portal 1630.

The robot 100 includes mobility (for example, a drive system 200), a reliable, safe and stable robot intelligence system such as a control system 510 (Fig. 13) executed on the controller 500, a power supply 105, and a sensing system 400. , A variety of core robot features can be provided that may include optional operations using a manipulator communicating with the controller 500. The control system 510 can provide head and speed control, body attitude control, navigation, and core robot applications. The sensing system 400 includes vision (eg, via camera 320), depth map imaging (eg, via 3-D imaging sensor 450), collision detection, obstacle detection and obstacle avoidance, and / or (. For example, inertial measurement (via the inertial measurement unit 470) can be provided.

The computing device 310 is a portable electronic device such as a tablet computer, a telephone or a mobile information terminal, or a tablet or display having no data processing power (for example, a tablet that acts as a monitor for an atomic scale PC in a robot body 110). It may be. In some embodiments, the tablet computer may have a touch screen for displaying the user interface and receiving user input. The computing device 310 provides software applications for security, drug compliance, telepresence, behavioral guidance, social networking, active alarms, home management, etc. (eg, stored in memory and run on a processor). You may run one or more robot applications 1610 that may include. The computing device 310 may provide communication capabilities (eg, secure wireless connectivity and / or cellular communications), precision application development tools, voice recognition, and personal or object recognition capabilities. Computing device 310, in some examples, Google, Utilize an interaction / COMS-equipped operating system such as Android provided by Inc., iPad OS provided by Apple, Inc., other smartphone operating systems, or special robot operating systems such as RSS A2.

The Cloud 1620 provides cloud computing and / or cloud storage capabilities. Cloud computing may provide Internet-based computing, which allows shared servers to provide resources, software, and data to computers and other devices on demand. For example, a cloud 1620 is a cloud computing service that includes at least one server computing device that can include a service abstraction layer and a hypertext transfer protocol wrapper on a server virtual machine instantiated on it. May be good. The server computing device may be configured to parse HTTP requests and send HTTP responses. Cloud computing may be a technology that uses the Internet and a central remote server to maintain data and applications. Cloud computing can allow users to access and use application 1610 without installation and access personal files on any computer using Internet access. Cloud computing enables relatively efficient computing by concentrating memory, memory, processing, and bandwidth. The Cloud 1620 can provide scalable on-demand computing power, storage, and bandwidth while reducing robot software requirements (eg, by freeing up CPU and memory usage). Robot connectivity to the cloud 1620 enables automatic data collection of robot movements and usage history without requiring the robot 100 to return to the base station. In addition, continuous data collection over time can result in a wealth of data that can be data mined for marketing, product development, and support.

The cloud storage device 1622 can be a model of a network computer data storage device in which data is stored on multiple virtual servers, generally hosted by a third party. By providing communication between Robot 100 and the Cloud 1620, the information collected by Robot 100 can be safely viewed by authorized users via a web-based information portal.

Portal 1630 may be a web-based user portal for collecting and / or providing information such as personal information, housing status information, and robot status information. The information can be integrated with third party information to provide additional functions and resources to the user and / or the robot 100. Robot system architecture 1600 can facilitate positive data collection. For example, application 1610 running on computing device 310 collects data and reports about actions performed by robot 100 and / or individuals or environments viewed by robot 100 (using sensing system 400). You may. This data can be a unique characteristic of Robot 100.

In some embodiments, Portal 1630 is a personal portal website on the World Wide Web. Portal 1630 may provide personalization capabilities and routes to other content. Portal 1630 may use distributed applications, different numbers and types of middleware and hardware to serve from several different sources. In addition, the Business Portal 1630 can share collaborative behavior in the workplace and provide content that can be used on multiple platforms such as personal computers, personal digital assistants (PDAs), and mobile / mobile phones. Good. Information, news, and updates are examples of content that can be delivered through portal 1630. The personal portal 1630 may relate to any particular topic, such as providing information to friends on social networks or providing links to external content that may be useful to others.

"High density data" vs. "low density data" and "high density features" vs. "low density features" are referred to herein with respect to spatial datasets. Without limiting or reducing the meaning as one of ordinary skill in the art would interpret such terms to mean, "high density" vs. "low density" generally means many data point pairs per spatial representation. It means a small number of data points, and in particular, it may mean the following.

(i) In the context of 2-D image data, or 3-D "images" containing 2-D data and ranges, "high density" image data is almost completely populated or lost and /. Or it contains image data that can be rasterized into pixels with virtually no arch factor from the original image capture (including substantially uncompressed, unprocessed, or undegraded compressed images), while "low density". Images are quantized, sampled, degraded compressed, vectorized, segmented (eg to superpixels, nodes, edges, surfaces, points of interest, boxels), or otherwise captured originally. The fidelity from is significantly reduced, or it must be interpolated when rasterized to pixels to represent the image.

(ii) In the context of 2-D or 3-D features, "high density features" are all features that can be detected and recorded, with almost unconstrained data input until the resolution of the detection approach, and / Or a feature recognized by a detector that is recognized to collect many features (HOG, wavelet) across sub-images, a "low density feature" is a feature input, lateral suppression, and / or feature. In the number of choices, the number can be intentionally constrained and / or recognized by a detector that is recognized to identify a limited number of isolated points (Harris corners, edges, Shi-Tomasi) in the image. ..

Regarding the 3-D environment structure, the robot 100 may acquire an image such as a high-density image 1611 of the scene 10 around the robot 100 while moving around the work surface 5. In some embodiments, the robot 100 uses a camera 320 and / or an imaging sensor 450 (eg, a volume point cloud imaging device) to acquire a high density image 1611. The controller 500 communicating with the camera 320 and / or the sensor 450 provides information 1613 such as accelerometer data tracking, mileage measurement data and / or other data from the sensor system 400 along with the time stamp with the high density image 1611. It can be associated (eg, high density image 1611 is annotated or tagged with data). In some embodiments, the robot 100 captures the streaming sequence 1615 of the high density image 1615, annotates the high density image sequence 1615 with annotation data 1613, and provides the annotated high density image sequence 1615a. Robot 100 periodically transmits image data 1601 (eg, via controller 500 or web pad 310) to cloud storage 1622, which can store a potentially very large image data set 1603 over working hours. You may. The image data 1601 is tagged data such as raw sensor data (eg, point cloud or signal or high density image sequence 1615) or annotated high density image sequence 1615a (eg Java Script Object). It may be a data object having characteristics or attributes using a Notation (JSON) object or the like). The cloud service 1620 processes the received image data 1601 (eg, high density image sequence 1615 or annotated high density image sequence 1615a) and transfers the processed dataset 1617 to robot 100, eg controller 500 and / or webpad. You may reply to 310. Robot 100 may issue drive command 1619 (eg, via controller 500 or web pad 310) to drive system 200 based on received processed dataset 1617 to operate around scene 10. Good.

After the threshold period, or after the threshold amount of image data 1601, 1603 has been accumulated in the cloud storage device 1622, the cloud service 1620 processes the image data set 1603 to produce a high density of scene 10 (environment) 3- D-map or model 1605, then simplify this high-density 3D map or model 1605 to be a 2-D map with height data at each point (eg, similar to a 2-D terrain map) 2 -D You may perform one of various offline methods to make the height map 1607. In some embodiments, 2-D height map 1607 is a terrain map with X and Y coordinates along with Z data. Each X, Y coordinate may have one or more Z points (ie, height data). Unlike high density 3-D maps, which can have many Z points (eg, hundreds or thousands of Z points) for each X, Y coordinate, the 2-D height map 1607 has 2 to 20 It may have fewer Z points than the threshold for each X, Y coordinate, such as a point in between (eg, 10). The 2-D height map 1607, which is derived from the 3-D map of the indoor table, is the first Z point on the bottom surface of the table surface for each X and Y coordinate along the table, and the second on the top surface of the table surface. Can indicate the Z point of. This information allows the robot 100 to determine if it can pass under the table surface. By reducing the Z points from the high density dataset in the continuous range of the Z points for each X, Y coordinate to the low density dataset of the selected number of Z points indicating the detected object 12, the robot 100 is clouded. A 2-D height map 1607 can be received, which has a size relatively smaller than the 3-D map used by service 1620. This, in turn, allows Robot 100 to store a 2-D height map 1607 on local memory with a practical and cost-effective size compared to the scalable memory space available for cloud service 1620. Allows you to. Robot 100 is a navigation system for future work within Scene 10.

Further methods and features of 3-D map data comparison are incorporated herein by reference in their entirety, "Multi-Level Surface Maps For Outdoor Terrain Mapping" by R. Triebel, P. Pfaff, and W. Burgard. and Loop Closing , disclosed at the IEEE / RSJ International Conference on Intelligent Robots and Systems, 2006.

The Cloud 1620 provides Robot 100 with on-demand expansion of resources (eg, computation, processing, memory, etc.) that would otherwise not be practical or cost effective on Robot 100. For example, the cloud 1620 extends to a first size for storing and / or processing a relatively large amount of data 1601, which can only be used for a short period of time, then destroyed, and then shrunk again to a second size. It is possible to provide an expandable cloud storage device 1622. In addition, the Cloud 1620 can provide computer processing power to perform relatively complex computations or "brute force" algorithms that may not otherwise be possible on the robot. By moving the computer processing power and memory to the scalable cloud 1620, the robot 100 can use the controller 500, which has relatively little computing power and memory, thus providing a cost-effective solution. .. In addition, Robot 100 may perform real-time tasks (on controller 500 or webpad 310), such as obstacle avoidance, while passing non-real-time and non-time-dependent tasks to the cloud 1620 for processing and subsequent recovery. Good.

The Cloud 1620 provides one or more filters for processing the image dataset 1603 into a 3-D representation (eg, bundle adjustment, RANSAC, expected value maximization, SAM, or other 3-D structure estimation algorithm). You may do it. Once processed and the high density 3-D map 1605 is created or updated, the image dataset 1603 can be destroyed from cloud storage 1622, freeing resources and allowing the cloud 1620 to expand accordingly. Can be done. As a result, the robot 100 does not require any built-in storage or processing to handle the storage and processing of the image dataset 1603 due to the use of cloud-based resources. The cloud 1620 may return the processed navigation data 1601 or map 1607 (eg, compressed 2-D height map) to robot 100, which is then used for relatively simple localization and navigation processing. can do.

Further methods and features of 3-D reconstruction are incorporated herein by reference in their entirety, 3D Models From Extended Uncalibrated Video Sequences: Addressing Key-frame selection and projective drift by J. Repko and M. . Pollefeys; disclosed at Fifth International Conference on 3-D Digital Imaging and Modeling, 2005.

With respect to floor classification, robot 100 may acquire an image of work surface 5 while maneuvering on it around scene 10. Controller 500 may receive the image and execute an object detection routine for object detection and obstacle avoidance (ODOA). In some embodiments, the controller 500 associates information such as accelerometer data tracking, mileage measurement data, and / or other data from the sensor system 400 with the image (eg, the image in data). (Tag). Images can capture falling objects, tufts of rugs, socks on rugs, etc. As in the previous embodiment, the robot 100 can stream the image data 1601 to the cloud storage device 1622 while operating for subsequent batch processing. When the robot encounters an undesired event (ie, accidental collision), a special "danger" tag is inserted into the dataset so that the data moment before the danger may be identified for the learning algorithm. Will be done. Once the annotated image dataset and associated tag 1603 are accumulated (possibly along with datasets from many other robots 100), the parallel cloud host 1620 is, for example, a temporally leading danger tag. It may be invoked to process the annotated image dataset 1603 using a monitored learning algorithm that calculates the "dangerous image" category from many images in the real environment. Once the Danger Image Category model 1609 has been trained, the parameters of that model 1609 (small amount of data) can be downloaded back to many different robots 100. Therefore, the entire 100 robot corps can learn the elements of the environment online. Learning methods applicable to this method include genetic algorithms, neural networks, and support vector machines. All of these can be too complex and occupy so much memory that the low cost Robot 100 can operate online (ie, on a local robot processor).

The "classifier" 1625 typically employs iterative training algorithms to minimize error functions, optimize / maximize cost functions, or use training data to otherwise improve performance. It is a machine learning algorithm. Typically, the algorithm has several parameters whose values are learned from the training data. The three types include monitored regression, monitored classification, and unmonitored classification via clustering or dimension regression. Examples of classifier 1625 include support vector machines (SVMs) of various types and kernels (using gradient descent or other cost function minimization techniques), naive Bayes classifiers, logistic circuits, AdaBoost, and K-nearest neighbors. ("K-NN") and / or K-NN regression, neural networks, random forests, and linear models. A "classifier" can be represented by vectors, matrices, "descriptors", or other datasets, such as orientation gradient histograms (HOGs), shape context histograms (SCHs), color patches, textures. Identified by scale-invariant feature descriptors from patches, brightness patches, SIFT, SURF, or equivalents, affine-invariant histograms such as MSER, labeled superpixels or segments in images, Hough transforms, or RANSAC line detection. Classify the "features" to be obtained, and other features that can be combined with them.

The classifier 1625 can be run online from photographs alone to help robot systems avoid dangers in their environment. Once the model parameters have been determined, the image dataset 1603 stored in the cloud storage device 1622 can be discarded. This example can be combined with the previous example, where determination of the 3-D structure of the environment allows identification of traversable areas for training classification algorithms. One learning technique example is incorporated herein by reference in its entirety, Long-Term Learning Using Multiple Models For Outdoor Autonomous Robot Navigation, by Michael J. Procopio, Jane Mulligan, and Greg Grudic, 2007. Disclosure at the IEEE International Conference on Intelligent Robots and Systems.

Streaming is generally distinguished from dispatching batch processing tasks (which can occur at very high data rates) or penetrating images in a delayed manner, with packets being transmitted and received. Means that an image sequence is transmitted, with or without buffering, with or without buffering, with or without some relation to the order and speed at which the image sequences were captured in real time, whether or not they are sorted between. .. The term "streaming" dispatches batch processing tasks to distinguish between the speed at which different data is moved within and between entities in a robotic system, based on real-world bandwidth, processing, and storage constraints. It is distinguished from "that".

With reference to FIGS. 16D and 16E, the robot 100 at a practical cost level may have limitations on the robot's bandwidth, computation, and storage. By annotating image 1611 with locally processed data 1613 (ie, data from the robot sensor system 400), the robot 100 is very infrequently (eg, every few seconds instead of many times per second). (Times) Image 1611, high-density image sequence 1615, 1615a, or image data 1601 can be transmitted and the upstream bandwidth can be dramatically reduced compared to continuous streaming of data. In addition, the controller 500 and / or the web pad 310 can store the image 1611 and annotation data 1613 on the robot 100 when the wireless connectivity is inaccessible, and the wireless connectivity is accessible. It can include a local cache (memory) that can sometimes be transmitted. Buffering is a complete dataset 1601 when the robot 100 collects room-equivalent data, for example, in a room with poor signal reception, and once wireless connectivity is available again, such as in an adjacent room. , Allows 1603 to still be transmitted. Therefore, data management on the robot 100 may be necessary to adapt to various communication environments.

As the mobile robot is guided, oriented, or automatically navigated throughout the scene 10 (eg, home, office, etc.), one or more cameras 320 and / or imaging sensors 450 become robot 100. One or more sequences 1615 of image 1611 may be acquired, respectively, corresponding to the fields of view acquired in the posture (position and orientation) of the robot 100 along the trajectory of the robot 100. A timer (for example, on the controller 500) provides a reference time stamp for each image 1611, which is associated with annotation 1613, which corresponds to position, localization, speed or acceleration, sensor-based orientation, etc. May be good. At least some images 1611 may be annotated with information, and all images 1611 may be annotated with a time stamp or other metadata that reflects the robot state, image state, and the like. Image 1611 can be captured at some real-time capture rate, which does not have to be regular (eg, image 1611 is, for example, processing availability, bandwidth availability, time, task, purpose in the current image and It may be captured at an adaptive rate based on the density of features to be captured).

The camera 320 and / or the imaging sensor 450 is a hardware encoder, a high-speed bus available for memory, an internal processor, and a storage device for storing and storing images 1611 captured at a relatively high speed (eg, flash memory or). It may be communicating with a hard disk drive). Image 1611 that stores one or more machine visual algorithms to create a model, such as a set of parameters for the image classifier 1625, which itself does not consume more resources than can be supported on Robot 100. Can be applied to. However, there are some obstacles to overcome.

(1) Typically, the identification of at least some training data (eg, image 1611) or annotation 1613 representing the features to be modeled should be applied prior to significant image compression.

(2) Many algorithms and heuristics can be extended and adapted to the relatively large distributed computer infrastructure instantiated by cloud services to complete modeling in a short period of time, which spans the public Internet. Requires image 1611 to reach the target computer infrastructure.

(3) The bandwidth available to transport relatively large datasets is (i) Robot 100, which is mobile with radio bandwidth typically occupied by other traffic, and (ii). Bandwidth can be limited in the upstream direction Limited by internet access.

At least to overcome these obstacles, a high density image 1611 can be transmitted from the robot 100 to the local server 1640 (eg, wirelessly), at a local transmission speed that is relatively slower than the real-time capture speed. It may be a wired network with a storage device. The local server 1640 can buffer the high density image 1611 and annotate it with annotation 1613. In addition, local server 1640 may accumulate annotated high density image sequences 1615a for subsequent transmission to cloud service 1620. The local server 1640 has image data 1601 (eg, high density image 1611, high density image sequence) at a cloud transmission speed slower than the real-time capture speed, or at a speed suitable for serving by the cloud computing infrastructure at high speeds. 1615 and / or annotated high-density image sequences 1615a) can be transmitted to the cloud service 1620.

The cloud service 1620 derives from and represents the annotated high density image sequence 1615a, but for example, on image data 1601 to provide a simplified dataset 1617 that excludes any raw image data. By elastically dispatching a sufficiently fast, parallel, and / or large image set classifier 1625 that processes the instance to train the classifier 1625 (eg, on high density image 1611 and related note 1613). The received image data 1601 can be processed. The cloud service 1620 can, for example, transmit the simplified dataset 1617 to the local server 1640 communicating with the robot 100 or directly to the robot 100 after the processing interval. Elements of dataset 1617 may be used by robot 100 to issue commands to drive system 200 to maneuver robot 100 for scene / environment 10. For example, a "code" or "sock" or "ingestible debris" classifier 1625 trained on image data 1601 of one of many robots identifies an image pattern, away from hazards, or purpose. It may be used with a parameter returned to the robot 100 to orient the robot 100 in any of the areas.

When a wireless connection or consumer / commercial asymmetric broadband service is bandwidth-limited in the transmission direction, it buffers the high-density image sequence 1615 into a large storage device for packet switching, sorting, correction, and service. By utilizing quality and the like, it is possible to upload for a period relatively longer than the collection time (that is, the period for collecting the high-density image sequence 1615). For example, the upload may occur overnight, for a period of several hours when the robot's trajectory was in minutes, for a period of several days when the robot's trajectory was in hours, and so on.

A service related to Robot 100's robot software platform that provides a high-density image 1611 showing a sequence of environmental visions along the orbit of Mobile Robot 100 captured at real-time capture speed on local server 1640 or cloud service 1620. Can be received by 1623. Service 1623, also known as a cloud gateway, may be an agent of the robot software platform or may be independently controlled by a third party. The cloud gateway 1623 instantiates / maintains the high-density image sequence 1615 as one or more virtual servers 1621 within the cloud service 1620 (for example, depending on the annotation content), algorithms, classifiers 1625, and application platforms. May match the instance and package the high density image sequence 1615 and any annotation 1613 as training data for classifier 1625. As an alternative, the cloud gateway 1623 processes the same information as pre-packaged information from Robot 100, local base stations, etc., and is an adaptive and elastic cloud service application programming interface (API) (eg, Amazon). You may simply start dispatching enough predetermined virtual server instances 1621 to elastically add other instances that are EC2) compliant. The local server 1640, cloud gateway 1623, cloud service manager, or initial virtual processor instance 1621 receives at least some of the high-density image sequences 1615, 1615a in the high-density image sequence 1615, each of which is a high-density image sequence. Dispatch a batch processing task that derives from 1615, 1615a and reduces to the dataset 1617 that represents it. If desired, a new virtual processor instance 1621 can be serviced / instantiated (either all at once or when the training task becomes more complex). High-density image 1611 and trained model 1609 can be retained in long-term memory instances. The parameters of the trained classifier 1625 or model 1609 are returned to Robot 100, excluding the sequence of raw images 1611 (eg, directly or through the agent of Robot 100).

FIG. 16F provides an example arrangement 1600f for a method of navigating the robot 100. This method captures the streaming sequence 1615 of the high-density image 1611 of the scene 10 around the robot 100 along the trajectory of the robot 100's motion at real-time capture speed 1602f and notes 1613 out of the high-density image 1611. Includes 1604f and associates with at least some of. The method also sends the high density image 1611 and annotation 1613 to the remote server 1620 at a transmission speed slower than the real-time capture speed 1606f and receives the datasets 1607, 1617 from the remote server 1620 after the processing time interval. Including 1608f. Data sets 1607, 1617 derive from and represent at least a portion of the high density image sequences 1615, 1615a and the corresponding annotation 1613, but exclude the raw image data of sequences 1615, 1615a of the high density image 1611. The method includes moving the robot 100 relative to the scene 10 1610f based on the received datasets 1607, 1617.

The method sends the high density images 1611 and annotation 1613 to the local server and buffer 1640 (Figures 16D and 16E), and then sends the high density images 1611 and annotation 1613 to the remote server 1620 at a transmission speed slower than the real-time capture speed. May include sending to. The local server and buffer 1640 may be within a relatively short range of Robot 100 (eg, within 20-100 feet or wireless communication range). For example, the local and buffer 1640 may be a personal computer in the user's home that houses the robot 100, or a local server in the building that houses the robot 100.

In some embodiments, note 1613 is a time stamp, such as an absolute time reference, corresponding to at least some of the high density images 1611 and of mileage measurement data, accelerometer data, tilt data, and angular velocity data. Includes posture-related sensor data, which may include at least one of. Note 1613 can be associated with a high density image 1611 that reflects a hazard event captured within a time interval for a robot's hazard response 100 (eg, avoiding a cliff, escaping a confinement situation, etc.). In a further embodiment, associating annotation 1613 1604f may include associating a keyframe identifier with a subset of high density image 1611. The keyframe identifier may allow identification of the high density image 1611 based on the characteristics of the keyframe identifier (eg, flag, type, group, motion, stillness, etc.).

Note 1613 may include a set of low density 3-D points derived from the structural and motion restoration of features tracked during the high density image 1611 of the streaming sequence 1615 of the high density image 1611. The low density set of 3-D points can be derived from the volume point imaging device 450 on the robot 100. In addition, note 1613 may include camera parameters such as camera orientation with respect to individual 3-D points of the low density set of 3-D points. The crossable and non-crossable area markings for scene 10 may be annotation 1613 for high density image 1611.

Datasets 1607, 1617 are one or more texture maps extracted from high density image 1611, such as 2-D height map 1607, and / or terrain map 1607 representing features within high density image 1611 of scene 10. May include. Data sets 1607, 1617 may include a trained classifier 1625 for classifying features within the new high-density image 1611 captured in scene 10.

Figure 16G provides an example arrangement of 1600 g for a method of abstracting mobile robot environment data. The method comprises receiving 1602 g of sequence 1615 of high density images 1611 of robotic environment 10 from mobile robot 100 at a reception rate. The high-density image 1611 is captured along the motion trajectory of the mobile robot 100 at a real-time capture speed. The reception speed is slower than the real-time acquisition speed. The method also receives 1604 g of annotation 1613 associated with at least some of the high density images 1611 in the sequence 1615 of the high density image 1611 and high in at least some of the high density images 1611. Includes 1606g and 1606g of dispatching batch processing tasks to reduce density data to datasets 1607, 1617, which represent at least a portion of sequence 1615 of high density image 1611. The method also includes transmitting 1608 g of dataset 1617 to mobile robot 100. Data sets 1607, 1617 exclude raw image data from sequence 1615 of high density image 1611.

In some embodiments, the batch processing task processes the sequence 1615 of the high-density image 1611 into the high-density 3-D model 1609 of the robot environment 10 and the high-density 3-D model 1609. Includes making a terrain model 1607 for the 2-D position and at least one height coordinate system from floor G. In some embodiments, the terrain model 1607 is for a coordinate system of two-D positions and multiple occupied and unoccupied height boundaries from floor G. For example, a terrain model where the room with the table provides data showing the upper and lower bound heights of the associated table surface so that the robot 100 can determine if it can pass under the table. It is 1607.

The batch processing task is a high-density image sequence that corresponds to multiple robotic environments 10 (for example, so that the cloud 1620 can build a classifier 1625 to identify the desired features in any environment). It may include accumulating 1615, 1615a. Therefore, the batch processing task may include training multiple classifiers 1625 and / or one or more classifiers 1625 on sequence 1615 of high density image 1611. For example, the batch processing task associates annotation 1613, which reflects a hazard event, with a high-density image 1611 captured within a time interval for the hazard response of the mobile robot 100, eg, model parameters of classifier 1625. To train the classifier 1625 of the danger-related high-density image 1611 using the relevant hazard notes 1613 and the corresponding high-density image 1611 as training data to provide the datasets 603, 1607, 1617 of. It may be included. The classifier 1625 may include at least one support vector machine that builds at least one hyperplane for classification, and model parameters can classify datasets 1603, 1607, 1617 into risk-related classifications. Define a trained hyperplane. The model parameters may include sufficient parameters that define the kernel of the support vector machine, as well as soft margin parameters.

In some embodiments, the batch processing task involves instantiating multiple scalable virtual processes 1621 that are proportional to the scale of the high-density image sequences 1615, 1615a being processed. At least some of the virtual processes 1621 are released after transmission of datasets 1607, 1617 to robot 100. Similarly, the batch processing task may include instantiating multiple scalable virtual memories 1622 proportional to the scale of the stored high density image sequences 1615, 1615a. At least some of the virtual memory 1622 is released after transmission of datasets 1607, 1617 to robot 100. The batch processing task is also to distribute multiple scalable virtual servers 1621 according to the geographical proximity to the mobile robot 100 and / or one of the network traffic from the mobile robot 100. May include.

Certain et al., US Patent Publication No. 2011/0238857, "Committed Processing Rates for Shared Resources," published September 29, 2011, is incorporated herein by reference in its entirety. The cloud processing infrastructure described by Certain et al. Is one species that can be combined with it, such as the management system 202 or node manager module 108, which is the seed or part of the cloud gateway 1623, the elastic processing server 1621. A program execution service (PES) and / or virtual machine 110, as well as archive storage 222 or block data service (BDS) 204 or archive manager 224, which is the seed or part of long-term storage instance 1622.

FIG. 16H is a schematic diagram of an example mobile human interface robot system architecture 1600d. In the example shown, application developer 1602 generates an application 1610 that can run on a web pad 310 or computing device 1604 (eg, desktop computer, tablet computer, mobile device, etc.) communicating with the cloud 1620. You can access and use the application development tool 1640 as you do. An example application development tool 1640 is an integrated development environment 1642, software development kit (SDK) library 1644, development or SDK tool 1646 (eg, software code modules, simulators, cloud usage monitors and service configurators, and cloud service extension uploaders). / Developer), and / or source code 1648, but not limited to them. SDK library 1644 may allow enterprise developer 1602 to leverage Robot 100's mapping, navigation, scheduling, and conferencing techniques in application 1610. An example application 1610 may include, but is not limited to, a map builder 1610a, a mapping and navigation application 1610b, a video conferencing application 1610c, a scheduling application 1610d, and a utilization application 1610e. The application 1610 may be stored on one or more application servers 1650 (eg, cloud storage 1622) in the cloud 1620 and can be accessed through a cloud service application programming interface (API). Cloud 1620 may include one or more databases 1660 and simulator 1670. Web service APIs include Robot 100 and Cloud 1620 (for example, A. Allows communication between application server 1650, database 1660, and simulator 1670). The external system 1680 may also interact with the cloud 1620, for example to access application 1610.

In some embodiments, the map builder application 1610a uses camera 320 or 3-D imaging sensor 450 with reference coordinates as provided by mileage measurement, global positioning system, and / or midpoint navigation. By concatenating together the photos or videos captured by, a map 1700 (Fig. 17A) of the environment around Robot 100 can be constructed. The map may provide an indoor or outdoor street or route display of the environment. For malls and shopping centers, maps can provide route patrols throughout the mall, with each store marked as a reference location with additional linked images or videos and / or promotional information. Maps and / or configuration images or videos can be stored in database 1660.

Application 1610 may communicate seamlessly with cloud services that can be customized and extended based on the needs of each user entity. Enterprise developer 1602 may upload a cloud-side extension to cloud 1620 that fetches data from an external proprietary system for use by application 1610. Simulator 1670 allows developers 1602 to build enterprise-scale applications without Robot 100 or associated robot hardware. Users may use SDK Tools 1646 (eg, Usage Monitor and Service Configurator) to add or disable cloud services.

Referring to FIGS. 17A and 17B, in some situations, the robot 100 receives an occupation map 1700 of the object 12 in the scene 10 and / or work area 5, or the robot controller 500 receives the image sensor 450 over time. Occupation map 1700 is generated (and may be updated) based on image data and / or image depth data received from (eg, second 3-D image sensor 450b). Simultaneous localization and mapping (SLAM) is (given) to build map 1700 within an unknown environment or scene 10 (without using deductive knowledge), or at the same time tracking its current position. A technique that can be used by Robot 100 to update Map 1700 within a known environment (using deductive knowledge from the map). Map 1700 may be used to depict the environment for planning and navigation to determine its location within the environment. Map 1700 assists in assessing the actual position by recording the information obtained from a form of perception and comparing it with the current set of perceptions. The benefits of Map 1700 in assisting position evaluation increase as the accuracy and quality of current perception diminishes. Map 1700 generally represents the state at which Map 1700 was provided or generated. This does not necessarily match the state of the environment when Map 1700 is used. Other localization techniques include monocular visual SLAM (MonoSLAM), and implementations that use an extended Kalman filter (EKF) for the MonoSLAM solution.

Controller 500 may perform scale-invariant feature transformation (SIFT) to detect and represent local features in the captured image. For any object 12 in the image, points of interest on the object 12 can be extracted to provide a "feature description" of the object 12. This description, then extracted from the training image, can be used to identify the object 12 when attempting to locate the object 12 within a test image containing many other objects. For reliable recognition, it is important that the features extracted from the training image are detectable even under changes in image scale, noise, and illumination. Such points are usually located on high contrast areas of the image, such as the edges of an object. For object recognition and detection, Robot 100 has a prominent key point that is invariant to position, scale, and rotation, and robust to affine transformations (scale, rotation, shear, and position changes) and lighting changes. SIFT may be used to find out. In some embodiments, the robot 100 captures multiple images of scene 10 or object 12 (using camera 320 and / or imaging sensor 450) (eg, under different conditions, from different angles, etc.). , Store the image in the matrix. The robot 100 can access the stored image so as to identify a new image by comparison, filter, or the like. For example, SIFT features can be taken from the input image and matched to the SIFT feature database obtained from the (previously captured) training image. Feature matching can be done through the Euclidean distance-based nearest neighbor approach. The Hough transform may be used to increase object identification by clustering these features that belong to the same object and reject matches excluded in the clustering process. The SURF (accelerated robust feature) may be a robust image detector and descriptor.

In addition to the localization of the robot 100 in the scene 10 (eg, the environment around the robot 100), the robot 100 uses the sensor system 400 to reach other points in the connection space (eg, work area 5). You may move. Robot 100 maps the neighborhood area around robot 110 and is short (eg, mounted on the back of the torso 140 as shown in FIGS. 1 and 3) to identify relatively close objects 12. A distance-based imaging sensor 450a and a relatively large area around the robot 100 are mapped to identify a relatively distant object 12 (eg, on the head 160 as shown in FIGS. 1 and 3). It may include a long-range imaging sensor 450b (mounted). Robot 100 identifies known objects 12 as well as obstructions 16 in scene 10 (eg, where object 12 should or should not be seen from the current point of view, but cannot be seen). You can use the Occupation Map 1700 to do this. Robot 100 registers a block 16 or a new object 12 in the scene 10 and attempts to bypass the block 16 or the new object 12 to verify the position of the new object 12 or any object 12 in the block 16. be able to. Alternatively, using Occupation Map 1700, Robot 100 can determine and track the movement of Object 12 in Scene 10. For example, the imaging sensors 450, 450a, 450b do not detect the mapped position of the object 12 in the scene 10, while the new position 12 of the object 12 in the scene 10 ́ may be detected. The robot 100 can register the position of the old object 12 as the block 16 and try to bypass the block 16 so as to verify the position of the object 12. Robot 100 may compare the new image depth data with the previous image depth data (eg, map 1700) and assign a confidence level for the position of object 12 in scene 10. The position confidence level of the object 12 in the scene 10 can time out after the threshold period. The sensor system 400 can update the position confidence level of each object 12 after each imaging cycle of the sensor system 400. In some embodiments, the new blockage 16 (eg, missing object 12 from Occupation Map 1700) detected within the blockage detection period (eg, less than 10 seconds) is a "living" object in scene 10. It may represent 12 (eg, moving object 12).

In some embodiments, the desired second object 12b, located behind the detected first object 12a in scene 10, may not be initially undetected as a block 16 in scene 10. .. The block 16 can be an area within the scene 10 that is not easily detectable or visible by the imaging sensors 450, 450a, 450b. In the embodiment shown, the sensor system 400 of the robot 100 (or a portion thereof of the imaging sensors 450, 450a, 450b, etc.) has a viewing angle θ so as to visually recognize the scene 10._VHas a field of view 452 with (which can be any angle between 0 and 360 degrees). In some embodiments, the imaging sensor 450 has a 360 degree viewing angle θ._VIn other embodiments, the imaging sensors 450, 450a, 450b have a viewing angle θ of less than 360 degrees (eg, between about 45 and 180 degrees), while including omnidirectional optics for_VHave. Viewing angle θ_VIn the embodiment where is less than 360 degrees, the imaging sensors 450, 450a, 450b (or their components) have a 360 degree viewing angle θ._VMay be rotated relative to the robot body 110 to achieve. In some embodiments, the imaging sensors 450, 450a, 450b or parts thereof can be moved relative to the robot body 110 and / or the drive system 200. Also, to detect the second object 12b, the robot 100 is driven around the scene 10 in one or more directions to obtain a viewpoint, which allows the detection of the second object 10b. The imaging sensors 450, 450a, 450b may be moved by (eg, by translation and / or rotation on the working surface 5). Robotic movement or independent movement of the imaging sensors 450, 450a, 450b or parts thereof can also solve monocular difficulty.

Confidence levels may be assigned to the detected position or tracked movement of object 12 within work area 5. For example, when generating or updating the occupation map 1700, the controller 500 may assign a confidence level for each object 12 on the map 1700. The confidence level can be directly proportional to the probability that the object 12 is actually located within the work area 5, as shown on map 1700. The confidence level can be determined by several factors such as the number and type of sensors used to detect the object 12. For example, the contact sensor 430 may provide the highest level of reliability when the contact sensor 430 senses the actual contact of the robot 100 with the object 12. The imaging sensor 450 can provide different levels of reliability, which can be higher than the proximity sensor 430. Data received from more than one sensor in the sensor system 400 can be aggregated or accumulated to provide a relatively higher level of reliability than any one sensor.

The mileage measurement uses data from the movement of the actuator in order to estimate the change over time (movement distance) of the position. In some embodiments, encoders are placed on the drive system 200 to measure wheel rotation, and thus the distance traveled by the robot 100. The controller 500 may use mileage measurement to evaluate the confidence level of the object position. In some embodiments, the sensor system 400 includes an odometer and / or an angular velocity sensor (eg, a gyroscope or IMU470) for sensing the distance traveled by the robot 100. A gyroscope is a device for measuring or maintaining orientation based on the principle of conservation of angular momentum. The controller 500 uses the odometer and / or gyro signal received from the odometer and / or the angular velocity sensor, respectively, to determine the position of the robot 100 within work area 5 and / or on the occupation map 1700. You may. In some embodiments, controller 500 uses dead reckoning. Dead reckoning is the process of estimating the current position based on a previously determined position and advancing that position over time and elapsed time based on known or estimated speed. Knowing the robot position within work area 5 (via, for example, mileage measurement, gyroscope, etc.) and the perceived position of one or more objects 12 within work area 5 (via sensor system 400). Allows the controller 500 to evaluate a relatively higher level of confidence (compared to those without mileage measurement or gyroscope) of the position or movement of object 12 on the occupation map 1700 and within work area 5. ..

Mileage measurement based on wheel motion can be electrically noisy. The controller 500 obtains image data of the environment or scene 10 around the robot 100 from the image sensor 450 in order to calculate the robot motion through the visual mileage measurement independently of the wheel-based mileage measurement of the drive system 200. You may receive it. The visual mileage measurement may require the use of optical flow to determine the motion of the imaging sensor 450. The controller 500 can use the motion calculated based on the imaging data of the imaging sensor 450 to correct any errors in the wheel-based mileage measurement, thus enabling improved mapping and motion control. To. The visual mileage measurement may have a limitation due to low texture or low light scene 10 if the imaging sensor 450 is unable to track features in the captured image.

Other details and features of mileage measurement and imaging systems that may be combined with those described herein are incorporated herein by reference in their entirety. It can be found in 7,158,317 (describes a "depth of field" imaging system) and in US Pat. No. 7,115,849 (describes a wave surface coded interfacial contrast imaging system).

When the robot is the first time in the building in which it will operate, the robot needs to be guided or provided with a map of the building (eg, room and corridor locations) for autonomous navigation. possible. For example, in a hospital, the robot may need to know the location of each room, nursing station, etc. In some embodiments, the robot 100 receives a layout map 1810, such as that shown in FIG. 18A, and can be trained to learn the layout map 1810. For example, while guiding the building to the robot 100, the robot 100 may record a specific position corresponding to the position on the layout map 1810. The robot 100 may display a layout map 1810 on the web pad 310, and when the user takes the robot 100 to a specific position, the user (eg, a touch screen or other pointing device of the web pad 310) You can tag its position on the layout map 1810 (using). The user may choose to enter a tagged location sign such as room name or room number. When tagging, Robot 100 may store tags along with points on layout map 1810 and corresponding points on robot map 1820, such as those shown in FIG. 18B.

Using the sensor system 400, the robot 100 may build a robot map 1820 as it moves around. For example, the sensor system 400 can provide information about how far the robot 100 has moved and the direction of movement. Robot Map 1820 may include fixed obstacles in addition to the walls provided within Layout Map 1810. Robot 100 may use Robot Map 1820 to perform autonomous navigation. In the robot map in 1820, the "wall" is perfectly straightened, for example, by the detected packing generated along the wall in the corresponding corridor and / or the furniture detected inside the various small chambers. May not be visible. In addition, there may be differences in rotation and resolution between layout map 1810 and robot map 1820.

After map training, when the user wants to send the robot 100 to a position, the user can point to an indicator / tag (eg, enter a sign or tag in the position text box displayed on the webpad 310). Either the robot 100 can display the layout map 1810 to the user on the web pad 310, and the user can select a position on the layout map 1810. If the user selects a tagged layout map position, Robot 100 can easily determine the position on Robot Map 1820 that corresponds to the selected position on Layout Map 1810, and the selected position. You can proceed to navigate to.

If the selected position on layout map 1810 is not a tagged position, robot 100 determines the corresponding position on robot map 1820. In some implementations, Robot 100 uses existing tagged positions to calculate the scaling size, start point mapping, and rotation between layout map 1810 and robot map 1820, and then (eg, for example). Apply the calculated parameters to determine the robot map position (using affine transformations or coordinates).

The robot map 1820 does not have to have the same orientation and scale as the layout map 1810. Further, the layout map does not have to be proportional to the actual size and may have different distortions depending on the map area. For example, layout maps 1810 created by scanning fire evacuation maps typically found in hotels, offices, and hospitals are usually not drawn in proportion to their actual size and in different areas of the map. It can even have different scales. The robot map 1820 can have its own distortion. For example, the position on the robot map 1820 may be calculated by counting the rotation of the wheels as a measure of distance, not if the floor is slightly slippery or turning a corner causes extra rotation. Accurate rotation calculations can cause the robot 100 to determine the inaccurate position of the mapped object.

A method of mapping a given point 1814 on a layout map 1810 to a corresponding point 1824 on a robot map 1820 is to map the layout map 1810 and the robot in a region containing the layout map points (eg, within a threshold range). It may include using existing tagged 1812 points to calculate the local distortion to and from the map 1820. The method further involves applying a strain calculation to the layout map point 1814 to find the corresponding robot map point 1824. Robot maps starting from a given point on the flop 1820, for example, to determine its current position in the robot, when it is desired heading the corresponding position on the layout map 1810 can perform reverse.

FIG. 18C provides an example arrangement 1800 for moving the robot 100 to navigate around the environment using the layout map 1810 and the robot map 1820. With reference to FIGS. 18B and 18C, the actions are 1802c to receive the layout map 1810 corresponding to the environment of the robot 100 and 1804c to move the robot 100 in the environment to the layout map position 1812 on the layout map 1810. Using the 1806c and the recorded robot map position 1822 and the corresponding layout map position 1812, the robot map is recorded, the robot map position 1822 on the robot map 1820, which is environment-aware and generated by the robot 100. Determining the distortion between the 1820 and the layout map 1810 1808c and determining the corresponding target robot map position 1824, which allows the robot to navigate to the selected position 1814 on the layout map 1810. Includes applying the determined distortion to the target layout map position 1814, as in 1810c. In some implementations, the behavior uses existing tagged positions to determine the scaling size, start point mapping, and rotation between the layout map and the robot map, and the target layout map position. Includes resolving selected target robot map positions corresponding to 1814. The action may include applying the affine transformation to the determined scaling size, start point mapping, and rotation to resolve the robot map position.

With reference to FIGS. 19A-19C, in some embodiments, the method ensures that at least one triangle 1910 at the layout map point 1912 with vertices covers the entire area of the layout map 1810. Using tagged layout map points 1912 (also known as recorded layout map positions) to derive a triangulation of the area inside the boundary shape containing tagged layout map points 1912. Including. The method further finds the triangle 1910 containing the selected layout map points 1914 and the corresponding triangles 1920 (ie,) mapped in the layout map 1810 and the corresponding triangles 1910 mapped in the robot map 1820. Includes determining scale, rotation, parallel movement, and skew to and from a robot map triangle with the same tagged vertices. The method involves applying the determined scale, rotation, translation, and skew to the selected layout map point 1914 to find the corresponding target robot map point 1924.

FIG. 19C provides an arrangement 1900 that is an example of an operation for determining the target robot map position 1924. The action is to determine the triangulation between the layout map positions that indicate the boundaries of the target layout map position 1902 and between the triangle mapped in the layout map and the corresponding triangle mapped in the robot map. To determine the scale, rotation, parallel movement, and skew of 1904 and to apply the determined scale, rotation, parallel movement, and skew to the target layout map position to determine the corresponding target robot map point. Including 1906.

With reference to FIGS. 20A and 20B, in another embodiment, the method determines the distance of all tagged points 1912 in the layout map 1810 to point 1914 of the selected layout map. Includes determining the centroid 2012 of the tagged point 1912 in the layout map. The method also includes determining the centroid 2022 of all tagged points 1922 on the robot map 1820. For each tagged layout map point 1912, this method transforms the vector 2014 extending from the layout map centroid 2012 to the selected layout point 1914 into the vector 2024 extending from the robot map centroid 2022 to the robot map point 1924. Includes determining the rotation and length scaling required for. Using this data, the method further comprises determining average rotation and scale. For each tagged layout map point 1912, the method further applies translation, average rotation, and average scale of the center of gravity to the selected layout map point 1914, thereby applying the "ideal robot map coordinate" point 1924i. Includes determining. Furthermore, for each tagged layout map point 1912, the method determines the distance from that layout map point 1912 to the selected layout map point 1914 and these distances from the shortest distance to the longest distance. Includes classifying layout map points 1912 that are tagged separately. The method uses one of the inverse squares of the distance between the tagged layout map point 1912 and the selected layout map point 1914 to determine the "influence factor" for the tagged layout map point 1912. Including judging. Then, for the tagged layout map points 1912, the method is proportionally distributed by using the influencing factors of the tagged layout map points 1912, the "ideal robot map coordinates" point 1924i and the robot map points 1924. Is the difference between Includes determining the vector. The method involves summing the proportionally distributed vectors and adding them to the "ideal robot map coordinates" point 1924i for the selected layout map point 1914. The result is the corresponding robot map point 1924 on the robot map 1820. In some embodiments, this method / algorithm includes only the nearest N tagged layout map points 1912, rather than all tagged layout map points 1912.

FIG. 20C provides an example arrangement 2000 for determining the target robot map position 1924 using the layout map 1810 and the robot map 1820. The operation is to determine the distance between all layout map positions and the target layout map position 2002, to determine the center of gravity of the layout map position 2004, and to determine the center of gravity of all recorded robot map positions. In 2006 and for each layout map position, determine rotation and length scaling so that the vector extending from the layout map centroid to the target layout position is transformed into a vector extending from the robot map centroid to the target robot map position. including.

With reference to FIG. 16C again, the robot 100 may operate autonomously, but the user may wish to control or manage the robot 100 through application 1610. For example, the user may wish to control or define the movement of the robot 100 within the environment or scene 10 by providing navigation points on the map and the like. In some embodiments, the map builder application 1610a allows the user to occupy the scene 10 based on the sensor data generated by the sensor system 400 of one or more robots 100 in the scene 10 (Fig. 17A). Or it allows you to create a layout map 1810 (Figure 18A). In some embodiments, the robot map 1810 can be post-processed to generate one or more vector-based layout maps 1810. The map builder application 1610a may allow the user to customize maps 1700, 1810 (eg, arranging walls that should look parallel). Annotations recognizable by Robot 100 and / or other applications 1610a can also be added to maps 1700, 1810. In some embodiments, the map builder application 1610a uses cloud services to ensure robot map data, layout map data, user-defined objects, and annotations in cloud storage 1622 on the cloud 1620. You may memorize it in. The relevant dataset may be pushed from the cloud service to the appropriate Robot 100 and Application 1610.

In some embodiments, the mapping and navigation application 1610b (Figure 16C) allows the user to identify a destination location on the layout map 1810 and request the robot 100 to drive to the destination location. To do. For example, the user may run the mapping and navigation application 1610b on a computing device such as a computer, tablet computer, mobile device, etc. that is communicating with the cloud 1620. The user accesses the layout map 1810 of the environment around the robot 100, marks the destination position on the layout map 1810, or sets it differently and requests the robot 100 to move to the destination position. Can be done. Robot 100 may then autonomously navigate to a destination location using layout map 1810 and / or corresponding robot map 1820. To navigate to the destination location, the robot 100 determines its local perceived space (ie, the space around the robot 100 as perceived through the sensor system 400) and an object detection obstacle avoidance (ODOA) strategy. May depend on its ability to perform.

With reference to FIGS. 11B and 21A-21D, in some embodiments, the robot 100 (eg, control system 510 shown in FIG. 13) has its local perceived space as an obstacle (black) 2102, unknown (gray). Divide into three categories: 2104 and known free space (white) 2106. Obstacle 2102 is an observed (ie, sensed) point above the ground G below the height of the robot 100, and an observed point below the ground (eg, holes, gradual reduction, etc.) ). The known free area 2106 corresponds to the area where the 3-D image sensor 450 can see the ground G. Data from all sensors in the sensor system 400 can be incorporated into a discretized 3-D voxel grid. The 3-D grid can then be analyzed and transformed into a 2-D grid 2100 using three local perceptual spaces. FIG. 21A provides an example schematic of the local perceptual space while the robot 100 is stationary. The information in the 3-D voxel grid is persistent but, if not reinforced, deteriorates over time. When the robot 100 is in motion, it has more known free space 2106 to navigate for sustainability.

The object detection obstacle avoidance (ODOA) navigation strategy of control system 510 may include either accepting or denying the robot position that may be attributed to the command. Different commands can be used to generate robot paths 2110 with many levels of depth, resulting in robot positions at each level. FIG. 21B provides an example schematic of the local perceptual space while the robot 100 is moving. The ODOA behavior 600b (Fig. 13) can evaluate each predicted robot path 2110. These evaluations can be used by the motion selection engine 580 to determine favorable results and corresponding robot commands. For example, for each robot position 2120 in robot path 2110, ODOA behavior 600b identifies each cell in grid 2100 within the bounding box around the corresponding position in robot 100 and classifies each cell. Methods for object detection and obstacle avoidance can be implemented, including receiving. For each cell classified as an obstacle or unknown, a collision check is performed by extracting the grid points corresponding to the cells and determining if the grid points are within the collision circle around the position of the robot 100. If the grid points are within a collision circle, the method further performs a three-point comparison test to see if the grid points are within a collision triangle (eg, the robot 100 can be modeled as a triangle). including. If the grid points are within a collision triangle, the method involves rejecting the grid points. If the robot position is inside the sensor system view of the parent grid points on the robot path 2110, the "unknown" grid points will be known by the time the robot 100 reaches these grid points. Ignored because it is supposed to be.

The method is modeled by a robot path region (eg, a rectangle) between continuous robot positions 2120 within robot path 2110 to prevent robots from colliding during the transition from one robot position 2120 to the next. It may include determining whether or not any obstacle collision exists within).

FIG. 21C shows the robot 100's local perceived space and sensor system field of view 405 (control system 510 uses only specific sensors such as the first and second 3-D image sensors 450a, 450b to determine the robot path). A schematic diagram of) is provided. Utilizing the holonomic mobility of the drive system 200, the robot 100 uses the known ground G persistence to allow the sensor system field of view 405 to be actively driven in a direction that is out of range. be able to. For example, if the robot 100 sits stationary with the first and second 3-D image sensors 450a, 450b pointing forward, the robot 100 can be driven laterally, but the robot. The control system 510 is proposed because the robot 100 does not know what is on its side, as illustrated in the embodiment shown in FIG. 21C, which shows an area classified as unknown on the side of 100. Refuse to move. When the robot 100 is driven forward with the first and second 3-D image sensors 450a and 450b pointing forward, when the robot 100 is driven forward, the first and second 3-D images Since both sensors 450a and 450b can see the ground G as empty and the persistence of the classification has not yet deteriorated, the ground G next to the robot 100 may be classified as a known empty 2106 ( For example, see Figure 21B). In such a situation, the robot 100 can be driven laterally.

Referring to FIG. 21D, in some embodiments, the ODOA behavior 600b is seen by the robot where the robot is heading (not currently), taking into account the large number of possible trajectories due to holononomic mobility. However, the locus may be selected. For example, the robot 100 can anticipate sensor field orientation, which allows the control system 510 to detect an object. Since the robot can rotate while translating, the robot can increase the sensor field of view 405 while driving.

By understanding the field of view 405 of the sensor system 400 and what it will see at different positions, the robot 100 can select a moving trajectory that will help it see where it is heading. For example, when turning a corner, as shown in FIG. 21E, the robot 100 can eventually reach the robot position 2120, which is currently ignorant, rather than the sensor system field of view 405 at the parent robot position 2120. May reject a trajectory that makes a sharp turn at a corner. Instead, the robot 100 selects a moving trajectory that quickly turns in the desired direction of motion, moves sideways, and then moves straight around the corner, as shown in FIG. 21F. Holonomic mobility may be used.

In some embodiments, the mapping and navigation application 1610b (Figure 16C) provides remote control capabilities. For example, the user can drive the robot 100 using video midpoint drive (eg, using one or more of cameras or imaging sensors 320, 450). The user has a height H of the body 140 (Fig. 14)._THeight H of leg 130 so as to raise / lower to change the field of view of one of the imaging sensors 450_LThe robot head 160 may be rotated and / or tilted to change and / or change the field of view of the supported camera 320 or imaging sensors 450, 450b (see, eg, FIGS. 11A and 11B). In addition, the user can use the drive system 200 to rotate the robot around its Z-axis to obtain another view of the camera or imaging sensors 320, 450. In some embodiments, the mapping and navigation application 1610b allows the user to switch between multiple layout maps 1810 (eg, for different environments or different robots 100) and / or multiple robots on one layout map 1810. Allows you to manage 100. The mapping and navigation application 1610b may communicate with the cloud services API to enforce the policies for proper robot use described by the owner or organization of Robot 100.

With reference to FIG. 16C again, in some embodiments, the video conferencing application 1610c allows a user to initiate and / or participate in a video conferencing session with another user. In some embodiments, the video conferencing application 1610c is connected to the Internet where the user uses a user of Robot 100, a remote user on a computing device connected to the cloud 1620, and / or a mobile handheld device. Allows you to start and / or join a video conferencing session with another remote user. The video conferencing application 1610c may provide an electronic whiteboard, image viewer, and / or PDF viewer for sharing information.

Scheduling application 1610d allows users to schedule the use of one or more Robots 100. When there are fewer robots 100 than a person wants to use, the robots 100 can become an inadequate resource and require scheduling. Scheduling resolves conflicts in resource allocation and enables higher resource utilization. The scheduling application 1610d can be robot-centric and may be integrated with a third party calendering system such as Microsoft Outlook or Google Calendar. In some embodiments, the scheduling application 1610d communicates with the cloud 1620 through one or more cloud services to dispatch the robot 100 at a pre-scheduled time. Scheduling application 1610d transfers time-related data (eg, maintenance schedules, etc.) to other robot data (eg, robot position, health) so that the cloud service can select the robot 100 for missions identified by the user. It may be integrated with the state etc.).

In one scenario, a doctor schedules on a computing device (eg, a portable tablet computer or handheld device) communicating with the cloud 1620 to schedule a remote hospital round later in the week. You may access application 1610d. The scheduling application 1610d can schedule the robot 100 in the same way as it allocates a conference room on an electronic calendar. The cloud service manages the schedule. In the middle of the night, when a doctor receives a call to see a critically ill patient in a remote hospital, the doctor requests Robot 100 using the scheduling application 1610d and / or the mapping and navigation application 1610b. It can be used to send Robot 100 to the hospital room. Physicians may use the video conferencing application 1610c to access medical records and patient videos or images on their computing devices (eg, by accessing cloud storage 1622). Cloud services may be integrated with robot management, electronic health recording systems, and medical imaging systems. The doctor may control the movement of the robot 100 so that it interacts with the patient remotely. If the patient speaks only Portuguese, the video conferencing application 1610c may automatically translate the language, or a third-party translator is communicating with the cloud 1620 (eg via the internet). You may use another computing device to participate in the video conference. You can use cloud services to request, perform, record, and charge translation services.

The usage / statistics application 1610e can be a general purpose application for users to monitor robot usage, generate robot usage reports, and / or manage robot corps 100. The application 1610e may also provide general behavior and troubleshooting information for Robot 100. In some embodiments, the usage / statistics application 1610e registers the use of one or more simulators 1670, the user adds / disables services associated with the use of the robot 100, and the usage policy for the robot. It is possible to modify and so on.

In another scenario, the business may have at least one Robot Corps 100 for telepresence applications. The location manager uses the utilization / statistics application 1610e running on a computing device communicating with the cloud 1620 (eg, over the Internet) to state one or more robots 100 states (eg, location,). Usage and maintenance schedules, battery information, location history, etc.) may be monitored. In some embodiments, the location manager may assist the user with robot problems by sharing a user session. The location manager uses application 1610 to navigate the corresponding robot 100, speak through the robot 100 (ie, telepresence), enter power saving mode (eg, reduce functionality), find a charger, and so on. It can be used to issue commands to any of Robot 100. The location manager or user can use application 1610 to manage users, security, layout maps 1810, video visibility, add / remove robots from corps, and so on. Robot 100 remote operator schedules / reschedules / cancels robot reservations (eg, using scheduling application 1610d) and simulates location (eg, using simulator 1670 running on a cloud server) You can participate in a training course using a simulated robot that roams around.

SDK library 1644 may contain one or more source code libraries for use by the developer 1602 of application 1610. For example, a visual component library can provide a graphical user interface, or a visual component having an interface for accessing encapsulation features. Examples of visual components include layout map tiles and code categories for drawing robots, holding video conferences, viewing images and documents, and / or displaying calendars or schedules. The robot communication library (for example, web service API) is RESTful (Representational State Transfer) and JSON (JavaScript Object) for direct communication with the robot 100. Notation) -based API can be provided. Robotic communication libraries can provide Objective-C bindings (eg for iOS development) and Java® bindings (eg for Android development). These object-oriented object APIs allow application 1610 to communicate with Robot 100 while encapsulating the underlying Robot 100 data transfer protocol from developer 1602. An individual who follows a routine in the robot communication library may return the video screen coordinates corresponding to the individual tracked by the robot 100. The face recognition routine of the robot communication library returns the coordinates of the face in the camera field of view of the camera 320 and, optionally, the name of the recognized tracking individual. Table 1 provides a typical list of robot communication services.<tables num="1"><img file="JP2014505934A_D0001.tif" /></tables> table 1

The cloud service communication library allows application 1610 to communicate with cloud 1620 (eg, with cloud storage 1622, application server 1650, database 1660, and simulator 1670) and / or robot 100 communicating with cloud 1620. May include API. Cloud service communication libraries can be provided in both Objective-C and Java® bindings. Examples of cloud service APIs include navigation APIs (eg, for retrieving locations, setting destinations, etc.), map storage and recovery PAIs, camera feed APIs, remote control APIs, usage statistics APIs, and more. ..

The cloud service extensibility interface may also allow the cloud service to interact with the web service from an external source. For example, a cloud service may define a set of extended interfaces that allow enterprise developer 1602 to implement an interface for an external proprietary system. Extensions can be uploaded to the cloud infrastructure and deployed. In some embodiments, the cloud service can employ standard scalable interfaces defined by various industrial communities.

Simulator 1670 may allow debugging and testing of application 1610 without connectivity to Robot 100. Simulator 1670 can model or simulate the behavior of Robot 100 without actually communicating with Robot 100 (eg, to plan a route and access a map database). To run the simulation, in some embodiments, the simulator 1670 generates a map database (eg, from layout map 1810) without the use of robot 100. This may involve image processing (eg, edge detection) so that features (such as walls, corners, columns, etc.) are automatically identified. Simulator 1670 can use a map database to simulate route planning within the environment determined by layout map 1810.

The cloud service extension uploader / developer uploads the extension to the cloud 1620, connects to an external third party user authentication system, to an external database or storage device (eg, patient information for pre- and post-consultation). It may be possible to access, access the image for illustration in a video conference session, and so on. The cloud service extension interface may allow the integration of proprietary systems with the cloud 1620.

The various implementations of the systems and techniques described herein include digital electronic circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or them. It can be realized by the combination of. These various embodiments are concatenated to receive data and instructions from the storage system and to transmit data and instructions to it, which may be dedicated or general purpose. May include implementations in one or more computer programs executable and / or interpretable on a programmable system, including a programmable processor, at least one input device, and at least one output device. it can.

These computer programs (also known as programs, software, software applications, or code) include machine instructions for programmable processors, high-level procedures and / or object-oriented programming languages, and / or assemblies /. It can be realized in a machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. Refers to any computer program product, device, and / or device (eg, magnetic disk, optical disk, memory, programmable logical device (PLD)) used to. The term "machine readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

Realizations and functional operations of the subject matter described herein are computer software, firmware, or hardware, or them, including digital electronic circuits, or the structures disclosed herein and their structural equivalents. It can be realized by a combination of one or more of them. Embodiments of the subject matter described herein are one or more computer program products, i.e., computer programs encoded on a computer-readable medium, for execution by a data processor or for controlling its operation. It can be implemented as one or more modules of instructions. A computer-readable medium can be a machine-readable storage device, a machine-readable storage circuit board, a memory device, a composition that affects a machine-readable propagating signal, or a combination of one or more of them. The term "data processor" includes, by way of example, all devices, devices, and machines for processing data, including programmable processors, computers, or multiple processors or computers. In addition to the hardware, the device contains code that creates the execution environment for the computer program in question, such as processor firmware, protocol stacks, database management systems, operating systems, or a combination of one or more of them. Can include. The propagating signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver.

Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including languages that are compiled or interpreted, and as a stand-alone program. , Or can be deployed in any form, including deployment as a module, component, subroutine, or other unit suitable for use in a computing environment. Computer programs do not necessarily have to deal with files in the file system. A program may be part of a file that holds other programs or data (eg, one or more scripts stored in a markup language document) in a single file dedicated to the program in question, or multiple adjustments. It can be stored in a file (for example, a file that stores one or more modules, subprograms, or parts of code). Computer programs can be deployed to run on one computer, or on multiple computers located in one location, or distributed across multiple locations and interconnected by communication networks. ..

The processes and logical flows described herein are performed by one or more programmable processors that operate on input data and perform functions by producing outputs, run one or more computer programs. can do. Processes and logic flows can also be implemented by dedicated logic circuits, such as FPGAs or ASICs, and devices can also be implemented as them.

Suitable processors for running computer programs include, for example, both general purpose and dedicated microprocessors, and any one or more processors of any type of digital computer. Generally, the processor receives instructions and data from read-only memory, random access memory, or both. An integral part of a computer is a processor for executing instructions, as well as one or more memory devices for storing instructions and data. In general, computers also include, or receive data from, one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks. It is operably linked to it for transfer or for both. However, the computer does not need to have such a device. In addition, computers can be embedded in other devices, such as mobile phones, personal digital assistants (PDAs), portable audio players, and Global Positioning System (GPS) receivers, to name a few. Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks. , And all forms of non-volatile memory, media, and memory devices, including CD ROMs and DVD-ROM disks. Processors and memory can be complemented or incorporated into dedicated logic circuits.

Implementations of the subject matter described herein include, for example, a middleware component, including a back-end component as a data server, eg, a front-end component, including an application server, eg, a user through which the specification. A computing system, including a client computer, having a graphical user interface or web browser that can interact with the implementation of the subject matter described in, or one or more such backends, middleware, or frontends. It can be realized by any combination of components. The components of the system can be interconnected in any form or by a medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LAN) and wide area networks (WAN), such as the Internet.

The computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact through communication networks. The client-server relationship arises from computer programs that run on their respective computers and have a client-server relationship with each other.

Although the present specification contains many details, they should not be construed as limiting the invention or its alleged scope, but rather as an explanation of features specific to a particular embodiment of the invention. It should be. Also, the particular features described in the context of the distinct embodiments herein can also be realized in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment can also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, features are described above to work in a particular combination, and may even be claimed as such in the first place, but in some cases, one or more features of the claimed combination may be cut out from the combination. The alleged combination may be a sub-combination or a variant of the sub-combination.

Similarly, the actions are depicted in the drawings in a particular order, which means that such actions are performed in the particular order shown or in the order in which they occur in order to achieve the desired result. , Or should not be understood as requiring that all illustrated actions be performed. In certain situations, multitasking and parallelism may be advantageous. Moreover, the separation of the various system components in the embodiments described above should not be understood to require such separation in all embodiments, and the program components and systems described are generally simply simple. It should be understood that they can be integrated together into one software product or packaged into multiple software products.

Many implementations have been described. However, it is understood that various modifications may be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the subsequent claims. For example, the actions referred to in the claims can be performed in a different order to still achieve the desired result.