Interfacing with a mobile telepresence robot

0001The present invention relates to a mobile telepresence robot.

0002A robot is an electromechanical machine generally guided by a computer or electronic programming. Telepresence robots have the ability to move around and are not fixed in one physical position. Currently commonly used telepresence robots are automated guided vehicles (AGVs). AGVs are telepresence robots that typically track markers or wiring in the floor or use vision systems or navigation lasers. Telepresence robots are found in industrial, military and security environments. They are also seen as consumer products for entertainment or consumer products that perform certain tasks such as home support.

0003One aspect of the present invention provides a telepresence robot system including a local terminal and a remote telepresence robot. The local terminal can include an electronic display, a processor and a memory that communicates with the processor, which memory contains instructions that can be executed by the processor. A plurality of executable commands include a process of reading at least a part of a plan view map representing a robot navigable area on the robot operating surface, each of which has tag coordinates and tag information (which may include annotations) that describe the relative position of the tag. The process of reading at least one of the tags of, the process of receiving a video feed from the remote telepresence robot's imaging system, the process of receiving positioning information, the process of displaying the video feed from the remote telepresence robot's imaging system, The process of displaying the current position of the telepresence robot on the plan view map, using the tag coordinates, on the plan view map and at least one of the video feeds of the at least one tag. It can be configured to cause the processor to perform a process of displaying the rendition of the tag annotation and a process of sending one or more commands to the remote telepresence robot.

0004In some embodiments, the instructions that can be executed by the processor further give the processor a distortion (two-dimensional coordinate system) between the plan view map and the video feed received from the imaging system of the remote telepresence robot. The process of determining the coordinate transformation between and the 3D coordinate system), applying the distortion to the tag coordinates of the at least one tag, and the position and perspective of the at least one tag with respect to the video feed. The process of determining the corresponding video coordinates and perspective data for describing the above, the process of overlaying the three-dimensional rendition of the tag annotation of the at least one tag on the video feed using the tag video coordinates. It is configured to let the processor do it.

0005In some embodiments, the 3D rendition of the tag annotation can be dynamically re-rendered based on the current position of the remote telepresence robot and the perspective of the at least one tag with respect to the video feed.

0006In some embodiments, the 3D rendition of the tag annotation can be overlaid on the video feed for objects detected in the video feed.

0007In some embodiments, the 3D rendition of the tag annotation can be overlaid along the wall detected in the video feed.

0008In some embodiments, the tag information of the at least one tag comprises a telepresence robot motion modifier, wherein the telepresence robot is within a predetermined range of tag coordinates of the at least one tag. It can be configured to provide an execution command to the control system of the telepresence robot to perform a first action in response to being located in.

0009In some embodiments, the instructions that can be executed by the processor further indicate to the processor that the telepresence robot is located within a predetermined range of tag coordinates of the at least one tag. It can be configured to cause the control system to execute a process of transmitting the execution instruction.

0010In some embodiments, the robot motion modifier further comprises commands relating to one of the times and one of the positions on the plan map that the control system of the telepresence robot should perform the first motion. ..

0011In some embodiments, the processor-executable instructions further provide the processor with a process of receiving a sequence of coordinates with respect to the plan map that forms the path traveled by the remote telepresence robot. A process of storing a series of coordinates to be formed as a path tag having tag information that can include tag coordinates and tag annotations, and a process of reading the path tag when the remote telepresence robot reaches within a predetermined distance of the tag coordinates. , And the tag coordinates are configured to perform a process of displaying the rendition of the tag annotation of the route tag on at least one of the plan map and the video feed.

0012In some embodiments, the local terminal of the telepresence robot system further comprises at least one user input device, the sequence of coordinates forming the path can be provided by the user input device.

0013In some embodiments, the sequence of coordinates forming the path can be provided by the remote telepresence robot.

0014In some embodiments, the telepresence robot system further comprises a communication system configured to facilitate communication between the local terminal of the telepresence robot system and the remote telepresence robot.

0015In some embodiments, the local terminal further comprises at least one user input device, wherein the user is at least one of the floor plan map and the video feed from the imaging system of the remote telepresence robot. It can be configured above to provide instructions for the desired destination of the remote telepresence robot, and the command transmitted to the remote telepresence robot can include the desired destination.

0016In some embodiments, the sequence of coordinates forming the local path can be at least partially based on the tag information associated with the at least one tag.

0017In some embodiments, the commands that can be executed by the processor further generate in the processor a robotic path between the current position of the remote telepresence robot and the desired destination of the remote telepresence robot. Therefore, the command is configured to execute a process of determining a sequence of coordinates with respect to the plan view map, and the command transmitted to the remote telepresence robot includes a sequence of coordinates forming the robot path.

0018In some embodiments, the instructions that can be executed by the processor are further configured to cause the processor to overlay a sequence of coordinates forming the robot path on the floor plan map. ..

0019In some embodiments, the instructions that can be executed by the processor further determine to the processor the distortion between the plan map and the video feed received from the imaging system of the remote telepresence robot. The process of applying the strain to a sequence of coordinates forming the robot path to determine corresponding video coordinates and perspective data describing the position and perspective of the sequence of coordinates relative to the video feed, and on the video feed. Is configured to execute a process of overlaying a three-dimensional rendition of a series of coordinates forming the local path.

0020In some embodiments, the three-dimensional rendition of a sequence of coordinates forming the robot path can be overlaid on the video feed with respect to the floor detected in the video feed.

0021In some embodiments, the instructions that can be executed by the processor are further directed to the processor from the navigation system of the remote telepresence robot to the current position of the remote telepresence robot and the desired purpose of the remote telepresence robot. A process of forming a robot path to and from the ground, a process of receiving a sequence of coordinates with respect to the plan view map, and a process of overlaying a sequence of coordinates forming the robot path on the plan view map are executed. It is composed.

0022In some embodiments, the instructions that can be executed by the processor further distort (eg, two-dimensionally) the processor between the plan map and the video feed received from the imaging system of the remote telepresence robot. The process of determining the coordinate transformation between the coordinate system and the three-dimensional coordinate system), applying the strain to the sequence of coordinates forming the robot path, and describing the position and perspective of the sequence of coordinates with respect to the video feed. It is configured to execute a process of determining the corresponding video coordinates and perspective data, and a process of overlaying a three-dimensional rendition of a series of coordinates forming the local path on the video feed.

0023In some embodiments, a three-dimensional rendition of a sequence of coordinates forming the robot path can be overlaid on the video feed with respect to the floor detected in the video feed.

0024In some embodiments, the tag information is the availability of wireless communication signals, the moving speed of the remote telepresence robot, the position of a point of interest, the position of a person, the position of a docking station, the position of a stop area, a glass wall. It has information about the position of the vehicle, the position of the slope, the position of the object, the best route for navigating a narrow area, the best route for navigating a crowded area, and one of the actions that the telepresence robot should perform. ..

0025In some embodiments, the tag information can be about a position, path and / or volume, and the control system can be configured to perform an operation with respect to the position, path and / or volume. ..

0026In some embodiments, the processor-executable instructions further perform a process on the processor to receive coordinates on the floor plan map of the obstacle detected by the sensor system of the remote telepresence robot. It is configured to let you.

0027In some embodiments, the floor plan map and the plurality of tags are stored remotely.

0028In some embodiments, the floor plan map and the plurality of tags are stored within the remote telepresence robot.

0029In some embodiments, the instructions that can be executed by the processor further distort the processor between the floor plan map and the video feed received from the imaging system of the remote telepresence robot (eg, two-dimensional). Generate a mixed map view that includes the process of determining the coordinate transformation between the coordinate system and the 3D coordinate system) and a mixed view of the floor plan map and the video feed from the imaging system of the remote telepresence robot. It is configured to execute the process.

0030In some embodiments, the mixed map view comprises a three-dimensional representation of the plan map overlaid on the video feed.

0031In some embodiments, the local terminal of the telepresence robot system further comprises at least one user input device, and instructions that can be executed by the processor further provide the processor with the at least one input device. A process of receiving a rendered look-ahead view request for the virtual position of the remote telepresence robot on the plan map, a video feed received from the plan map and the imaging system of the remote telepresence robot. The process of determining the distortion between and (for example, the coordinate transformation between the 2D coordinate system and the 3D coordinate system), the process of generating a virtual 3D video feed based on the virtual position of the remote telepresence robot. , And the process of displaying the virtual 3D video feed based on the virtual position of the remote telepresence robot.

0032In some embodiments, the tag information of the at least one tag comprises a set of coordinates that define the protected area with respect to the plan map, and the tag information of the at least one tag indicates the presence of the protected area. It can be configured as shown.

0033In some embodiments, the instructions that can be executed by the processor further include a process of receiving a request from the processor to generate a new tag, the relative position of the tag information that may include the new tag and the tag annotation. To execute the process of associating the tag coordinates that describe the new tag with the new tag, and the process of displaying the rendition of the tag annotation of the new tag on the plan map and at least one of the video feeds using the tag coordinates. It is composed.

0034In some embodiments, the request to generate the new tag can be generated by the remote telepresence robot.

0035In some embodiments, the request to generate the new tag can be automatically generated based on the object detected in the video feed.

0036In some embodiments, the new tag can be a temporary tag configured to time out as soon as the detected object is no longer present in the video feed.

0037In some embodiments, the object can be a person, and the tag information of the new tag can include identification information associated with that person.

0038In some embodiments, the object can be a person, and the tag information of the new tag can include actions that the remote telepresence robot may perform on that person.

0039In some embodiments, the request to generate the new tag can be generated by a user input device that communicates with the local terminal of the remote telepresence robot system.

0040In some embodiments, the request to generate the new tag is made to the video feed.

0041In some embodiments, the request to generate the new tag is made to the floor plan map.

0042In some embodiments, the request to generate the new tag is made to the current position of the remote telepresence robot.

0043In some embodiments, the tag information is the availability of wireless communication signals, the moving speed of the remote telepresence robot, the position of a point of interest, the position of a person, the position of a docking station, the position of a stop area, a glass wall. Information about the position of the vehicle, the position of the slope, the position of the object, the best route to navigate in a small area, the best route to navigate in a crowded area, and one of the actions the telepresence robot should perform. To be equipped.

0044In other embodiments, the telepresence robot can communicate with the remote terminal. The telepresence robot communicates with a drive system configured to move the telepresence robot according to a drive command, the drive system, and generates a drive command for moving the telepresence robot to the drive system. The control system configured in, the imaging system that communicates with the control system, the plan view map that communicates with the control system and represents the robot navigable area on the robot operating surface, and the tag coordinates that each describe the relative position of the tag. And a mapping module configured to access a map data source with multiple tags with tag information (which may include annotations) to communicate with the control system to provide positioning information associated with the current location. The positioning system configured in the above, the tag identification system configured to identify at least one tag related to the navigation path of the telepresence robot, and the easy communication between the control system and the remote terminal. The control system is configured to perform an action based on the identified tag, the tag information comprising a telepresence robot action modifier.

0045In some embodiments, the tag information of the identified tag comprises instructions regarding the time the control system should perform an operation and at least one of its positions on the floor plan map.

0046In some embodiments, the control system can be configured to transmit video feed from the imaging system to the remote terminal via the communication system, and the control system is from the remote terminal to the plane. The desired destination on the diagram map can be configured to be received via the communication system.

0047In some embodiments, the telepresence robot further comprises a plurality of sensors configured to identify obstacles near the telepresence robot, and obstacles that communicate with the plurality of sensors and the control system. An evasion system can be provided, which can be further configured to generate additional commands to evade obstacles near the telepresence robot.

0048In some embodiments, the plurality of sensors include at least one of a proximity sensor, a contact sensor, an odometry sensor, and a three-dimensional image sensor.

0049In some embodiments, the plurality of sensors may include a three-dimensional image sensor that forms a point cloud that includes a three-dimensional occupancy of the obstacle, and the drive command occupies the three-dimensional occupancy of the obstacle. It can be configured to avoid it.

0050In some embodiments, the telepresence robot can further include a map generation system configured to communicate with the control system and autonomously generate a plan view map of the robot's operating surface. The control system moves the telepresence robot over the entire operating surface of the robot to generate a drive command for acquiring a plurality of measured values, and the map generation system uses the plurality of measured values to generate the plan view map. To generate.

0051In some embodiments, the telepresence robot is further configured to generate a navigation path with a sequence of coordinates from a current position on the floor plan map to a desired destination on the floor plan map. It can be equipped with a navigation system.

0052In some embodiments, the telepresence robot can transmit the coordinates of the detected obstacle to the plan map to the remote terminal via the communication system.

0053In some embodiments, the sequence of coordinates forming the navigation path can be at least partially based on the tag information associated with the identified tag.

0054In some embodiments, the navigation system is configured to generate the navigation path by selecting one navigation path from a plurality of possible navigation paths, and tags associated with the navigation path of the telepresence robot. Is associated with the plurality of possible navigation routes, and the navigation system is configured to select the navigation route based at least in part on the identified tag.

0055In some embodiments, the sequence of coordinates forming the navigation path is transmitted to the remote terminal via the communication system.

0056In some embodiments, the telepresence robot uses a sequence of coordinates that form the navigation path to provide a new tag that includes the sequence of coordinates, tag information associated with the navigation path, and tag annotations relating to the navigation path. Can be configured to generate.

0057In some embodiments, the tag information of each of the plurality of tags includes the availability of wireless communication signals, the moving speed of the remote telepresence robot, the position of a point of interest, the position of a person, the position of a docking station, and the like. The position of the stop area, the position of the glass wall, the position of the slope, the position of the object, the best route for navigating a narrow area, the best route for navigating a crowded area, and the actions that the telepresence robot should perform. Have information about one of them.

0058In some embodiments, the control system can be further configured to receive a navigation path from a current position on the plan map to a desired destination on the plan map, and the control system is Further, the drive system can be configured to generate a drive command for moving the telepresence robot to the desired destination based on the navigation path.

0059In some embodiments, the communication system can be configured to detect interruptions in communication between the telepresence robot and a remote terminal, and the control system can also be configured to detect interruptions in communication. Can be configured to continue to generate drive commands to move autonomously.

0060In some embodiments, the map data source can be stored remotely and the mapping module can be configured to access the map data source via the communication system.

0061In some embodiments, the map data source can be stored within the telepresence robot and the mapping module can be configured to access an internal map data source.

0062In some embodiments, the internal map data source can be synchronized with a remotely stored map data source.

0063In some embodiments, the position-fixing system can be further configured to provide a robot posture with respect to the floor plan map.

0064In some embodiments, the telepresence robot associates a new tag with tag coordinates that describe the relative position of the new tag with one of the floor plan map and the video feed generated by the imaging system. It can be configured to be generated by associating the tag information with the new tag and associating the tag annotation with the new tag.

0065In some embodiments, the new tag can be generated in response to the telepresence robot detecting an object in the video feed.

0066In some embodiments, the object can be a person and the tag information of the new tag can include identification information associated with the person.

0067In some embodiments, the object can be a person, and the tag information of the new tag can comprise an action that the remote telepresence robot can perform on the person.

0068In some embodiments, the tag information is the availability of wireless communication signals, the moving speed of the remote telepresence robot, the position of a point of interest, the position of a person, the position of a docking station, the position of a stop area, a glass wall. Information about the position of the vehicle, the position of the slope, the position of the object, the best route to navigate in a small area, the best route to navigate in a crowded area, and one of the actions the telepresence robot should perform. To be equipped.

0069In some embodiments, the telepresence robot system further comprises an RFID reader that communicates with the position-fixing system, which places a plurality of RFID chips at corresponding coordinates on the plan view map. In connection with, the positioning system can be configured to determine the current position of the telepresence robot based at least in part on the position of one or more RFID chips located within the range of the RFID reader.

0070Various control methods can be used in the systems and methods of the present invention. For example, a local terminal of a telepresence robot system comprises an electronic display, a processor that communicates with the electronic display interface, and a memory that communicates with the processor, the memory comprising instructions that can be executed by the processor. The instruction is a process of reading at least a part of a plan view map representing a robot navigable area of the robot operating surface to the processor, a process of receiving a video feed from the imaging system of the remote telepresence robot in the first perspective, the process of receiving the video feed from the imaging system of the remote telepresence robot. To execute the process of receiving the current position of the remote telepresence robot with respect to the plan view map, the process of displaying the video feed from the imaging system of the remote telepresence robot, and the process of transmitting a command to the remote telepresence robot. The user input device is configured to allow the user to select the movement of the remote telepresence robot, and the selection of the movement is the purpose of the remote telepresence robot. By incrementally advancing the ground in one of at least four directions with respect to the video feed, the plan map, and the remote telepresence robot's current position. , Including selection.

0071In some embodiments, the move selection comprises selecting another perspective of the video feed by selecting one point in the video feed. This mode can be used for medium distances to reach a position in the view of the video feed.

0072In some embodiments, the movement selection comprises selecting another perspective of the video feed by selecting one point in the floor plan map. This mode can be used for long distances (eg, corridors, rooms, etc.) to positions that are not in the view of the video feed. In some embodiments, the choice of movement involves using manual control such as a joystick or meta-joystick. This mode can be used, for example, for fine tuning in a room in the immediate vicinity of a person / parent.

0073In some embodiments, the movement selection includes selecting another perspective of the video feed by incrementally panning or tilting the imaging device while keeping the remote telepresence robot stationary.

0074In some embodiments, the choice of movement can be associated with rotating one of the lower part of the remote telepresence robot and the upper part of the remote telepresence robot.

0075In some embodiments, the modes can be switched, eg, using a multi-node interface, one can choose to move the head imaging system of the remote telepresence robot or move the berth (bottom).

0076In some embodiments, when selecting control of movement of the head imaging system, mouse-based position-based box-zoom head motion or mouse-based speed-based head motion can be selected, if desired.

0077In some embodiments, when selecting control of the base (bottom) of the remote telepresence robot, (1) click-on-map, ie, map view top-down and click on a target destination, as needed. Select from a destination list, (2) click-on video, that is, position-based control that allows the robot to click on a location in the video, and (3) joystick or meta-joystick, such as mouse speed-based control or You can select one of the arrows that point to the front, left, right, and so on.

0078In some embodiments, the functions / information that need to be accessed by the user at all times while the robot base is moving are: (1) remote view, i.e. the robot is facing forward (view is safe operation). Must be large enough to give the user significant visual information for), (2) Surveillance control mode, override function may be required for cancel / cancel operations as needed, including.

0079In some embodiments, the instructions that can be executed by the processor further include a process of receiving the processor from the user input device to select the destination of the remote robot, the current position of the remote telepresence robot, and so on. The remote includes a process of determining a sequence of coordinates with respect to the floor plan map to generate a navigation path to and from a destination selected by the remote telepresence robot, and a command comprising a sequence of coordinates forming the navigation path. It is configured to execute the process of transmitting to the telepresence robot.

0080In some embodiments, the instructions that can be executed by the processor are further configured to cause the processor to overlay a sequence of coordinates forming the navigation path on the floor plan map. ..

0081In some embodiments, the instructions that can be executed by the processor further distort (eg, two-dimensionally) the processor between the plan map and the video feed received from the imaging system of the remote telepresence robot. The process of determining the coordinate transformation between the coordinate system and the three-dimensional coordinate system), the distortion is applied to the sequence of coordinates forming the navigation path, and the position and perspective of the sequence of coordinates with respect to the video feed are described. It is configured to execute a process of determining the corresponding video coordinates and perspective data, and a process of overlaying a three-dimensional rendition of a series of coordinates forming the navigation path on the video feed.

0082In some embodiments, the three-dimensional rendition of the coordinate sequence forming the navigation path can be overlaid on the video feed with respect to the floor detected in the video feed.

0083In some embodiments, the instructions that can be executed by the processor are further such that the processor receives a selection of a destination for the remote robot from the user input device, the operation corresponding to the selected destination. The process of transmitting the destination coordinates with respect to the plan view map to the remote telepresence robot, between the current position of the remote telepresence robot and the desired destination of the remote telepresence robot from the navigation system of the remote telepresence robot. It is configured to execute a process of receiving a sequence of coordinates with respect to the plan view map forming the robot path and a process of overlaying the sequence of coordinates forming the robot path on the plan view map.

0084In some embodiments, the instructions that can be executed by the processor further distort (eg, two-dimensionally) the processor between the plan map and the video feed received from the imaging system of the remote telepresence robot. The process of determining the coordinate transformation between the coordinate system and the three-dimensional coordinate system), applying the strain to the sequence of coordinates forming the robot path, the position of the sequence of coordinates relative to the video feed, and It is configured to execute a process of determining corresponding video coordinates and perspective data for describing a perspective, and a process of overlaying a three-dimensional rendition of a series of coordinates forming the navigation path on the video feed.

0085In some embodiments, a three-dimensional rendition of a sequence of coordinates forming the robot path can be overlaid on the video feed with respect to the floor detected in the video feed.

0086In some embodiments, the instructions that can be executed by the processor further cause the processor to perform a process of receiving coordinates on a plan map of an obstacle detected by the sensor system of the remote telepresence robot. It is configured as follows.

0087In some embodiments, the floor plan map is stored remotely.

0088In some embodiments, the floor plan map is stored within the remote telepresence robot.

0089In some embodiments, the instructions that can be executed by the processor are distortions (eg, 2D and 3D coordinates) between the floor plan map and the video feed received from the imaging system of the remote telepresence robot. To execute a process of determining (coordinate transformation with and from the system) and a process of generating a mixed map view including a mixed view of the floor plan map and the video feed from the imaging system of the remote telepresence robot. It is composed.

0090In some embodiments, the mixed map view comprises a three-dimensional representation of the plan map overlaid on the video feed.

0091In some embodiments, the instructions that can be executed by the processor further include processing the processor to receive a look-ahead view request for a virtual position of the remote telepresence robot on the plan view map, said plan view. A process of determining the distortion between the map and the video feed received from the imaging system of the remote telepresence robot (eg, coordinate transformation between a 2D coordinate system and a 3D coordinate system) of the remote telepresence robot. It is configured to execute a process of generating a virtual 3D video feed based on a virtual position and a process of displaying the virtual 3D video feed based on the virtual position of the remote telepresence robot.

0092In some embodiments, the robot can be configured to independently rewind and / or control the top and bottom so that it is visible to humans. For example, the telepresence robot communicates with and drives the upper part, the lower part rotatably connected to the upper part, the drive system configured to drive the telepresence robot according to a drive command, and the drive system. A control system configured to generate a drive command to move the telepresence robot to the system, and by rotating the upper part and the lower part independently, the robot is moved from the first traveling direction to the second traveling direction. It comprises a rotation system configured to rotate to.

0093In some embodiments, the rotation system rotates the top of the robot in the second direction of travel and detects that the top of the robot has reached the pan limit of the top of the robot relative to the bottom of the robot. Then, at the pan limit of the upper part of the robot, the lower part of the robot starts to rotate to the second traveling direction, and it is detected that the upper part of the robot has reached the second traveling direction, and the upper part of the robot moves. The robot is moved in the second direction by continuously rotating the lower part of the robot to the second direction of travel and at the same time rotating the upper part of the robot in the opposite direction so as to maintain the second direction of travel. It can be configured to rotate in a direction.

0094In some embodiments, the pan limit may be reached when the upper part of the robot cannot physically rotate further with respect to the lower part of the robot.

0095In some embodiments, the pan limit may be reached when the upper part is misaligned with respect to the lower part by a predetermined number of revolutions.

0096In some embodiments, the pan limit can be a function of the number of revolutions at which the upper part is misaligned with respect to the lower part and the time the upper part is misaligned with respect to the lower part.

0097In some embodiments, the rotation system rotates the upper part of the robot at a first rotational speed to the second traveling direction and the lower part of the robot at a second rotational speed in the second traveling direction. Detects that the upper part of the robot has reached the second traveling direction, and the lower part of the robot maintains the second traveling direction so that the upper part of the robot maintains the second traveling direction. By rotating the upper part of the robot in the opposite direction at the same time as continuing to rotate in the direction, the robot can be configured to rotate in the second traveling direction.

0098In some embodiments, the telepresence robot further communicates with an imaging system that communicates with the control system and the current position of the robot with respect to the plan map and the upper part of the telepresence map with respect to the plan map. With a positioning system configured to provide the current alignment, the control system sends a video feed from the imaging system, the current position of the robot and the current alignment of the top to the remote terminal. As a result, the remote terminal distorts between the plan view map and the video feed received from the imaging system of the telepresence robot (eg, coordinate transformation between a two-dimensional coordinate system and a three-dimensional coordinate system). And apply the distortion to a tag having coordinates associated with the plan map to determine the corresponding video coordinates and perspective data describing the position and perspective of the tag with respect to the video feed, and the video coordinates. It can be used to configure the three-dimensional rendition of the tag to overlay on the video feed.

0099The above embodiments are described in terms of robots and / or local terminals. Those skilled in the art will appreciate that the above embodiments can be implemented as a system, as a method performed by the system, and can be embodied in a computer-readable medium that can be implemented by the system. For example, a method of changing the traveling direction of a robot is described by means of a step of transmitting the traveling direction to a robot control system configured to communicate with a drive system and move the robot according to a drive command, and by the drive system. A step of rotating the upper part of the robot in the traveling direction independently of the lower part of the robot can be provided.

0100In some embodiments, the method of controlling a remote telepresence robot is to receive at least a portion of a plan map representing the robot's navigable area of the robot's motion surface, tag coordinates, each describing the relative position of the tag, and A step of receiving at least one of a plurality of tags having tag information, a step of receiving a video feed from an imaging system of a remote telepresence robot, a step of receiving positioning information related to the current position of the remote telepresence robot, The step of displaying the video feed from the imaging system of the remote telepresence robot on the electronic display, using the tag coordinates to display the rendition of the tag information of the at least one tag on the video feed on the electronic display. A step of sending a command to the remote telepresence robot can be provided. The method of controlling the telepresence robot is a step of reading at least a part of the plan view map, a step of reading out at least one of a plurality of tags each having tag coordinates and tag information describing the relative position of the tag, and the plan view map. Based on the identified tag having tag information including a step of determining the current position with respect to, a step of the plurality of tags related to the navigation path of the telepresence robot, a step of identifying one tag, and a tag information including a telepresence behavior modifier. It is possible to provide a step of executing the operation.

0101In some embodiments, the method of controlling the telepresence robot is to read at least a portion of a plan map representing the robot's navigable area of the robot's motion surface, a video feed from the remote telepresence robot's imaging system. A step of receiving in the first perspective, a step of receiving positioning data related to the current position of the remote telepresence robot, a step of displaying the video feed from the imaging system of the remote telepresence robot, the remote telepresence. It comprises a step of sending a command to the robot and a step of receiving a plurality of movement selections from a user input device, the movement selections being made (1) for the video feed and (2) for the plan view map. And / or by incrementally advancing the remote telepresence robot in one direction with respect to the current position of the remote telepresence robot.

0102Details of one or more embodiments of the present disclosure are shown in the accompanying drawings and are described below. Other aspects, features and advantages will become apparent from the description herein, drawings and claims.

0103<figref num="1">It is a perspective view of a model telepresence robot.</figref><figref num="2">It is an elevation angle perspective view of a model telepresence robot.</figref><figref num="3A">It is a schematic diagram of a model telepresence robot.</figref><figref num="3B">It is a schematic diagram of a model telepresence robot.</figref><figref num="3C">It is a schematic diagram of a model telepresence robot.</figref><figref num="4A">It is a front perspective view of the model base part of a mobile human interface robot.</figref><figref num="4B">It is a rear perspective view of the base part shown in FIG. 4A.</figref><figref num="4C">It is the top view of the base part shown in FIG. 4A.</figref><figref num="4D">It is a top view of the model base part of a telepresence robot.</figref><figref num="4E">It is a bottom perspective view of a model drive system of a telepresence robot.</figref><figref num="4F">It is a schematic diagram of the drive system shown in FIG. 4E.</figref><figref num="5">It is a schematic diagram of an exemplary control system executed by a controller of a telepresence robot.</figref><figref num="6A">A schematic diagram of an exemplary robot system containing a large number of robots communicating with a robot endpoint is shown.</figref><figref num="6B">Shows a teleoperations software application running on a robot or terminal.</figref><figref num="6C">An example of a screenshot of a user interface that controls the navigation of a semi-autonomous telepresence robot is shown.</figref><figref num="6D">Shows a screenshot in which the relative area of the screen assigned to the map window has been increased.</figref><figref num="7">It is a schematic diagram of an exemplary robot system architecture.</figref><figref num="8A">It is a schematic diagram of a model occupancy map.</figref><figref num="8B">It is a schematic diagram of a telepresence robot having a field of view of a scene in a work area.</figref><figref num="8C">It is a schematic diagram of a model layout map.</figref><figref num="8D">It is a schematic diagram of a model robot map corresponding to the layout shown in FIG. 8C.</figref><figref num="8E">A layout map and a robot map are used to show an exemplary configuration in which a robot is operated to navigate in an environment.</figref><figref num="8F">Demonstrates how to use the robot position and perspective to determine the distortion between the video feed and the plan map.</figref><figref num="9A">It is a schematic diagram of an exemplary remote video view from a robot located in the middle.</figref><figref num="9B">It is a schematic diagram of an exemplary mixed map consisting of a combination of a remote video view shown in FIG. 9A and a map showing room numbers.</figref><figref num="10A">Demonstrates an exemplary remote view of a remote video window for a telepresence software application.</figref><figref num="10B">FIG. 10A is a schematic map of an exemplary map of the area shown by the remote view in A.</figref><figref num="10C">Schematic of an exemplary look-ahead view of a telepresence software application.</figref><figref num="10D">FIG. 6 is a schematic diagram of a map shown in FIG. 10B with a robot icon and a corresponding camera field of view.</figref><figref num="10E">Schematic of an exemplary look-ahead view of a telepresence software application.</figref><figref num="10F">FIG. 6 is a schematic diagram of a map shown in FIG. 10B with a robot icon and a corresponding camera field of view.</figref><figref num="10G">Shows an exemplary array of look-ahead routine operations for telepresence software applications.</figref><figref num="11A">Shown is a schematic of an exemplary user interface that allows a user to specify a robot destination within an identified navigable area.</figref><figref num="11B">An exemplary array of operations for how to navigate a robot to its destination is shown.</figref><figref num="11C">It is a schematic diagram of an exemplary user interface that warns the user that a slope has been selected as the destination of the robot.</figref><figref num="11D">It is a schematic diagram of an exemplary user interface that warns the user that an obstacle has been selected as the destination of the robot.</figref><figref num="12">FIG. 6 is a schematic representation of an exemplary user interface that allows a user to specify a robot drive path within an identified navigable area.</figref><figref num="13">It is a schematic diagram of an exemplary user interface with built-in hypertags and context sensitive commands.</figref><figref num="14">It is a perspective view of a model telepresence robot that maintains a sensor field of view in a person.</figref><figref num="15A">It is a schematic diagram of an exemplary 3D map view including hypertags.</figref><figref num="15B">It is a schematic diagram of an exemplary 2D map view including hypertags.</figref><figref num="16A">It is a schematic diagram of a model robot system.</figref><figref num="16B">It is a schematic diagram of an exemplary interaction with a map data source.</figref><figref num="16C">It is a schematic diagram of an exemplary interaction between a robot control system and a map data source.</figref><figref num="16D">It is a schematic diagram of a model robot system.</figref><figref num="17">It is a schematic diagram of an exemplary user interface including an enlarged overlay corresponding to a telepresence robot.</figref><figref num="18">It is a schematic diagram of a series of exemplary robot movements.</figref><figref num="19">FIG. 6 is a schematic representation of a user interface with a screen indicator overlaid on a remote video feed received from a telepresence robot.</figref><figref num="20A">An exemplary array of operations for recovering from the loss of robotic communication is shown.</figref><figref num="20B">An exemplary array of operations for recovering from the loss of robotic communication is shown.</figref><figref num="20C">An exemplary array of operations for recovering from the loss of robotic communication is shown.</figref>

0104Similar reference numbers in various figures indicate similar elements.

0105The telepresence robot can interact with or interface with humans to provide a number of services, such as remote consultations for doctors and healthcare professionals, home support, and commercial support. In the home support example, telepresence robots can assist the elderly in their daily work, including, but not limited to, maintenance of dosing plans, mobility support, and communication support (eg video conferencing, communications, internet access, etc.). Includes provision of home or site surveillance (indoor and / or outdoor), personal surveillance, and / or personal emergency response system (PERS). For commercial support, telepresence robots can provide video conferencing (eg, hospital settings), point-of-sale terminals, interactive information / marketing terminals, and more.

0106Referring to FIG. 1-3B, in some embodiments, the telepresence robot 100 includes a robot body 110 (or chassis) that defines a forward direction F. The robot 100 also includes a drive system 200 (FIG. 4D), an interface module 300 and a sensor system 400, which are supported by the robot body 110 and communicate with the controller 500 (FIG. 5) that controls the movement and movement of the robot 100. To do. The power source 105 (eg, a battery) is supported by the robot body 110 and electrically connected so that power can be supplied to each of these components as needed.

0107In the illustrated example, the robot body 110 includes a base 120, at least one leg 130 extending upward from the base 120, and a torso 140 supported by at least one leg 130. The robot body (lower part) 110 also includes a neck portion 150 supported by the body portion 140. The neck portion 150 supports the head portion (upper portion) 160, and the head portion 160 supports at least a part of the interface module 300. The base 120 contains sufficient weight (for example, by supporting the power supply 105 (battery)) and the center of gravity CG of the base 120._BAnd the overall center of gravity CG of the robot 100_RKeep low to maintain mechanical stability.

0108With reference to FIGS. 2 and 4A-4C, in some embodiments, the base 120 defines a triangular symmetry (eg, a triangle when viewed from above). For example, the base portion 120 includes a base chassis 122 that supports a base body 124 having first, second, and third base body portions 124a, 124b, 124c corresponding to each side of the triangular base portion 120. Can be done (see, eg, Figure 4A). Each base body portion 124a, 124b, 124c can be movably supported by the base chassis 122 so that it moves independently of the base chassis 122 in response to contact with an object. The triangular symmetry of the base 120 enables collision detection at 360 ° around the robot 100. Each base body portion 124a, 124b, 124c has an associated contact sensor (eg, capacitance sensor, reed switch, etc.) and can detect the movement of the corresponding base body portion 124a, 124b, 124c with respect to the base chassis 122. ..

0109In some embodiments, the drive system 200 provides omnidirectional and / or holonomic movement control of the robot 100. As used herein, "omnidirectional" refers to the ability to move in almost any plane direction, ie side-to-side (lateral), front-back and rotation. These directions are commonly referred to here as x, y and θz, respectively. Further, "holonomic" is almost the same as the term used in the literature, and refers to a plane direction having three plane degrees of freedom, that is, two translations and one rotation. Therefore, the holonomic robot has the ability to move in the plane direction at a speed consisting of almost any ratio of the three plane velocities (lateral direction, front-back and rotation), and has the ability to change these ratios almost continuously. ..

0110Robot 100 operates in a human environment (eg, an environment typically designed for bipedal occupants) using wheel movement. In some embodiments, the drive system 200 has first, second and third drive wheels 210a, which are equidistant (ie, triangulated) about the vertical axis Z at equal intervals (eg, 120 ° intervals). Includes 210b and 210c, but other arrangements are possible. Referring to FIG. 4D, the drive wheels 210a, 210b, 210c have a lateral arcuate rolling surface (ie, rolling direction D)._RThe curved surface shape in the direction crossing or at a right angle) can be specified, and the mobility of the holonomic drive system 200 can be given. The drive wheels 210a, 201b, 201c are coupled to the drive motors 220a, 220b, 220c, and the drive motors 220a-c independently move the drive wheels 210a, 210b, 210c forward and / or backward. Can be driven. Each drive motor 220a-c may have a separate encoder that feeds back wheel rotation to the controller 500. In some embodiments, each drive wheel 210a, 210b, 210c is mounted at or near one of the three vertices of an equilateral triangle and has a drive direction perpendicular to the bisector of the apex of each triangle. Has (forward / backward direction). When the triangulated holononomic base 120 is driven in the forward drive direction F, the robot 100 allows a transition to a non-forward drive direction for restraint or self-sustaining escape from the clutter, and then rotates after the escape is resolved. And / or translate and drive along the forward drive direction F.

0111With reference to FIGS. 4E and 4F, in some embodiments, the drive system 200 is arranged in a rectangular or rectangular shape when viewed from above (ie, equidistant from the Z axis). Includes 2, 3 and 4 drive wheels 210a-d. The drive system 200 can operate nonholonomically, allowing strafing. Each drive wheel 210a-d is coupled to a respective drive motor 220a-d, and each drive motor 220a-d can independently drive the drive wheels 210a-d in the forward and / or backward directions. Each drive motor 220a-d may have a separate encoder that feeds back wheel rotation to the controller 500. The base chassis 122 supports the drive motor 220a-d and the correspondingly coupled drive wheels 210a-d.

0112In some embodiments, as shown in FIG. 2, the first drive wheels 210a are arranged as leading wheels along the forward drive direction F and the remaining two drive wheels 210b, 210c are arranged as trailing wheels. .. In this arrangement, for forward drive, the controller 500 can generate a drive command to drive the second and third drive wheels 210b, 201c at a constant speed in the forward rolling direction, while the first drive wheel 210a It slips in the forward drive direction F. Further, this drive wheel arrangement can cause the robot 100 to stop suddenly (eg, it produces a rapid negative acceleration with respect to the forward drive direction F). This is due to the inherent dynamic stability of the three-wheel drive design. If the forward drive direction F is along the bisector of the angle between the two forward drive wheels, the sudden stop produces a force that causes the robot 100 to fall, turning on the two "front" wheels. Instead, forward travel on one drive wheel 210a naturally supports the robot 100 and prevents it from tipping over while moving forward if a sudden stop is required. However, when accelerating from a stop, the controller 500 is the total center of gravity AG of the robot 100._RThe moment of inertia I from can be considered.

0113In some embodiments of the drive system 200, each drive wheel 210a, 210b, 210c rolls in a radial direction with a vertical Z axis perpendicular to the X and Y axes of the robot 100._RHave. The first drive wheel 210a can be arranged as a front wheel along the forward drive direction F, and the remaining two drive wheels 210b and 210c can be arranged as trailing wheels. In this arrangement, for forward drive, the controller 500 drives the first drive wheel 210a in the forward rolling direction and the second and third drive wheels 210b, 210c at a speed equal to the first drive wheel 210a. It is possible to generate a drive command that drives in the opposite direction.

0114In another embodiment, the drive system 200 is arranged such that the first and second drive wheels 210a, 210b align the bisector of the angle between them with the forward drive direction F of the robot 100. Can be configured. In this arrangement, for forward drive, the controller 500 drives the first and second drive wheels 210a, 210b at a constant speed in the forward rolling direction, but does the third drive wheel 210c drive in the opposite direction? , Drive commands can be generated to be dragged behind the first and second drive wheels 210a, 210b. To turn left or right during forward drive, controller 500 can generate a drive command to drive the corresponding first or second drive wheels 210a, 210b at a relatively fast or slow speed. Other drive system configurations can also be used. The drive wheels 210a, 210b, 210c can be cylindrical, circular, oval or polygonal.

0115With reference to FIG. 1-3B again, the base portion 120 supports at least one leg portion 130 extending upward in the Z direction from the base portion 120. The legs 130 can be configured to be adjustable in height to raise and lower the torso 140 with respect to the base 120. In some embodiments, each leg 130 comprises a first and second leg portion that moves relative to each other (eg, telescopic, linear and / or angular movement). In the illustrated example, the second leg portion 134 does not have smaller diameter extrusions in the order of nesting in and out of each other from the relatively large base extrusion portion, and the second leg portion 134 nests around the first leg portion 132. It can be moved, and thus other components can be placed along the second leg portion 134 and moved with the second leg portion 134 to the near point of the base portion 120. The leg 130 may include an actuator assembly that moves the second leg 134 relative to the first leg 132. Actuator assembly 136 may include a motor driver that works with the lift motor and an encoder that feeds back the position to the controller.

0116Generally, the telescopic device is the center of gravity CG of the leg 130._LIn order to keep the base 120 as low as possible, the relatively large extrusions of the base 120 include the smaller diameter extrusions in the order of nesting in and out. In addition, stronger and / or larger components can be placed at the bottom to cope with the large torque applied to the base 120 when the legs 130 are fully extended. However, this approach presents two problems. First, when a relatively small component is placed at the top of the leg 130, rain, dust and other particles will flow down or down into the extrusion section and enter the gap between the extrusion sections, thus extruding. The movement of the part is hindered. This creates extremely difficult sealing problems and attempts are still being made to maintain full mobility / joints in leg 130. Secondly, it is desired to mount luggage or accessories on the robot 100. One common place to mount accessories is at the top of the fuselage 140. If the second leg portion 134 nests in and out of the first leg portion 132, the accessories and components of the second leg portion 134 should be moved with the torso 140. It can only be mounted above the whole. Otherwise, the components mounted on the second leg portion 134 will limit the telescopic movement of the leg portion 130.

0117By configuring the second leg portion 134 to move around the first leg portion 132 in a nested manner, the second leg portion 134 can move perpendicular to the base portion 120 as an additional luggage loading point. I will provide a. In this type of configuration, water and aerial particles flow down the outside of the body 140 and all leg 132,134 (extruded) and do not enter the gap between the leg 132,134. This greatly simplifies the sealing of all seams on the leg 130. Further, the luggage / accessory mounting mechanism of the torso 140 and / or the second leg 134 is always exposed and can be used no matter how the leg 130 is stretched.

0118Referring to FIG. 2, the leg 130 may support the torso 140, which may have a step 142 extending over the base 120. In the illustrated example, the body 140 has a downward surface or bottom surface 144 (eg, facing the base) and an opposite upward surface or top surface 146 (FIG. 3A) that form at least a portion of the step 142, between which. Side 148 extends. The torso 140 has various shapes or shapes such as a circle and an ellipse, and has a central portion 143 supported by the leg portion 130 and a peripheral free portion 143 that extends laterally beyond the lateral limit of the leg portion 130. It may provide an overhang that has and thus forms a downward surface 144. In some embodiments, the body 140 has a polygonal or other complex shape that forms a step that provides an overhang that extends beyond the legs 130 to cover the base 120.

0119Robot 100 can include one or more accessory ports 170 (eg, mechanical and / or electrical interconnect points) that accept cargo. The accessory port 170 can be arranged (eg, on the bottom and / or top 144,146 of the torso 140) so that the received luggage does not block or block the sensor of the sensor system 400.

0120The outer surface of the torso 140 may be sensitive to touch or touch by the user in order to receive touch commands from the user. For example, when the user touches the top surface 146 of the torso 140, the robot 100 responds by lowering the torso height with respect to the floor (eg, reducing the height of the legs 130 supporting the torso 140). .. Similarly, when the user touches the bottom surface 144 of the torso 140, the robot 100 responds by increasing the height of the torso relative to the floor (eg, increasing the height of the legs 130 supporting the torso 140). To do. Further, when the user touches the front surface, the back surface, the right side surface or the left side surface portion of the body portion 140, the robot 100 responds by moving in the corresponding directions (front, rear, right, left, respectively) of the received touch command. The outer surface of the torso 140 can include a capacitance sensor that communicates with a controller to detect user contact.

0121With reference to FIG. 1-3B, the torso 140 supports the neck 150, which provides the pan and chilch of the head 160 with respect to the torso 140. In the illustrated example, the neck portion 150 includes a rotor 152 and a tilt portion 154. The rotor 152 has an angular motion range θ between about 90 ° and 360 °._RProvide (eg, centered on the Z axis). Further, in some embodiments, the rotor 152 provides unlimited continuous 360 ° rotation of the head 160 with respect to the torso 140 while maintaining telecommunications between the head 160 and the rest of the robot 100. Includes electrical connectors or contacts that allow for. The tilt portion 154 includes an electrical connector or contact that allows the head portion 160 to rotate relative to the body portion 140 while maintaining telecommunications between the head portion 160 and the rest of the robot 100. The rotor 152 can include a rotor motor coupled to or engaged with a ring (eg, a toothed ring rack). The tilt portion 154 has the head portion independent of the rotor 152, and the angle θ with respect to the body portion 140 (for example, about the Y axis)._TCan be moved with. In some embodiments, the tilt portion 154 includes a tilt motor, which tilts the head portion 160 at an angle θ of ± 90 ° with respect to the Z axis._TMove within the range of. Other ranges such as ± 45 ° are also possible. The robot 100 can be configured such that the legs 130, the body 140, the neck 150 and the head 160 fit within the boundaries of the base 120 in order to maintain the stable mobility of the robot 100.

0122The head unit 160 may be sensitive to a touch or touch by the user in order to receive a touch command from the user. For example, when the user pulls the head 160 forward, the head 160 tilts forward with passive resistance and holds its position. Further, when the user pushes / pulls the head portion 160 vertically downward, the body portion 140 can lower the head portion 160 (due to the reduction in the length of the leg portion 130). The head portion 160 and / or the neck portion 150 may include a strain gauge and / or a contact sensor 165 that detects a user's contact or operation.

0123In some embodiments, the head portion 160 supports one or more portions of the interface module 300. The head portion 160 may include a dock portion 302 that detachably accepts one or more computer tablets (also referred to as web pads or tablet PCs, each having a touch screen). The web pad 310 can be forward, backward or upward. In some embodiments, the web pad 310 includes a touch screen, optional I / O (eg buttons and / or connectors such as micro USB), a processor, and a memory that communicates with the processor. Representative Webpad 310 includes Apple's Apple iPad®. In some embodiments, the web pad 310 either functions as controller 500 or assists controller 500 in controlling robot 100. The touch screen detects, monitors, and / or reproduces the user's touch points in order to receive user input and provide a touch-interactive graphic user interface. In some embodiments, the web pad 310 includes a touch screen caller that allows the user to find when the web pad has been removed from the robot 100.

0124The interface module 300 can include a camera 320 placed above the head 160 (see Figure 3A), which is used to capture video from the raised vantage point of the head 160. Can be used (eg for video conferencing). In the embodiment shown in FIG. 2, the camera 320 is arranged on the neck portion 150. In some embodiments, the camera 320 operates only when the web pad 310 is removed or detached from the dock 302. When the web pad 310 is attached to or docked with the dock 302 of the head 160 (covering the camera 320 as needed), the robot 100 can use the camera of the web pad 310 to take a video. In such a case, the camera 320 can be positioned behind the docked web pad 310 and enters the active state when the web pad 310 is removed from the head 160, and the web pad 310 enters the head 160. Becomes inactive when mounted or docked in.

0125The robot 100 can provide video conferencing via the interface module 300 (eg, using a web pad 310, a camera 320, a microphone 330 and / or a speaker 340). Video conferencing can be multi-party. The robot 100 can make eye contact between members of the video conference by operating the head unit 160 so as to face the user. Further, the robot 100 can have a gaze angle of &lt;5 ° (eg, an angle away from the axis perpendicular to the front surface of the head portion 160). At least one 3D image sensor 450 and / or camera 320 on the robot 100 can take full-scale images including body language. Controller 500 can synchronize audio and video (with a time difference of &lt;50ms). The camera 320 can be moved within a free range of at least 1 ° apart from the web pad 310. The head 160 may include one or more speakers 340 so that audio is emitted from the head 160 in the vicinity of the web pad 310 displaying the video conference.

0126The interface module 300 can include a microphone 330 (or microphone array) that receives audio input and one or more speakers 340 that are located in the robot body 110 to deliver audio output.

0127With reference to Figure 1-3C, in order to achieve reliable and robust autonomous movement, the sensor system 400 includes several different types of sensors, which are intelligent about the actions that the robot 100 takes in the environment. Used in conjunction with each other to generate sufficient robotic environmental awareness to make decisions. The sensor system 400 can include one or more types of sensors supported by the robot 110, which sensors can include obstacle detection obstacle avoidance (ODOA) sensors, communication sensors, navigation sensors, and the like. For example, these sensors are not limited to: proximity sensors, contact sensors, 3D image / depth map sensors, cameras (eg visible and / or infrared cameras), sonars, radars, photodetection ranging. (LIDAR) (which may require optical remote sensing to measure the characteristics of scattered light to detect distance and / or other information at a remote target), laser detection ranging (LADAR), and the like can be included. In some embodiments, the sensor system 400 includes a range-finding sonar sensor 410 (eg, nine around the base 120), a proximity cliff detector 420, a contact sensor 430 (FIG. 3A), a laser scanner 440, one or more. Includes 3D image / depth sensor 450 and image sonar 460.

0128In some embodiments, the sensor system 400 is located in one or more areas or parts of the robot 100 to detect nearby or invading obstacles (eg, base body portions 124a, 124b of the robot body 110). , Includes a set or set of proximity sensors 410,420 that communicate with the controller 500). Proximity sensors 410,420 are convergent infrared (IR) emitter-sensor elements, sonar sensors, ultrasonic sensors, and / or image sensors (eg, 3D depth) that signal the controller 500 when an object is within a predetermined range of the robot 100. Map image sensors) can be included.

0129In the example shown in FIG. 4A-4C, the robot 100 includes a series of sonar-type proximity sensors 410 arranged in an upward field of view around the body 124 of the base 120 (eg, at approximately equal intervals). First, second and third sonar proximity sensors 410a, 410b, 410c are located at or near the first (front) base body portion 124a, and at least one of these sonar proximity sensors is the first. It is arranged near the outermost radial end 125a of the base body portion 124a. Fourth, fifth and sixth sonar proximity sensors 410d, 410e, 410f are located at or near the second (right) base body portion 124b, and at least one of these sonar proximity sensors is the second. It is arranged near the outermost radial end 125b of the base body portion 124b. Seventh, eighth and ninth sonar proximity sensors 410g, 410h, 410i are located at or near the third (left) base body portion 124c, and at least one of these sonar proximity sensors is the third. It is arranged near the outermost radial end 125c of the base body portion 124c. This configuration provides at least three detection areas.

0130In some embodiments, a set of sonar proximity sensors 410 (eg 410a-410i) placed around the base body portion 124 is placed pointing upwards (eg approximately in the Z direction) and optionally Z. Angled outward from the axis to generate a detection curtain 412 around the robot 100. Each sonar proximity sensor 410a-410i is a cover or radiation guide that guides sonar radiation upwards, or at least not toward other parts of the robot body 110 (eg, not detecting movement of the robot body 110 itself). Can have 414. The radiation guide 414 can be in shell or half shell shape. In the illustrated example, the base body portion 124 extends laterally beyond the leg 130, and the sonar proximity sensor 410 (eg 410a-410i) is on the base body portion 124 around the leg 130 (eg, the base). Arranged (along the periphery of the body portion 124). In addition, the upward sonar proximity sensors 410 are spaced around the leg 130 to produce a continuous or nearly continuous sonar detection curtain 412.

0131The upward observation sonar proximity sensor 410 mainly provides the ability to detect objects on a horizontal surface such as a table surface. Due to their aspect ratio, these objects can be misdetected by other sensors in the sensor system, such as the laser scanner 440 or the image sensor 450, thus causing problems for the robot 100. The upward search sonar proximity sensor 410, located along the periphery of the base 120, provides a means of detecting or detecting these types of objects / obstacles. In addition, the sonar proximity sensor 410 is angled slightly outward near the maximum width point of the base periphery so that it is not blocked or blocked by the body 140 or head 160 of the robot 100. It can be arranged so that false positives of a part of the robot 100 itself do not occur. In some embodiments, some sonar proximity sensors 410 are arranged (upward and outward) so that the perimeter volume of the torso 140 is out of the field of view of the sonar proximity sensor 410, and thus are loaded. Or allow attachments, such as basket 360, to be freely accepted. The sonar proximity sensor 410 can be embedded in the base body 124 to make it visually invisible and to eliminate external protrusions that collide with or get caught in obstacles.

0132The sensor system 400 may include one or more sonar proximity sensors 410 (eg, rear proximity sensor 410j) pointing backwards (eg, in the direction opposite to the forward drive direction F) to detect obstacles during retreat. .. The rear sonar proximity sensor 410j can include a radiation guide 414 to direct its radiation to the sonar detection field of view 412. In addition, the rear sonar proximity sensor 410j can be used to determine the distance between the robot 100 and the detected object in the field of view of the rear sonar proximity sensor 410j (eg, backward alarm). In some embodiments, the rear sonar proximity sensor 410j is embedded and mounted within the base body portion 124 so as not to give any visual or functional discontinuity to the housing shape.

0133With reference to FIGS. 2 and 4B, in some embodiments, the robot 100 drives the drive wheels 210a, 210b, 210c to allow cliff detection before the drive wheels 210a, 210b, 210c encounter cliffs (eg, stairs). Includes Cliff Proximity Sensor 420 located near or around 210b, 210c. For example, the cliff proximity sensor 420 can be located at or near each or between the radial outermost edges 422 of the base body portion 124a-c. In some examples, cliff sensing is performed using infrared (IR) proximity sensing or real-range sensing so that the radiation and detection fields overlap and the detection areas are positioned at what is expected to be the floor. An infrared emitter 422 and an infrared detector 424 angled in the direction of are used. IR proximity sensing has a relatively narrow field of view, depends on the surface albedo for reliability, and can have surface-to-surface distance accuracy. As a result, a large number of individual sensors can be placed on the periphery of the robot 100 to appropriately detect cliffs from a large number of points on the robot 100. Moreover, IR proximity-based sensors generally cannot distinguish between cliffs and safety events, such as immediately after the robot 100 has overcome a threshold.

0134The cliff proximity sensor 420, for example, when the robot 100 encounters a falling edge of the floor Detect when encountering stairs. The controller 500 (which executes the control system) can cause the robot 100 to perform an action such as changing the traveling direction when an edge is detected. In some embodiments, the sensor system 400 includes one or more secondary cliff sensors (eg, other sensors configured for cliff detection and optionally other detection type sensors). Cliff detection proximity sensor 420 can be arranged to provide early detection of cliffs and provide data to distinguish between actual cliffs and safety events (such as overcoming thresholds), as well as their field of view. It can be positioned downward and outward to include a region away from the robot body 110 with at least a portion of the robot body 110. In some embodiments, the controller 500 identifies an increase in the distance across the edge of the supporting work surface (eg, the floor), the edge of the work surface and / or the increase in distance between the robot body 110 and the work surface. Execute the cliff detection routine to be detected. By performing this routine, 1) early detection of potential cliffs (which can enable faster movement speeds in unknown environments), 2) controller 500 receives cliff image information from cliff detection proximity sensor 420. Increased reliability of autonomous mobility and reduced false positives for cliffs to know if the cliff event is really unsafe or crossable safely (eg overcoming thresholds). (For example, by using edge detection for a large number of individual IR proximity sensors with a narrow field of view) is possible. Additional sensors arranged as "falling wheel" sensors can also be used for redundancy and to detect situations where the distance detection camera cannot reliably detect a given type of cliff.

0135The threshold and step detection allows the robot 100 to make an effective plan to cross a overcomeable threshold or avoid a step that is too high. This can be the same for chaotic objects on the work surface that the robot 100 can or cannot safely traverse. By knowing the height of obstacles or thresholds that the robot 100 determines can be overcome, in order to maximize the smoothness of the overcoming and minimize the instability due to sudden acceleration. The 100 will be able to decelerate appropriately if necessary to allow for a smooth ride. In some embodiments, threshold and step detection is performed based on the height of the object on the working surface, along with shape recognition (eg, distinguishing between thresholds or electrical cables and small chunks such as socks). The threshold can be recognized by edge detection. The controller 500 receives image data from the cliff detection proximity sensor 420 (or another image sensor on the robot 100), executes an edge detection routine, and generates a drive command based on the result of the edge detection routine. Controller 500 can also use pattern recognition to identify objects. Threshold detection allows Robot 100 to turn around the sill to maximize its ability to overcome the sill.

0136Proximity sensors 410,420 may function independently, but as an alternative, they may function in combination with one or more contact sensors 430 (eg, collision switches) for redundancy. For example, one or more contact or collision sensors 430 on the robot body 110 detect when the robot 100 physically collides with an obstacle. Such sensors can utilize physical properties such as capacitance or physical displacement within the robot 100 to detect when the robot 100 collides with an obstacle. In some embodiments, each base body portion 124a, 124b, 124c of the base portion 120 detects the movement of the corresponding base body portion 124a, 124b, 124c with respect to the base chassis 122, the associated contact sensor 430 (eg, capacitance). It has a sensor, a reed switch, etc. (see, for example, Figure 4A). For example, each base body portion 124a-c can move radially with respect to the Z axis of the base chassis 122, thus providing collision detection in three directions.

0137Referring again to FIG. 1-4C, in some embodiments, the sensor system 400 includes a laser scanner 440 mounted on the front of the robot body 110 and communicating with the controller 500. In the illustrated example, the laser scanner 440 (eg, having a field of view along the forward drive direction F) is mounted forward above or above the first base body portion 124a of the base portion 120 (eg, the drive direction of the robot). To get the maximum imaging range along F). Furthermore, installing the laser scanner at or near the front tip of the triangular base 120 means that the outer angle (eg 300 °) of the base of the robot is larger than the field of view 442 (eg ~ 285 °) of the laser scanner 440. Therefore, it means that the base portion 120 does not block or block the detection field of view 442 of the laser scanner 440. The laser scanner 440 can be embedded and mounted in the base body 124 as much as possible so as not to block the field of view, and the portion where the laser scanner protrudes from the base body 124 can be minimized (for example, aesthetic appearance and obstacles). To minimize conflicts with).

0138The laser scanner 440 scans around the robot 100, and the controller 500 uses the signal received from the laser scanner 440 to generate an environment map or object map of the scan area. Controller 500 can use the object map for navigation, obstacle detection and obstacle avoidance. In addition, the controller 500 can use sensor inputs from other sensors in the sensor system 400 for object map generation and / or navigation.

0139In some examples, the laser scanner 440 is a scanning lidar, which assigns an area in one dimension, as a "main" scanning line, to a fast-manipulating laser and to each pixel generated in the scanning line. A flight time image sensor that uses phase difference or similar techniques can be used to (return a two-dimensional depth line in the scanning plane). To generate a 3D map, lidar can perform an "auxiliary" scan in a second direction (eg, by vibrating the scanner vertically). This mechanical scanning technique, if not complemented, is intended to provide, for example, a "flash" LIDAR / LADAR and a "Swiss ranger" focal plane image sensor sensor, a depth at each pixel or a series of depths at each pixel. It can be complemented by techniques that allow the calculation of flight times for pixels in a full two-dimensional matrix using semiconductor stacks.

0140The sensor system 400 can include one or more 3D image sensors 450 that communicate with the controller 500. If the 3D image sensor 450 has a limited field of view, the controller 500 or sensor system 400 scans the 3D image sensor 450a side-to-side to perform robust obstacle detection / obstacle avoidance (ODOA). Driven by an equation to generate a relatively wide field of view. Referring to FIG. 1-3B, in some embodiments, the robot 100 includes a scanning 3D image sensor 450a mounted on the front of the robot body 110 and having a field of view along the forward drive direction F (eg,). , To obtain the maximum imaging range along the robot drive direction F). The scanning 3D image sensor 450a can be used mainly for ODOA. In the illustrated example, the scanning 3D image sensor 450a is mounted under the step 142 of the body 140 or on the bottom surface 144, eg, to prevent the user from coming into contact with the scanning 3D image sensor 450a, FIG. It is embedded in the torso 140 (eg, coplanar with or along the bottom surface 144) as shown in. The scanning 3D image sensor 450 leaves the robot body 110 approximately downward to obtain a downward field of view 452 (eg, blocked by the base 120 or other parts of the robot body 110) in front of the robot 110 for ODOA. It can be configured to aim in a direction. By installing the scanning 3D image sensor 450a on or near the front edge of the body 140, the field of view (for example, ~ 285 °) of the 3D image sensor 450 can be changed to the external surface angle of the body 140 with respect to the 3D image sensor 450. It can be smaller than (eg, 300 °) so that the body 140 does not block or block the detection field of view 452 of the scanning 3D image sensor 450a. In addition, the scanning 3D image sensor 450a (and related actuators) can be embedded and mounted within the torso 140 as much as possible without obstructing the field of view (eg, aesthetically pleasing and minimal collision with obstacles). To make it ). The distracting scanning motion of the scanning 3D image sensor 450a is invisible to the user and does not interfere with the interaction. Unlike the protruding sensor or functional part, the embedded scanning 3D image sensor sensor 450a does not actually have a moving part that protrudes beyond the cover of the upper body 140, so it interacts unintentionally with the environment, especially when moving or operating. Does not cause (getting caught in people or obstacles).

0141In some embodiments, the sensor system 400 includes an additional 3D image sensor 450 located on the base body portion 124, neck portion 150 and / or head portion 160. In the example shown in FIG. 1, the robot 100 includes a 3D image sensor 450 on a base body portion 124, a body portion 140, and a head portion 160. In the example shown in FIG. 3A, the robot 100 includes a 3D image sensor 450 on the legs 1130, torso 140 and neck 150. Other configurations are possible. One 3D image sensor (eg, above the neck 150 and above the head 160) can be used for human recognition, gesture recognition and / or video conferencing, but another 3D image sensor 450 (eg, base). 120 top and / or leg 130 top) can be used for navigation and / or obstacle detection and obstacle avoidance.

0142A forward-looking 3D image sensor 450 located at the neck and / or head can be used to recognize people, faces and / or gestures of people around the robot 100. For example, using the signal input from the 3D image sensor 450 of the head 160, the controller 500 generates a 3D map of the observed / captured user's face and the generated 3D map of a known person. Compare with the 3D image of the face to determine a match with one of the known 3D facial images. Face recognition can be used to authenticate the user as a legitimate user of Robot 100. In addition, one or more of the 3D image sensors 450 can be used to determine a person's gestures observed by the robot 100, and if necessary, the determined gestures (eg pointing, waving and / Or it can react based on hand signals). For example, the controller 500 can generate a drive command in response to a recognized pointing in a particular direction.

0143The 3D image sensor 450 can generate the following types of data: (i) depth maps, (ii) reflectance-based intensity images and / or (iii) standard intensity images. The 3D image sensor 450 can obtain such data by image pattern matching, measurement of flight time and / or phase delay shift of light emitted from a light source and reflected from a target.

0144In some embodiments, the inference or control software that can be executed on the processor (eg, robot controller 500) uses a combination of algorithms that are executed using the various data types generated by the sensor system 400. The inference software processes the data collected from the sensor system 400 and outputs data that makes a navigation decision that allows the robot 100 to move, for example, without colliding with an obstacle. By accumulating image data around the robot over time, the inference software can then apply effective methods to selected sections of the detected image to improve the depth measurements of the 3D image sensor 450. This can include the use of appropriate temporal and spatial averaging techniques.

0145The reliability of robot collision avoidance movement is (i) reliability level constructed by advanced inference over time, (ii) three main types of analytical data: (a) depth image, (b) active illumination image. And (c) ambient illumination image: can be based on a depth perception sensor, which accumulates. Algorithms for recognizing different types of data can be run on each of the images obtained by the Depth Perceptual Image Sensor 450. Aggregated data can improve reliability levels compared to systems that use only one of these types of data.

0146The 3D image sensor 450 can obtain depth and luminance data from a scene (eg, a sensor view portion of a room or work area) that includes one or more objects around the robot 100. The controller 500 can be configured to determine the occupancy data of an object based on the light reflected and captured from the scene. In addition, the controller 500 issues drive commands to the drive system 200 based on at least a portion of the occupied data to avoid obstacles (ie, objects in the scene) in some examples. The 3D image sensor 450 can repeatedly capture the depth image scene in order for the controller 500 to make real-time decisions to navigate the robot 100 so that it does not collide with objects in the scene. For example, the speed or frequency at which the depth image data is obtained by the 3D image sensor 450 can be controlled by the shutter speed of the 3D image sensor 450. In addition, the controller 500 can receive event triggers from (eg, other sensor components such as proximity sensors 410,420 of the sensor system 400) to notify the controller 500 of nearby objects or obstacles. In response to the event trigger, the controller 500 can increase the frequency at which the depth image is captured by the 3D image sensor 450 and the occupancy information is obtained.

0147In some embodiments, the robot includes a sonar scanner 460 for audiovisual of the area surrounding the robot 100. In the examples shown in FIGS. 1 and 2, the sonar scanner 460 is arranged in front of the main body portion 124 of the base portion 120.

0148Referring to FIG. 1-3B, in some embodiments, the robot 100 uses a laser scanner or laser rangefinder 440 for redundant sensing and a backward sonar proximity sensor 410j for safety, both sensors. Both are oriented parallel to the earth G. The robot 100 can include first and second 3D image sensors 450a, 450b (depth cameras) to provide robust sensing of the environment surrounding the robot 100. The first 3D image sensor 450a is mounted on the body 140 at a fixed angle downward toward the ground G. By angling the first 3D image sensor 450a downward, the robot 100 has a strong sensor range in the area immediately in front of the robot 100 or adjacent to the robot 100, which is a small distance forward of the robot 100. Corresponds to running. The backward sensor 410j provides object detection while the robot is retreating. If the robot 100 is constantly retreating, the robot 100 may include a third rearward downward 3D image sensor 450 to provide a strong sensor range in the area immediately behind or adjacent to the robot 100.

0149A second 3D image sensor 450b is mounted on the head 160, which can be panned and tilted at the neck 150. The second 3D image sensor 450b can be useful for remote driving because the human operator can see where the robot is heading. The neck 150 allows the operator to tilt and / or pan the second 3D image sensor 450b to see near and distant objects. Panning the second 3D image sensor 450b increases the associated horizontal field of view. While traveling at high speed, the robot 100 tilts the second 3D image sensor 450b slightly downward to increase the overall or composite field of view of the 3D image sensors 450a, 450b, giving sufficient time to avoid obstacles. It can be given to Robot 100 (generally, the higher the speed, the shorter the reaction time to obstacles). At low speeds, the robot 100 can tilt the second 3D image sensor 450b upwards or approximately parallel to the ground G to track the person the robot 100 must follow. Further, while driving at a relatively low speed, the robot 100 can pan the second 3D image sensor 450b to increase its field of view around the robot 100. The first 3D image sensor 450a can remain fixed (eg, do not move relative to the base 120) to extend its perceptual range while driving the robot. In addition and / or instead, the first 3D image sensor 450a can scan at low speed while driving the robot to detect potential obstacles around the robot. In some examples, the height of the first 3D image sensor 450a can be adjusted upwards, for example using a Z lift, to optimize the field of view of the first 3D sensor 450a. ..

0150In some embodiments, at least one of the 3D image sensors 450 is placed on the robot 100 at a height greater than 1 or 2 feet above the ground (or about 1 or 2 feet above the ground) and on the floor surface. A volumetric point cloud imager (eg, speckle or flight time camera) directed to acquire a point cloud (point cloud) from a spatial volume containing the robot in the direction of movement of the robot (via an omnidirectional drive system 200). Can be. In the example shown in FIGS. 1 and 3, the first 3D image sensor 450a captures an image of the volume containing the floor (eg, volumetric point cloud) while driving (eg, for obstacle detection and obstacle avoidance). It can be placed on the base 120 at a height greater than 1 or 2 feet above the ground for acquisition. The second 3D image sensor 450b is mounted on the head 160 (eg, at a height greater than about 3 or 4 feet above the ground) to obtain a skeletal recognition and definition point cloud from the spatial volume adjacent to the robot 100. .. Controller 500 runs skeletal / display recognition software to analyze the acquired volumetric point cloud data.

0151With reference to Figure 3A-4A, the sensor system 400 is the overall center of gravity CG of the robot 100._RAn inertial measurement unit (IMU) 470 that communicates with controller 500 can be included to measure and monitor the moment of inertia of the robot 100 with respect to.

0152The controller 500 monitors the deviation from the threshold signal corresponding to the normal unweighted operation fed back from the IMU470. For example, if the robot begins to move out of its upright position, it may have been caught in a string, otherwise disturbed, or someone has suddenly added heavy loads. In these cases, it is necessary to take emergency actions (such as, but not limited to, avoidance actions, recalibration and / or acoustic / visual alarms) to ensure the safe operation of the robot 100.

0153Because the robot 100 operates in a human environment, it interacts with humans and operates in a space designed for humans (regardless of robot constraints). Robot 100 can limit its drive speed and acceleration in crowded or constrained environments such as cocktail parties and busy hospitals, or in highly dynamic environments. However, the Robot 100 may encounter situations where it is safe even at relatively high driving speeds, such as in a long empty corridor, but nevertheless, suddenly, such as when something crosses the robot's path. Can be decelerated to.

0154When accelerating from a stop, the controller 500 determines the overall center of gravity CG of the robot 100 to prevent the robot from tipping over._RThe moment of inertia from can be taken into account. The controller 500 can use a model of its attitude (including the current moment of inertia). When supporting the payload, the controller 500 has a total center of gravity CG_RIt is possible to measure the load affecting the robot and monitor the movement of the robot moment of inertia. For example, the body 140 and / or the neck 150 can include a strain gauge to measure strain. If this is not possible, the controller 500 will supply a test torque command to the drive wheels 210 and use the IMU470 to measure the actual linear and angular acceleration of the robot to experimentally determine the safety limits. Can be done.

0155During the sudden deceleration, the commanded load on the second and third drive wheels 210b, 210c is reduced, but the first drive wheel 210a (front wheel) slips in the forward drive direction to support the robot 100. When the loads of the second and third drive wheels 210b, 210c (rear wheels) are asymmetric, the robot 100 "yaws" and reduces dynamic stability. An IMU470 (eg, a gyro) can be used to detect this yaw motion and instruct the second and third drive wheels 210b, 210c to reconfigure the robot 10.

0156Referring to FIG. 5, in some embodiments, controller 500 executes control system 510, which includes behavioral system 510a and control arbitration system 510b. The control arbitration system 510b allows the application 520 to be dynamically added to and removed from the control system 510, facilitating each application 520 to control the robot 100 without having to know about the other application 520. In other words, the control arbitration system 510b provides a simple priority control mechanism between application 520 and robot 100 resource 530. Resource 530 may include a drive system 200, a sensor system 400, and / or any payload or controllable device that communicates with the controller 500.

0157The application 520 can control the robot 100 at the same time it is stored in the robot's memory or communicates with the robot to execute on the robot (eg, a processor). Application 520 can access each action 512 of action system 510a. The independently distributed application 520s are dynamically combined at run time and share the robot resources 530 of the robot 100 (eg, drive system 200, arms, head, etc.). A low level policy is enforced at runtime for dynamic sharing of robot resource 530 between applications 520. This policy determines which application has control of robot resource 530 required by that application (eg, generates a priority hierarchy within application 520). Application 520 can be started and stopped dynamically and can run completely independently of each other. The control system 510 can also allow complex actions 512 that can be combined to help each other.

0158The control arbitration system 510b includes one or more resource controllers 540, a robot manager 550, and one or more control arbiters 560. These components do not have to be in a common processor or computer, nor do they need to start in any particular order. The resource controller 540 provides an interface to the control arbitration system 510 for application 520. An instance of this component exists for every application. Resource controller 540 summarizes and encapsulates the complexity of authentication, distributed resource control arbiters, command buffering, and more. The robot manager 550 adjusts the prioritization of application 520 by controlling which application 520 has exclusive control of any robot resource 530 at any particular time. Since this is the central coordinator of information, there is only one instance of Robot Manager 550 per robot. The robot manager 550 executes a priority policy with a linear priority of the resource controller 540 and keeps track of the resource control arbiter 560 that provides hardware control. The control arbiter 560 receives commands from all applications 520, issues a single command based on the properties of the application, and publishes it to the associated resource 530. The control arbiter 560 also receives the state fed back from its associated resource 530 and returns it to application 520. Robot resource 530 can be a network of functional modules (eg, actuators, drive systems, and groups thereof) with one or more hardware controllers. The commands of the controller arbiter 560 are specific to resource 530 and perform unique actions.

0159The dynamic model 570, which can be run on the controller 500, can be configured to calculate the center of gravity (CG), the moment of inertia, and the inertial cross product of the various parts of the robot 100 for access to the current robot state. it can. The dynamic model 570 can further model the shape, weight and / or moment of inertia of these parts. In some examples, the dynamic model 570 communicates with the IMU470 and is located on the robot 100 with components (eg, accelerometer and / or gyro) that communicate with the controller 500 to calculate the various centroids of the robot 100. connect. The dynamic model 570 can be used by the controller 500, along with other processors 520 or behavior 512, to determine the motion envelope of the robot 100 and its components.

0160Each application 520 has an action selection engine 580 and a resource controller 540, one or more actions 512 are connected to the action selection engine 580, and one or more action models 590 are connected to the action selection engine 580. The action system 510a can determine the robot action by providing a predictive model and the action 512 working together to evaluate the possible outcomes of the robot action. In some examples, Action 512 is a plug-in component that provides a hierarchical, stateful evaluation function that provides perceptual feedback from multiple sources with apriori limits and information as evaluation feedback on the robot's permissible movements. Combine to. Actions 512 (eg, existing inside or outside application 20) can be plugged into application 520, so they do not need to modify application 520 and do not require other parts of control system 510. Can be removed and added to. Each action 512 is a standalone policy. To make action 512 more powerful, the output of multiple actions 512 can be added together to the input of another action to obtain complex combinatorial functions. Action 512 aims to realize a manageable portion of the entire cognitive range of Robot 100.

0161The motion selection engine 580 is a coordinating element of the control system 510 and, given the inputs of all actions 512, executes a fast optimal motion selection cycle (prediction / correction cycle) to search for the best motion. The motion selection engine 580 has three phases: nomination, motion selection search and completion. In the nomination phase, each action 512 is informed that the action selection cycle has started and is given limits on cycle start time, current state and robot actuator space. Based on internal policies or external inputs, each action 512 determines whether it wants to participate in the controller's action selection cycle. During this phase, a list of active behavioral primitives is generated, the input of which influences the choice of commands executed by Robot 100.

0162In the motion selection search phase, the motion selection engine 580 produces executable results from the available motion space (also called the motion space). The action selection engine 580 provides a pool containing executable commands (within limits) and the corresponding results as a result of simulation at different time steps in the future planned period of action for each command. Use the motion model 590. The application selection engine 580 calculates a favorable result based on the result evaluation of the action 512, sends the corresponding command to the control arbitration system 510B, and notifies the operation model 590 of the selected command as feedback.

0163In the completion phase, the command corresponding to the highest score cooperation result is combined as a comprehensive command, and this command is presented to the resource controller 540 for execution on the robot resource 530. The best results will be provided as feedback to Active Action 512 for use in future evaluation cycles.

0164The received sensor signal from the sensor system 400 can execute an action by interacting with one or more actions 512. For example, using the control system 510, the controller 500 selects an action (or move command) for each robot component (eg, a motor or actuator) from the corresponding operating space (eg, a group of possible actions or moves for that component). Efficiently achieve coordinated movement of each robot component to avoid collisions with itself and with any object around the robot 100. Controller 500 can issue tuned commands via a robot network, eg, the Ether IO network, as described in US Patent Application No. 61 / 305,069 filed February 16, 2010. .. The prior contents of this US application are incorporated herein by reference.

0165FIG. 6A illustrates a schematic of an exemplary robot system 600 with one or more telepresence robots 100 communicating with bridge 602, where bridge 602 is a local endpoint server 604a and a remote endpoint server 604b (eg, a cloud computing server). Communicate with 720 (Fig. 7)). The local robot endpoint server 604a communicates with the local technician computing device 606 and the remote endpoint server 604b communicates with the remote operator computing device 608.

0166With reference to FIGS. 2 and 4C, in some embodiments, the robot 100 includes a large number of antennas. In the illustrated example, the robot 100 includes a first antenna 490a and a second antenna 409b located on the base 120 (although these antennas are the robot 100's, eg, legs 130, torso 140, It can also be placed in other parts such as the neck part 150 and / or the head part 160). The use of multiple antennas results in the reception and transmission of robust signals. The use of multiple antennas provides robot 100 with multiple inputs and multiple outputs (MIMO), which uses multiple antennas for the transmitter and / or receiver to improve communication performance. MIMO results in a significant increase in data throughput and link range without the need for additional bandwidth or transmit power. This achieves higher spectral efficiency (more bits per second and per hertz of bandwidth) and link reliability or diversity (reduced fading). Due to these characteristics, MIMO is IEEE 802.11n (Wifi®), 4G, 3GPP-LTE (Long). It is an important part of modern wireless communication standards such as Term Evolution), WiMAX® and HSPA +. In addition, Robot 100 can act as a Wi-Fi bridge, hub or hotspot for other electronic devices around it. The mobility of the robot 100 and the use of MIMO allow the robot to act as a relatively reliable WI-Fi bridge 602.

0167Referring to FIGS. 6A and 6B, the teleoperation software application 601 is at least one of the robot controller 500, the local robot endpoint server 604a, the remote endpoint server 604b, the local technician computing device 606 and the remote operator computing device 608. Run on. The teleoperation software application 601 allows one or more users to interact with the robot 100 (eg, drive the robot 100) through the telepresence function of the robot 100 and / or other person or object near the robot 100. Allows you to interact remotely with.

0168FIG. 6C is a teleoperation software application that can be displayed on a display such as the touch screen 312 and / or remote operator computing device 608 of the web pad 310 to control the navigation, telepresence and / or other features of the robot 100. A schematic diagram of an exemplary user interface 605 of 601 is presented. User interface 605 includes a remote video feed window 610 that displays a remote view 612, such as the video feed of patient 614. The video feed can be generated by one of the cameras 320,450 on Robot 100. User interface 605 may display a floor plan map window 620 with a map 622 of the local area in which the robot 100 is operating. In the illustrated example, the map 622 displayed in the plan map window 620 is a two-dimensional top-down map 622a (FIG. 6D), but other types of maps are possible. The user interface 605 can also include a local video window 630 displaying a local view 632, such as a user's video feed (eg, far away from the robot 100). The video feed displayed in the local video window 630 can be sent to Robot 100 and displayed to Patient 614 using a display device such as Webpad 310 on Robot 100.

0169The dashboard 640 may provide information about the direction of the robot 100, robot battery level indications, radio data signal strength indications and / or network quality indications. The direction of the robot 100 can be indicated by an icon 642 indicating the direction of the head 160 of the robot 100 with respect to the body 140 or the base 120. Such instructions help the user point the robot 100 in the direction of looking at something of interest. The range of motion of the robot head unit 160 can be limited. Therefore, in some embodiments, the rotation position of the head portion 160 can be indicated and the rotation range of the head portion 160 can be displayed.

0170Media control 647 interacts with patient 614 using a variety of media, allowing the user to acquire and store media information that records the interaction between the user and patient 614. Media control 647 allows the user to play audio and / or video clips that can be used, for example, to teach patient 614 about medical conditions and medical procedures. Still images can also be acquired using the cameras 320,450 of Robot 100 to record various states. In addition, the robot 100 acquires audio (eg, using a microphone 330) or video (eg, using a camera 320) that records the interaction between patient 614 and the user, and optionally obtains the acquired audio / video. The audio / video stored in the memory of the controller 500 and / or acquired can be transmitted to a remote device or a cloud service.

0171In some embodiments, Media Control 647 allows the user to manage temporary connectivity issues. For example, video recording can be initiated at the time of an unexpected disconnection of a session. Robot 100 can continue to record video and store it in local memory, such as the memory of controller 500. In the event of an unexpected disconnection, the robot can display a message such as "End session-Continue video recording ...". Below the button, you can see the "Stop Recording" caption. A nurse near the robot can touch the "Stop Recording" button (eg on the touch screen 312) to end local recording. Otherwise, the recording will continue for the specified time. If the same user re-logins to Robot 100 during a specified time interval, the record button on the remote station can indicate that recording is in progress. Once the robot's local recording is complete, the video file can begin to be transmitted to a remote station or other location accessible to the disconnected user. Therefore, the user can know what happened during the interruption of the session.

0172In the example shown in Figure 6C, the remote video feed window 610 occupies a relatively large portion of the display area. The user interface 605 can have a remote video feed window 610 with a 640 x 480 pixel resolution, a local video window 630 with a 320 x 240 pixel resolution, and a plan map window 620 with a 530 x 200 pixel resolution. Therefore, this view may be most appropriate when the user communicates with patient 614 and / or when the user manually drives the robot 100. The layout of the default user interface window 605a shown in FIG. 6C allows the user to replace the contents of the plan map window 620 with the contents of the remote video feed window 610. You can swap views, for example by double-clicking on the map window 620. Both windows can be replaced by double-clicking on the remote video feed window 610.

0173The user can be presented with a different screen layout suitable for the work performed by the user. In the example shown in FIG. 6D, the size of the plan map window 620 has been increased in anticipation of moving the robot 10 from one position to another using semi-autonomous navigation. For example, the user interface 605,605a shown in FIG. 6C can be the default state for patient interaction, and the user interface 605,605b shown in FIG. 6D can be the default state for robot navigation.

0174The user interface map view toggle button 645 allows the user to call another user interface 605b, which includes a relatively large map window 620. For example, the user interface 605b shown in FIG. 6D can be primarily used to manually drive the robot 100 to a desired destination or to navigate autonomously. Clicking the map view toggle button 645 returns the user to the default user interface 605a again, which can be used for active consultations. Thus, the user can highlight or reduce (eg maximize or minimize) the floor plan map window 620 as needed. Some of the windows shown in the alternative user interface 605b, such as the remote video feed window 610 and the local video feed window 630, are also displayed. In some examples, the floor plan map window 620 is displayed at 880 x 700 pixel resolution, the remote video feed window 610 is displayed at 320 x 240 pixel resolution, and the local video feed window 630 is displayed at 160 x 120 pixel resolution. obtain.

0175With reference to FIG. 6D, the floor plan map window 620 can provide a robot position icon 650 indicating the position of the robot 100 in the local environment. The user can click or touch one point on the displayed map 622 to allow the robot to navigate semi-autonomously or autonomously to the selected point. In some examples, the user can zoom in and out by hovering the cursor over the floor plan map window 620 and using the mouse wheel or by touching a gesture when displayed on the touch screen 312. Can be done.

0176Simultaneous localization and mapping: Simultaneous localization and mapping: SLAM) technology can use laser distance sensors, odometers, acoustic rangefinders, or all of them to map the local environment and place the Robot 100 on top of that map. Images recorded by Robot 100 while traversing the environment (eg with camera 320 or 3D image sensor 450) can be stored in an internal database (eg controller 500) and / or a remote database (eg cloud device). When Robot 100 reacquires the current image in the database, the algorithm resets the robot's current position to the position recorded when the landmark was first entered into the database. This method helps counter the inherent drift of wheel encoder odometry. The system can also utilize RFID chip and / or triangulation of wireless access points. In addition, the name or identification number of a particular room can be associated with its location on the map. Image data can accumulate over time. Robot 100 can utilize remote data storage and / or remote processing for image data storage and processing, respectively, as a cost or space savings. For example, an RFID reader can detect RFID chips associated with coordinates on a floor plan map to identify the current position of the robot. An "RFID chip" includes an RFID device or RFID tag, as is known to those skilled in the art. RFID chips can be embodied as passive, active, or battery-assisted passive (BAP) RFID chips.

0177FIG. 7 presents a schematic of an exemplary robot system architecture 700, which robotic systems include a robot 100 (or a portion thereof, eg, a controller 500 or a drive system 200), a computing device 310 (eg, a head 160). It can include removable or fixedly mounted Cloud 720 (ie, cloud computing devices) and Portal 730. The computing device 310 can run one or more robot applications 710, which are software applications for security, medicine compliance, telepresence, behavioral guidance, social networking, active alarms, home management, etc. Can contain (stored in memory and executed by the processor). The computing device 310 provides communication functions (eg, secure wireless connection and / or mobile communication), sophisticated application development tools, voice recognition, and personal or object recognition functions. In some examples, the computing device 310 is an interaction / CMOS featured operating system, such as Android® provided by Google, iOS® provided by Apple, or other smartphones. An operating system or a dedicated robot operating system such as RSS A2® can be used.

0178Cloud 720 provides cloud computing and / or cloud storage capabilities. Cloud computing can provide internet-based computing, so shared servers can provide resources, software and data to computing and other devices on demand. For example, the cloud 720 can be a cloud computing service that includes at least one server computing device, which server computing device transfers the service summary layer and hypertext via a server virtual machine instantiated on it. Can include a protocol wrapper. This server computing device can be configured to parse HTTP requests and send HTTP responses. Cloud computing can be a technology for maintaining data and applications using the Internet and central remote servers. Cloud computing allows users to access and use applications to access their personal files on computers with Internet access without having to install them. Cloud computing enables relatively more efficient computing by concentrating storage, memory, processing and bandwidth. The Cloud 720 can provide scalable, on-demand computing power, storage and bandwidth while reducing robot hardware requirements (eg by freeing up CPU and memory usage). The robot's connectivity to the cloud 720 enables automatic data collection of robot movement and usage history without the need to return the robot 100 to the base station. In addition, continuous data collection over time can result in large amounts of data, from which data for marketing, product development and support can be unearthed.

0179Cloud storage 722 can be a model for networked computer data storage, where data is typically stored on a number of virtual servers hosted by third parties. By providing communication between Robot 100 and Cloud 720, the information collected by Robot 100 can be securely viewed by legitimate users via the web information portal.

0180Portal 730 can be a web-based user portal that collects and / or provides information such as personal information, home status information and robot status information. The information can be integrated with third party information to provide additional functionality and resources to the user and / or the robot 100. Robot System Architecture 700 facilitates aggressive data collection. For example, application 710 running on computing device 310 may collect data and reports (using sensor system 400) about actions performed by robot 100 and / or individuals or the environment observed by robot 100. it can.

0181"Dense data" vs. "sparse data" and "dense features" vs. "sparse features" are referred to herein for spatial datasets. Without limiting or narrowing how one of ordinary skill in the art interprets the meaning of such terms, "dense" vs. "sparse" generally means that the number of data points per spatial representation is large vs. small. , Specifically, it can mean the following.

0182(i) In relation to 2D image data or 3D "images" including 2D data and distance, "Dense" image data is almost completely pixel-occupied image data or loss from the original image capture and / Or contains image data that can be rasterized into pixels with little artifacts (including raw images that are compressed with little or no loss), but "sparse" images are quantized, sampled, and lossy compressed. , Vectorized, segmented (eg to subpixels, nodes, edges, surfaces, feature points, boxels), or re-images with substantially reduced fidelity from the original captured image, or images An image in which rasterized pixels must be interpolated for representation.

0183(ii) With respect to 2D or 3D features, "dented features" concentrate almost unconstrained up to the resolution of the detection approach, and include everything that can be detected and recorded, and / or many more than sub-images. It can be the feature amount recognized by the detector recognized to collect the feature amount (HOG, wavelet) of. A "sparse feature" can be deliberately constrained by the number of feature inputs, lateral prohibitions, and / or feature selections, and also a limited number of independent points in the image (Harris corners, edges, Shi- It can be recognized by a recognized detector to identify Tomasi).

0184For the three-dimensional environment structure, the robot 100 acquires an image of the scene 10 around the robot 100, for example, a dense image 701, while maneuvering in the work surface 5. In some embodiments, the robot 100 uses a camera 320 and / or an image sensor 450 (eg, a volumetric point cloud imaging device) to obtain a dense image 701. The controller 500, which communicates with the camera 320 and / or the image sensor 450, associates information with the dense image 701 (eg, marks up or tags the dense image 701 with data), such as an accelerometer data trace, odometry data, and / or a sensor. Associate other data from system 400 with the time stamp. In some examples, Robot 100 captures a streaming sequence of dense image 701, marks the dense image sequence with markup data, and provides the marked up dense image sequence. The cloud service 720 processes the received image data 701 and returns the processed data set to the robot controller 500, which sends a drive command to the drive system 200 based on the processed data set received for maneuvering in scene 10. Can occur.

0185The cloud service 720 performs one of various offline methods to process the stored image dataset 703 into a dense 3D map or model 705 of scene 10, and then processes this dense 3D map or model 705 into 2D. Simplify to height map 707. This map 707 can be a 2D map with height data at each point (eg, similar to a 2D topographic map). In some examples, the 2D height map 707 is a topographic map with X and Y coordinates along with Z data. Each X, Y coordinate can have one or more Z points (ie height data). Unlike the dense 3D map, which can have many Z points (eg hundreds or thousands of Z points) for each X, Y coordinate, the 2D height map 707 has a threshold number for each X, Y coordinate. It can have fewer (eg 10) Z points than (eg 2-20). The 2D height map 707 derived from the 3D map of the indoor table is the first Z point on the bottom surface of the table top and the second Z on the top surface of the table top with respect to the X and Y coordinates of the image along the table. Can indicate points. With this information, the robot 100 can determine whether it can pass under the table top. By reducing the Z points from a dense dataset with a continuous range of Z points for each X, Y coordinate to a sparse dataset with a selection of Z points representing the detected object 12, the robot 100 has a cloud service 720. Receives a 2D height map 707 with a size significantly smaller than the 3D map used by. This allows the robot 100 to store the 2D height map 707 in local memory of a practical and cost-effective size compared to the scalable memory space available for the cloud service 720. Robot 100 receives a 2D height map from Cloud 720, which provides the Robot 100 and associated Controller 500 with navigation data for future activities in Scene 10.

0186Additional methods and features of 3D map data compression are described in the paper "Multi-Level Surface Maps for Outdoor" by P. Triebel, P. Pfaff and W. Burgard presented at the IEEE / RSJ International Conference on Intelligent Robots and Systems, 2006. It is disclosed in "Terrain Mapping and Loop Closing", and the entire contents of this paper are incorporated herein by reference.

0187Cloud 720 provides robot 100 with on-demand scaling of resources that are otherwise not practical or cost-effective on robot 100 (eg, computation, processing, memory, etc.). For example, the Cloud 720 is scalable, scaled up to a first size for storing and / or processing a relatively large amount of data that is used for a short period of time and then discarded, and then scaled down to a second size. Cloud storage 722 can be provided. The Cloud 720 provides computer processing power to perform relatively complex calculations or powerful algorithms that might otherwise be impossible on a robot. By replacing the computer processing power and memory with the scalable cloud 720, the robot 100 can use the controller 500, which has a relatively low processing power and memory, thus achieving cost-effective problem solving. In addition, Robot 100 transfers non-real-time or non-time-dependent tasks to Cloud 720 for processing and later reading to perform real-time tasks such as obstacle avoidance (on controller 500 or webpad 310). Can be done.

0188The Cloud 720 can perform one or more filters (eg, bundle adjustment, RANSAC, expected value maximization, SAM or other 3D structure estimation algorithm) to process the stored image dataset 703 into a 3D representation. Once processed and the dense 3D map 705 is generated or updated, the image dataset 703 can be discarded from the cloud 720, freeing resources and scaling the cloud 720 accordingly. As a result, due to the use of cloud-based resources, Robot 100 does not require onboard storage, nor does it require a processor to store and process image dataset 703. Cloud 720 can send processed navigation data 701 or map 707 (eg, a compressed 2D height map) back to Robot 100, which will use this data or map for relatively easy positioning and navigation. Can be used for processes.

0189Additional methods and features of 3D reconstruction are disclosed in the paper "3D Models from Extended Uncalibrated Video Sequences" by J. Repko and M. Pollefeys, presented at the Fifth International Conference on three-dimensional Digital Imaging and Modeling, 2005. The entire contents of this paper are incorporated herein by reference.

0190Referring to FIGS. 8A and 8B, in some environments, the robot 100 receives an occupation map 800 of the object 12 in the scene 10 and / or the working surface 5, or the robot controller 500 receives an image sensor 450 (eg, a second). Occupation map 800 is generated (and updated) based on image data and / or image depth data received from the 3D image sensor 450b) over time. SLAM simultaneously tracks its current location to create an occupancy map 800 in an unknown environment or scene 10 (without prior knowledge), or to create an occupancy map 800 in a known environment. A technique that can be used in Robot 100 to update (with a priori knowledge from a given map).

0191Controller 500 sends the occupancy map 800 to the telepresence software application 601 to display map 622 on user interface 605. The user interface map 622 can be partially or wholly derived from the occupancy map 800. Further referring to FIG. 7, the telepresence software application 601 can receive periodic updates of the occupancy map 800 via the cloud service 720. For example, the cloud service 720 provides the telepresence software application 601 with a scene 10 dense 3D map or model 705 and / or a simplified 2D height map 707 for the robot 100 to generate a user interface map 622. Can be done. In an additional example, the cloud service 720 provides the telepresence software application 601 with a user interface map 622 based on a dense 3D map or model 705 and / or a 2D height map 707.

0192With reference to FIGS. 8A and 8B again, Map 800 can be used to determine location within environment 10 and draw the environment for planning and navigation. Map 800 assists in assessing the current position by recording information obtained from the form of perception and comparing it with the current set of perceptions. The advantage of Map 800 to assist in the evaluation of position currently increases as the accuracy and quality of perception diminishes. The map 800 generally represents the state at the time the map 800 was provided or generated. This does not necessarily match the state at the time Map 800 was used. Other positioning techniques include implementation using a monocular visual SLAM (MonoSLAM) and an extended Kalman filter (EKF) for MonoSLAM solutions.

0193Controller 500 can perform scale-invariant feature transformation (SIFT) to detect and describe local features of the captured image. For any object 12 in the image, the point of interest on the object 12 can be extracted to provide a "feature description" of the object 12. This description, extracted from the training image, can then be used to recognize the object 12 when attempting to position it in a test image that includes many other objects. In order to perform reliable recognition, it is important that the features extracted from the training image are detectable even under changing conditions of image scale, noise and illumination. Such points are usually located in high contrast areas such as the edges of an object. For object recognition and detection, Robot 100 provides feature key points that are invariant to position, scale and rotation and robust to affine transformations (scale, rotation, shear and position changes) and lighting changes. SIFT can be used to detect. In some embodiments, the robot 100 captures multiple images of scene 10 or object 12 (using camera 320 and / or image sensor 450) (eg, from different angles, under different conditions). .. Robot 100 can access stored images to identify new images by comparison, filtering, and so on. For example, SIFT features can be obtained from the input image and collated with the SIFT feature database obtained from the training image (captured earlier). Feature matching can be performed by the Euclidean distance-based nearest neighbor algorithm. Hough transforms can be used to improve object identification by clustering feature parts that belong to the same object and removing collating parts that do not enter the clustering process. Speed-up robust features (SURF) can be robust image detectors and descriptors.

0194In addition to positioning the robot 100 in scene 10 (eg, the environment around the robot 100), the robot 100 can use the sensor system 400 to move to other points in the connection space (eg work surface 5). it can. The robot 100 maps the adjacent area of the robot 100 and identifies a relatively close object 12 under the short-range image sensor 450a (eg, as shown in FIGS. 1 and 3) under the body 140. A long-range image sensor 450b (eg, as shown in FIGS. 1 and 3) for mapping a relatively large area of the robot 100 and identifying a relatively distant object 12. , Attached to the head portion 160). Robot 100 uses an occupancy map to identify objects in scene 10 as well as known objects 12 in concealment 16 (eg, where object 12 should or should be, but cannot be seen from the current vantage point). 800 can be used. The robot 100 registers a concealment unit 16 or a new object 12 in the scene 10 and tries to avoid the concealment unit or the new object 12 and confirm the position of the new object or an arbitrary object in the concealment unit 16. Can be done. Further, using the occupancy map 800, the robot 100 can track the movement of the object 12 in the scene 10. For example, image sensors 450, 450a, The 450b can detect the new position of the object 12 in the scene 10 while it does not detect the map position of the object 12 in the scene 10. The robot 100 can register the position of the old object 12 as the concealment unit 16 and try to avoid the concealment unit 16 in order to confirm the position of the object 12. Robot 100 compares the new image depth data with the previous image depth data (eg, map 800) and assigns a confidence level for the position of object 12 in scene 10. The position confidence level of object 12 in scene 10 can time out after the threshold period. The sensor system 400 can update the position confidence level of the object 12 after each imaging cycle of the sensor system 400. In some examples, a new concealment 16 (eg an object 12 that disappeared from the occupancy map 800) detected within the concealment detection period (eg 10 seconds or less) is a "live" object (eg a moving object 12) in scene 10. ) Can be expressed.

0195In some embodiments, the second object of interest 12b located behind the detected first object 12a in the scene 10 is not initially detected in the scene 10 and is referred to as the concealment portion 16. .. The concealment unit 16 can be an area within the scene 10 that cannot be easily detected or viewed by the image sensors 450, 450a, 450b. In the illustrated example, the sensor system 400 (or a portion thereof, eg, image sensors 450, 450a, 450b) of the robot 100 has a viewing angle θ to view scene 10._VIt has a field of view 452 (which can be any angle between 0 and 360 degrees). In some examples, the image sensor 450 has a 360 degree viewing angle θ._VIn other examples, the image sensors 450, 450a, 450b have a viewing angle θ smaller than 360 degrees._VHas (eg, between about 45 and 180 degrees). Viewing angle θ_VIn the example where is less than 360 degrees, the image sensors 450, 450a, 450b (or their components) have a 360 degree viewing angle θ._VCan be made rotatable relative to the robot body 110 to achieve. In some embodiments, the image sensors 450, 450a, 450b or a portion thereof may be movable relative to the robot body 110 and / or the drive system 200. Further, in order to detect the second object 12b, the image sensors 450, 450a, 450b are moved by driving the robot 100 around the scene 10 in one or more directions (for example, translating or rotating on the work surface 5). It can be moved to obtain a vantage point that allows the detection of the second object 12b. The movement of the robot or the independent movement of the image sensors 450, 450a, 450b (or a part thereof) can also solve the monocular problem.

0196A confidence level can be specified for the detection position or tracking movement of the object 12 in the work area 5. For example, when creating or updating the occupancy map 800, the controller 500 can specify a confidence level for each object 12 on the map 800. The confidence level can be directly proportional to the probability that the object 12 is actually located within the work area 5, as shown on map 800. The confidence level can be determined by multiple factors such as the number and type of sensors used to detect the object 12. For example, the contact sensor 430 can provide the highest level of confidence because the contact sensor 430 detects the actual contact of the robot 100 with the object 12. The image sensor 450 can provide different confidence levels, which can be higher than the proximity sensor 430. Data received from two or more sensors in the sensor system 400 can be summed or accumulated to provide a relatively higher level of confidence than any single sensor.

0197Odometri uses actuator motion data to estimate changes in position (moving distance) over time. In some examples, an encoder that measures wheel speed, and thus the distance traveled by the robot, is placed in the drive system 200. The controller 500 can use odometry to evaluate the confidence level of the object position. In some embodiments, the sensor system 400 includes an odometer and / or an angular velocity sensor (eg, a gyroscope or IMU470) that detects the distance traveled by the robot 100. A gyroscope is a device that measures or maintains orientation based on the principle of conservation of angular momentum. The controller 500 may use the odometer and / or the gyro signal received from the odometer and / or the angular velocity sensor, respectively, to determine the position of the robot 100 in the work area 5 and / or the position of the robot 100 on the occupied map 800. it can. In some examples, controller 500 uses dead reckoning. Dead reckoning is the process of estimating the current position based on a previously determined position and advancing that position based on a known speed or estimated speed, elapsed time and course. By knowing the position of the robot in work area 5 (eg by odometry, gyroscope) and the detection position of one or more objects 12 in work area 5 (by sensor system 400), the controller 500 (by odometry or gyroscope) It is possible to evaluate the position or movement amount of a relatively high reliability level of the object 12 on the occupancy map 800 and in the work area 5 (as opposed to not using).

0198Odometri based on wheel movement is electrically noisy. Controller 500 can use scan matching with or instead of wheel odometry. The use of scan matching can result in improved accuracy and / or reduced computational load. In such an embodiment, two submaps obtained using lidar and / or other mapping methods can be merged into one map. Two or more partial maps can be merged using known scan locations. Alternatively, two or more partial maps can be merged using the geometric features of the partial scan. The controller 500 can receive image data from the environment around the robot 100 or the image sensor 450 of the scene 10 to calculate the robot position by the visual odometry independently of the wheelbase odometry of the drive system 200. .. Visual odometry requires the use of optical flow to determine the movement of the image sensor 450. The controller 500 can use the motion calculated based on the image data of the image sensor 450 to correct the error of the wheelbase odometry and thus enable improved mapping and motion control. Visual odometry can be limited to low texture or low light scene 10 if the image sensor 450 is unable to track features in the captured image.

0199Other details and features relating to odometry and imaging systems that can be combined with those described herein are described in US Pat. No. 7,158,317 (described for the "depth of view" imaging system) and US Pat. No. 7,115,849. See (described in the Wave Surface Coded Interference Contrast Imaging System), the contents of these patent specifications are incorporated herein by reference in their entirety.

0200With reference to Figures 8C and 8D, when the robot works in the building for the first time, the robot needs to look around for autonomous driving or be given a map of the building (eg, the location of rooms and corridors). .. For example, in a hospital, the robot needs to know the position of each patient room, nurse station, and so on. In some embodiments, the robot 100 can receive the floor plan map 810 as shown in FIG. 8C and train the floor plan map 810. For example, while guiding the robot 100 around the building, the robot 100 can record a specific position corresponding to the position on the plan map 810. The robot 100 can display the floor plan map 810 on the web pad 310, and when the user wants to move the robot 100 to a specific position, the user can move the robot 100 to that position on the floor plan map 810 (for example, the web pad 310). Can be tagged (using a touch screen or other pointing device). The user may choose to enter a label, such as a room name or room number, for the tagged location. At the time of tagging, the robot 100 can store the tag along with the points on the plan map 810 and the corresponding points on the robot map 820 as shown in FIG. 8D. As shown in the figure, the robot map 820 can be a two-dimensional plan map similar to the plan map 810. In an alternative embodiment, the robot map 820 can be a three-dimensional map that includes a ground surface corresponding to a two-dimensional plan map similar to the plan map 810.

0201Using the sensor system 400, the robot 100 can move around to create a robot map 820. For example, the sensor system 400 can provide information about how much the robot has moved and the direction of movement. Robot map 820 may include fixed obstacles in addition to the walls provided on plan map 810. Robot 100 can use Robot Map 820 to perform autonomous travel. In Robot Map 820, the "wall" does not appear perfectly straight due to detection objects such as luggage boxes and / or furniture inside the chambers placed along the walls of the corresponding corridors. Further, there may be a rotation difference and a resolution difference between the plan view map 810 and the robot map 820.

0202Referring to FIG. 8E, in some embodiments, the telepresence software application 601 displays a tagging view 660 that allows the user to place the tag 662 on the floor plan map 810. The floor plan map 810 can be the same map as displayed by the floor plan map window 620, or it can be a different map used internally for navigation purposes.

0203The user (remote terminal) and / or the robot attaches a tag 662 to a specific position on the plan map 810 and / or the robot map 820 to mark the map position with information such as driving hazards, obstacles, robot assistance, etc. Can be inserted. For example, the user can drag on and drop the tag 662 to a specific location on the floor plan map 810. As described herein, a tag is a tag information that specifies the coordinates associated with a point or region, the purpose of the tag, the type of tag, the nature of the tag, user commands and / or the robot associated with the tag. And / or include other information about the tag, and finally, the tag can include annotations containing 2D and / or 3D graphics or text corresponding to the tag. An example of a tag annotation is an octagonal red stop sign associated with a tag that contains tag information indicating an area that the robot should not enter. Tag annotations can be human and / or machine readable. Tag coordinates can be points, lines, planes, surfaces, volumes, and / or 2.5D or mixed surfaces. Tags can be formed as a data structure with any number of additional fields and / or parameters. For example, tags include fields associated with time, scheduling, spatial coordinates and / or triggers for a given function.

0204As used herein, "commentary" includes text, words or other linguistic expressions. Thus, tag annotations can include pictures, graphic images, pictograms, hieroglyphs, and non-linguistic symbols. In addition, tag annotations can be in word, letter, phrase or other text format. For example, a tag associated with a nurse station can include a tag annotation with a textual representation of the nurse name. The text representation of the nurse name can be two-dimensional text or three-dimensional text. Also, the tag annotation associated with the nurse name can be an uppercase N or symbol (eg, a nurse cap or nurse symbol) representing the nurse station.

0205The tag 662 can include a wireless local area network (WLAN) warm tag 662a indicating a relatively good signal receiving area and a WLAN cold tag 662b indicating a relatively weak signal receiving area. Robot 100 can use this information to move from one position to another through a relatively good radio signal reception area, avoiding a relatively weak radio signal area.

0206The low traffic tag 662c indicates an area with relatively low traffic (humans and / or robots). The robot 100 can select a movement path through an area having a low traffic volume rather than an area having a relatively high traffic volume. In addition, if the robot 100 has to move through a high traffic area, the robot 100 will avoid one or more specific obstacle detection obstacles in order to move successfully without colliding with that area. (ODOA) can be performed.

0207The dog tag 662d indicates the location of the robot docking station. The low battery level event signals the controller 500 to recharge. Robot 100 uses the map position tagged with the dog tag 662d to position the robot docking station for recharging. For example, by applying the determined distortion between the plan map 810 and the robot map 820 (FIGS. 8C and 8D), the robot 100 determines the robot map position 824 corresponding to the tagged layout map position 814. You can then move to that tagged position and dock with the robot docking station. Determining distortion involves determining the distortion between two maps that use the same coordinate system. Both the robot map 820 and the plan map 810 can be two-dimensional, so strain determination can eliminate the need to determine coordinate transformations between different dimensions.

0208Several tags 662 can be used to indicate obstacles or special crossing areas. For example, the glass tag 662e indicates the location of a glass wall, window or door. Robot 100 can use this information to avoid glass-tagged structures because infrared proximity sensors cannot detect them. Lamp tag 662f indicates the position of the floor slope. For example, the robot 10 can detect a floor slope as an obstacle because the floor slope may have a vertical height that is greater than the sill height that can be overcome. When approaching a ramp tag slope, Robot 100 can perform a slope or crossing action to successfully pass through the slope. The tight tag 662g indicates the location of a relatively narrow passage or doorway. Robot 100 can avoid such areas to avoid confinement.

0209The low speed tag 662h indicates a place or area where the robot 100 travels relatively slowly. This location or area may coincide with a high traffic area. The avoidance tag 662i indicates a place or area where the robot 100 should be avoided (ie, not traveled). In some embodiments, the avoidance tag 622i can be mode dependent. For example, the avoidance tag 622i can only be applied when the robot is operating in fully autonomous mode. During remote control, the avoidance tag 662i can be effectively ignored by the robot. The operating room user interface (ORUI) tag 662j indicates the location and area of the hospital operating room. Robot 100 can use this tag to find the operating room, enter the OR area to provide telepresence assistance and / or display a specific user interface (eg, ORUI). The training tag 662k can be used to mark rough locations such as corridors and rooms in order for the robot 100 to learn and train about its environment 10.

0210The manual elevator tag 662l indicates the position of the elevator, in which case the robot 100 can allow the user to assist the user in getting on and off the elevator. Manual elevator negotiation may be based on the control of a remote user or the guidance of a robot local user. For maneuvering the remote user, the remote user supplies a drive command to the robot 100 (eg using a joystick). Regarding the guidance of the robot local user, a person near the robot 100 physically touches the robot 100, and the robot 100 moves in response to such a touch. A feature of robotic responsiveness to user touch that can be combined with that described herein is found in U.S. Patent Application No. 13.032,390 filed February 22, 2011, which U.S. application is by reference. It is incorporated herein by reference in its entirety.

0211The automatic elevator tag 662m indicates the position of the elevator in which the robot 100 can autonomously negotiate (get on and off). Robot 100 can perform the threshold-over action 512d (Fig. 5) to get on and off the elevator so as not to fall. A feature of robotic responsiveness to user touch that can be combined with that described herein is found in PCT / US11 / 59110 filed November 9, 2011, which is in its entirety by reference. Incorporated herein.

0212The keep light tag 662n indicates a map position or area where the robot 100 should pass on the right side. The user can place this tag along a predetermined corridor, such as a high traffic corridor. In response to the keep light tag 662n, the robot 100 can perform side-by-side actions and travel along the wall while traveling in the tagged area.

0213After map training, when the user wants to send the robot 100 to a position, the user refers to the label / tag 622 (eg, enter the label or tag in the position text box displayed on the web pad 310) or the robot. 100 can display the floor plan map 810 on the web pad 310 for the user, and the user can select a position on the floor plan map 810. If the user selects the tagged layout map position 814, the robot 100 can easily determine the corresponding robot map position 824 on the robot map 820 and proceed to navigate to the selected position 814. it can.

0214In some embodiments, the robot controller 500 performs a first action while maneuvering around the first area and then around the second area associated with a tag having the associated robot behavior modifier. You can perform a second action while maneuvering. For example, while executing the human follow-up action 512b, the robot controller stops executing this action when it reaches the map position 814 tagged with the slope tag 662f or the auto-elevator tag 662n, or simultaneously performs the threshold crossing action 512d. Can be executed.

0215If the selected position on the floor plan map 810 is not the tagged position 814, the robot 100 determines the corresponding position 824 on the robot map 820. In some embodiments, the robot 100 uses the current tagged position to calculate the scale, origin mapping and rotation between the plan map 810 and the robot map 820, and then the calculated parameters. To determine the robot map position (using, for example, affine transformations or coordinates).

0216The robot map 820 does not have to have the same orientation and scale as the floor plan map 810. Moreover, the layout map may not be at a constant scale and may have distortions that vary from map area to map area. For example, a plan map 810 generated by scanning a fire blame map commonly found in hotels, offices and hospitals is generally not drawn to a constant scale and may have different scales in different areas of the map. Robot map 820 can also have its own error. For example, the position on Robot Map 820 can be calculated by counting the number of wheel revolutions as a measure of distance and is inaccurate if the floor is slightly slippery or if it turns a corner (causing extra wheel revolution). Rotation calculations can cause the robot to inaccurately position the mapped object.

0217A method of mapping a given point 814 on the floor plan map 810 to the corresponding point 824 on the robot map 820 is to use the existing tagged points 812 within the region containing the layout map point 814 (eg, threshold). Calculate the local strain (in the same 2D coordinate system) between the plan view 810 (within the radius) and the robot map 820. The method further applies a strain calculation to the layout map point 814 to find the corresponding robot map point 824. For example, if you want to start from a given point on the robot map 820 and find the corresponding point on the floor plan map 810 to request the current position from the robot, you can do the above calculation in the opposite direction. it can.

0218Any of a wide variety of tag schemas and data structures can be used. For example, a tag can include attributes in the form of key-value pairs that specify the purpose of the tag, tag parameters, and tag attributes (generally "tag information"). Table 1 below provides a specific example.<tables num="1"><img id="000002" he="33" wi="159" file="JP5905031B2_D0001.tif" img-format="tif" img-content="drawing" /></tables>

0219Area-related tags have attributes associated with the area, and these attributes specify their purpose as well as parameters that influence the behavior associated with the area. These key-value pairs can be stored using a data structure similar to the example in Table 2 below.<tables num="2"><img id="000003" he="33" wi="159" file="JP5905031B2_D0001.tif" img-format="tif" img-content="drawing" /></tables>

0220The data structure for each tag can include tag coordinates and tag information, which can include annotations (eg, a 2D and / or 3D graphic representation of the tag). Table 3 below provides specific examples of tag data structures.<tables num="3"><img id="000004" he="87" wi="160" file="JP5905031B2_D0001.tif" img-format="tif" img-content="drawing" /></tables>

0221As described herein, tags can be associated with areas rather than specific points on the map. There can be a many-to-one relationship between tag information and tags. Specific examples of tag data structures associated with regions are provided in Table 4 below.<tables num="4"><img id="000005" he="105" wi="160" file="JP5905031B2_D0001.tif" img-format="tif" img-content="drawing" /></tables>

0222In some examples, the shape of the region can be divided into their centroids and the components of the offset from the centroids in order to allow the position and rotation of many objects to be rapidly updated. When CPU resources are available, in the bounding box of the final coordinates (polygon points with respect to the barycentric coordinate system converted to the map coordinate system by the orientation of the barycentric coordinate), use an R * tree for high-speed lookup or a similar data structure to make geometric constraints. Can be indexed based on. The points that make up the region polygon can be stored in a clockwise (or counterclockwise) order to facilitate point-in-polygon testing based on ray tracing.

0223As an example, a tag indicating a region that is a low speed zone can have a data structure as given in Table 5 below.<tables num="5"><img id="000006" he="44" wi="159" file="JP5905031B2_D0001.tif" img-format="tif" img-content="drawing" /></tables>

0224Table 5 shows an example of a slow zone tag, in which the speed limit is explicitly set based on the value associated with the region itself. The area is also defined so that the robot can interpret the area as a low speed zone. For example, in Table 6 below, the robot can interpret the area defined as an intersection as a low speed zone and reduce its speed to a predetermined speed.<tables num="6"><img id="000007" he="27" wi="159" file="JP5905031B2_D0001.tif" img-format="tif" img-content="drawing" /></tables>

0225Tags associated with points or areas on the map can include tag coordinates, tag information and a wide variety of arbitrary tag annotations. In addition, tags can be implemented using any of a wide variety of data types, including the data types shown in Table 1-6 above. Tag information and tag annotations refer to individual elements herein. However, according to various embodiments, the tag annotation can be part of the tag information. To be precise, the data structure may or may not include a separate filter for tag annotation. Rather, data structure fields for information tags can contain tag annotations.

0226FIG. 8F provides an array of exemplary operations for operating the robot 100 to navigate the environment using the plan map 810 and the robot map 820. This array of operations is generated by operation 802f, which receives the plan map 810 corresponding to the environment of robot 100, operation 804f, which moves the robot 100 to layout map position 812 on the plan map 810 in the environment, and robot 100. Operation 806f that records the robot map position 822 on the robot map 820 corresponding to the environment, and operation that determines the distortion (two-dimensional) between the robot map 820 and the plan view map 810 using the recorded robot map position 822. 808F, and the determined distortions include operations 810f to determine the target robot map location 824 corresponding to apply to the target layout map location 814, thereby robot navigating the door 100 to the selected position 814 in the plan view map 810 Will be possible. In some embodiments, the method uses the current tagged position to determine the scale, origin mapping and rotation between the plan map 810 and the robot map 820, as well as the selected target layout map. Includes an operation to determine the robot map position corresponding to position 814. This method of operation can include applying an affine transformation to the determined scale, origin mapping and rotation to determine the robot map position. Any of the above operations can be repeated any number of times to improve accuracy and / or efficiency. For example, the operation 804f to move the robot in the environment and the operation 806f to record the robot map position should be repeated many times to collect enough correction points for conversion and calculation between subsequent layout maps and robot maps. Can be done.

0227Other details and features that can be combined with this can be found in PCT application No. PCT / US11 / 60935 filed November 16, 2011, which application is incorporated herein by reference in its entirety.

0228With reference to FIGS. 9A and 9B, in some embodiments, the teleoperation software application 601 displays a mixed 3D image map 622b (mixed map) within the map window 620. The hybrid map 622b is a combination of the remote view 612 displayed in the remote video feed window 610 and the two-dimensional top-down map 622a displayed in the floor plan map 810, for example the floor plan map window 620 (FIG. 6D). can do. FIG. 9A shows a remote video view 612 that the user can see when the robot 100 is in the middle position. FIG. 9B shows a hybrid map 622b, which partially overlaps the floor plan map 810 and has been modified to fit the remote view 612, with the room numbers and area numbers in the area within the field of view of Robot 100. / Or indicates the room type. When watching a live video feed, the user can hover over the window and start moving the scroll wheel upwards. During the transition process, the perspective video view (from camera 320 on robot 100) gradually transitions between remote video view 612 and map 622. Map 622 is completely distorted to map perspective remote video view 612 at the beginning of the transition, gradually disappears, and returns to an undistorted view at the end of the transition. So if you raise the mouse wheel by 30%, the user will see a dissolve image containing 70% video and 30% map, the video part will be 30% undistorted, but the map will be 70% distorted. .. This embodiment allows for a single view that fluidly represents both the perspective live remote video view 612 and the map 622.

0229To provide the hybrid map 622b, the teleoperation software application 601 uses the recorded robot map position 822 and the corresponding layout map position 812 of the robot map 820 to distort between the remote view 612 and the plan map 810. (Distortion between 2D and 3D coordinates) can be determined. In some embodiments, the strain determination determines the scale, origin mapping, and rotation between the plan map 810 and the remote view 612, and applies, for example, to the scale, origin mapping, and rotation that determines the affine transformation. Includes operations to do. Determining the distortion between a 2D plan map and a 3D video feed can include operations that determine coordinate transformations between different coordinate systems.

0230With reference to FIGS. 6D and 10A-10E, in some embodiments, the user interface 605 displays a look-ahead view 612a rendered within a map window 620, a look-ahead command 624, a dedicated separate window, or some. Provides other windows. While driving the robot 100, the user can call the look-ahead command 624 to cause the robot 100 to stop physical movement, while the teleoperation software application 601 generates and displays the rendered look-ahead view 612a. Provides a perspective view of the proposed robot drive path, as if the robot 100 continued to move along that path. This can be achieved by using map data such as wall positions and configuring a perspective "virtual reality" view based on the virtual position of the robot 100. For example, the telepresence software application 601 can configure the Luclock ahead view 612a using the floor plan map 810, the robot map 820 and / or the stored image data 701 (FIG. 7). In a robotic system that uses the cloud computing service 720 as shown in FIG. 7, the telepresence software application 601 communicates with the cloud computing service 720 to store image data 701, 3D map 705 and / or 2D. Height map 707 (or instead 2. The look-ahead view 612a can be configured based on the 5D hybrid map) and then supplied to render the look-ahead view 612a into the map window 620. In this embodiment, the telepresence software application 601 can leverage the scalable computer processing and data storage capabilities of the cloud service (eg, the cloud service 720 flexibly scales up and then scales down to process the device. Therefore, the processing and memory requirements for computing devices running the telepresence software application 601 can be reduced.

0231FIG. 10A shows an exemplary remote view 612 of the remote video feed view 610 of the telepresence software application 601. FIG. 10B shows a complementary map 622 displayed in map window 620. Map 622 presents the current position of the robot 100, indicated by the robot icon 650, along with the camera field of view 322 of the robot camera 320. Figures 10C and 10E present an exemplary look-ahead view 612a displayed in the remote video feed window 610. The remote video feed window 610 can continue to display the remote view 612 from the robot camera 320 in, for example, a picture-in-picture window located at the corner of the remote video feed window 610. Figures 10D and 10F present an exemplary map 622 displayed in map window 620. While executing the look-ahead command, the telepresence software application 601 can render the robot icon 650 at the robot's current position, along with the robot's camera field of view 322. In addition or instead, the telepresence software application 601 can render a virtual robot icon 650a moving along the look-ahead path, along with a projected look-ahead camera field of view 322a, within the floor plan map window 620.

0232In some embodiments, the user uses a joystick to communicate with the telepresence software application 601 to drive the robot 100 along a corridor by issuing the look-ahead command 624 (eg, on the user interface 605 or the joystick). It can be activated (by selecting the corresponding button). For example, at position 50, 50 feet away from the corner of the corridor, the user can activate the look-a-head command 624 to generate the look-a-head view 612a and stop further movement of the robot 10 along the corridor. However, the user can continue to virtually move the robot 100 in look-ahead mode. User interface 605 can display a look-ahead view 612a (eg, a 3D model) at the same location in the same corridor. When the user drives the robot forward in look-ahead mode, advances it 50 feet, turns left, and continues to drive, the user can see the location of rooms and other corridors along the aisle in the 3D model / look-ahead view 612a. it can. In some examples, for the first 30 feet of "virtual drive", the telepresence software application 601 is further magnified and distorted to match the vertical position of the actual view (from a stationary physical robot). It is possible to display a mixed view of the perspective view) and the 3D model / look-ahead view 612a.

0233Figure 10G presents an exemplary sequence 1000 of operations that perform the look-ahead mode of the telepresence software application 601. This array of operations includes operation 1002 that initiates look-ahead mode (also referred to as fly-through mode) and operation 1004 that inspects the actual localization of the robot 100 (eg, the robot's current attitude and / or coordinates). The robot 100 determines its position based on the received sensor signal of the sensor system 400 and then transfers the position to the telepresence software application 601 and / or the cloud computing service 720. This array of operations further includes operations 1006 that generate the virtual position and / or attitude of the robot 100. The telepresence software application 601 or cloud computing service 720 uses the dynamic model 570 (Fig. 5) and image data 701 (Fig. 7) associated with the robot 100 (eg, volumetric point cloud image data) to virtualize and / or Can generate postures. The method can include operation 1008 to access the 3D rendering data corresponding to the determined virtual robot position and operation 1010 to generate a 3D rendering of the robot 100 and / or the local environment around the robot 100. It is a 3D map stored in image data 701 (eg, volumetric point point cloud image data) and / or locally or remotely in cloud storage 722 to configure a local 3D model / lookahead view 612a. May require an operation to access the 705. This view 612a can be displayed in the remote video feed window 610 by the telepresence software application 601. In addition, this is a virtual robot icon 650a and a projected look-ahead turtle It is possible to require an operation to generate a 3D model of the robot 100 shown in the field of view 322a in the map window 620. This operation array updates the displayed look-a-head view 612a or first-person view (POV) when the robot 100 virtually operates in look-ahead mode, and also updates the virtual position / attitude of the robot 100. Can include. Steps 1008-1014 can be repeated (eg, periodically) until step 1016, which exits the look-ahead / fly-through mode.

0234Referring to FIG. 11A, in some embodiments, the user interface 605 of the telepresence software application 601 displays the remote navigation view 612b within the remote video feed window 610 (or another window). The remote navigation view 612b can have a navigable area 616 rendered above the video feed of the remote view 612. The user can toggle between the remote view 612 and the remote navigation view 612b. The navigable area 616 can be determined based on the floor plan map 810 and / or the robot map 820. The navigable area 616 can be shown as an obstacle-free bounded area with a field of view of 832 for robot cameras 320,450. In addition, the navigable area 616 can be filled with colors or other signals that convey to the user that there are no obstacles or other obstacles.

0235The navigable areas on the layout map can be highlighted based on the information in the robot's internal obstacle map. In one embodiment, the navigable area can be identified as white pixels in the image. The robot returns its position on the robot map and the position and orientation of the 3D depth camera. The processor can use the robot position and the motion state of the head camera (eg, pan and tilt angles) to determine which pixels on the video screen represent the ground. That is, the processor can determine the coordinates of each ground pixel on the video feed using the robot position and the field of view of the video feed. White pixels indicating the navigable area can then be overlaid on the ground pixels on the video feed. Thus, the robot and / or the user controlling the robot can identify the navigable area by tracking the white pixels. In some embodiments, pixels of any color or other identification marks can also be used. Other data structures or marks can also be used in the plane of the white pixels. In particular, from the robot's POV, the coordinates of navigable ground pixels can be tagged in a wide variety of ways, as long as the robot can recognize them.

0236The user can select a robot destination 618 in the navigable area 616 and have the telepresence software application 601 issue a drive command to move the robot 100 to a position corresponding to the selected robot destination 618. In the illustrated example, the remote video feed window 610 of user interface 605 provides robot navigation view 612b of robot 100 in the hospital room. The user selects robot destination 618 as the location next to the patient bed in the room. The telepresence software application 601 uses a plan map 810, a robot map 820, a 3D map 705, a 2D (or 2.5D hybrid) height map 707 and / or stored image data 701 on the remote navigation view 612b. The distortion between the selected robot destination 618 and the corresponding robot map position 824 (distortion within the same dimension and / or between different dimensions) can be determined. The telepresence software application 601 then autonomously or semi-autonomously drives the robot 100 to the robot map destination 824 using the sensor system 400 and the action system 510a to avoid obstacles such as moving people. It is possible to issue a drive command to make it.

0237In one example, the map can be returned as an image such as a PNG, JPG or TIFF image from a robot API call. The robot cloud processes an image to detect pixels (eg, black pixels) that outline obstacles in the image. A curve fitting algorithm can be used to process the pixels that contour the obstacle. The resulting curve can then be used to generate an obstacle map. Additional processing can be performed to further improve the detection of obstacles and / or to further improve the curve fitting accuracy of the detected obstacle contours. For example, if the curve approximates a circle-like shape, the obstacle map can easily substitute a circle. Similar ideas can be applied to shapes such as rectangles or ellipses, humans, faces and / or objects that approximate a database of known object shapes from various fields of view.

0238The user interface 605 can provide a proximity sensor window 670 that displays the proximity of obstacles within the sensor field of view 442,452 (eg, within the field of view 452 of the 3D imaging sensor and / or the field of view 442 of the laser scanner).

0239In some embodiments, the user can use the avoidance tags 662,662i to mark protected areas / zones on the remote video feed window 610 and / or the floor plan map window 620 (not shown). The protected zone is treated as object 12 by the robot 100, so the protected zone can be evaded during autonomous driving. Protected zones can be used to create a large space around delicate equipment, or to allow the robot to avoid other areas. The user can place the avoidance tags 662,662i on the floor plan map 810 in the tagging view 660 or on the remote navigation view 612b. In addition, the user can place other tags 662 on the remote navigation view 612b. The telepresence software application 601 can determine the distortion between the remote navigation view 612b and the plan map 810 and / or the robot map 820, and then update the robot map 820 accordingly.

0240For example, determining the distortion between the floor plan map and the video feed can include the step of generating a transformation mapping between the coordinate points in the navigation view 612b, the floor plan map 810 and / or the robot map 820. Similar to overlaying a restricted area on the ground surface in a video feed, the ground surface of the 2D map can be effectively coordinate-mapped onto the detected ground surface of the video feed provided by the robot.

0241FIG. 11B shows a flowchart of an exemplary sequence 110 of operations for a method of navigating a robot to a selected robot destination 618. The method includes step 1102 identifying a navigable area 616 within the fields of view 322,442 of Robot 100. Identification of the navigable area 616 can be achieved using the sensor system 400 of the robot 100. The method visually displays the navigable area 616 on the user interface 605, for example by displaying a bounded area (a highlighted boundary filled with color or pattern) on the remote navigation view 612b. Step 1104 Also includes. The method can include step 1106 to receive user selection for robot destination 618 and step 1108 to determine if robot destination 618 is within the identified navigable area 616. If the robot destination is outside the identified navigable area 616, the method includes step 1110 prompting the user to select a valid robot destination 618 within the navigable area 616. If the robot destination 618 is within the identified navigable area 616, the method includes step 1112 to determine the route to the robot destination 618. It may be necessary to determine the distortion between the remote navigation view 612b and the robot map 820, and then determine the robot map position 824 corresponding to the selected robot destination 618. The method includes step 1114 allowing the robot 100 to navigate (autonomously or semi-autonomously) to the robot destination 618.

0242Figures 11C and 11D show an exemplary remote navigation view 612b, in which case the user is located beyond the navigable area 616 or on an obstacle 1120 (actual or perceived by Robot 100). The robot destination 618 to be located is selected. In the example shown in FIG. 11C, the user has selected robot destination 618 on the perceived obstacle 1120a (slope 1122). From a distance, the slope 1122 can have a perceived height that exceeds the traversable threshold height of the robot 100, so that the robot sensor system 400 can recognize the slope 1122 as an obstacle from a distance. Further, the robot action system 510a can perform the ODOA action 512c in response to a sensor event caused by the sensor system of the sensor system 400 exhibiting an obstacle having a height greater than the crossable threshold height. Robot 100 uses plan map 810 and / or robot map 820 to determine that its local perception of the environment is inaccurate and that slope 1122 is not a real obstacle, but rather a perception obstacle 1120a. Can be done.

0243Although slope 1122 is within navigable area 616, telepresence software application 601 can determine that robot destination 618 on slope 1122 is an unsafe place to stop. The telepresence software application 601 can display an alert dialog box 1130 to inform that the selected robot destination is not safe to stop. In the illustrated example, the alert dialog box 1130 indicates that the user has selected slope 1122 for robot destination 618 and presents just in front of slope 1122 as an alternative robot destination 619. Stopping the robot 100 on the slope 1122 is dangerous for both the person near the robot 100 and the robot itself if the robot 100 falls or falls off the slope 1122. By determining that the robot destination 618 is on slope 1122, the telepresence software application 601 prohibits such robot destination 618 and / or is a safe alternative destination 619 (in this example, slope 1122). Before) can be proposed.

0244Referring to FIG. 11D, when the user selects the actual obstacle 1120b, the telepresence software application 601 displays the alert dialog books 1130 and the selected robot destination 618 is outside the navigable area 616 or the obstacle 1120. Can be shown to be. In the illustrated example, the alert dialog box 1130 indicates that the user has selected obstacle 1120 for robot destination 618 and presents immediately in front of obstacle 1120 as an alternative robot destination 619.

0245Referring to FIG. 12, in some embodiments, the remote navigation view 612b displayed by the user interface 605 of the telepresence software application 601 follows the robot path 652 to the selected robot destination 618 within the navigable area 616. Can be specified to the user. The user can specify the robot path 652 using various input devices. For example, on a touch screen display, the user can drag a finger or stylus from the robot icon 650, which indicates the current position, to the robot destination 618. In an additional example, the user can drag the robot icon 650 (using the mouse or touch gesture) to the robot destination 618 along the specified robot path. In the illustrated example, the user can select the route setting button 1202 on user interface 605 to indicate that the gesture performed within the navigable area 616 should be interpreted as robot route 652. The user can track the robot path 652 in the remote video feed window 610. Similarly, the user can select the robot path 652 on the plan map 810, which is displayed in the map window 620 as the two-dimensional map 622a. After setting the robot path 652, the user can press the go button 1204 to set the robot 100 in motion. Similarly, the stop button 1208 can be used to stop the movement of the robot. The clear path button 1206 can remove or clear the set robot path 652 from the remote navigation view 612b.

0246The display window can include a flyout icon panel that appears when you hover your mouse over it. For example, the icon panel can pop out from the top left of the window. The icon panel allows the user to select manual drive, click-to-drive and head motion icons. In one embodiment, the user can use the spacebar to toggle the icon. Manual drives allow users to use click-to-objective values and / or click-and-drag routes. After drawing the route on the map, the user can right-click and select "Save Route" from the pop-up menu. This menu allows you to give a name to the route. The user can then perform a "route load" and the robot can move to the starting point of the route and then to the destination along the specified route. The route can be stored as a tag data structure including tag coordinates and tag information. The tagged route contains numerous stops along the route. When drawing a route, the user can indicate waypoints along the route. In some embodiments, waypoints can be represented by tag annotations that include a stop sign. Then, when the waypoint is reached while traveling on the route, the robot can blink the "stop" button to make the route brighter and / or translucent. At this point, the user can see, do a local drive, and then click "go" to resume the route. Therefore, the physician can save the route for night rounds and visit all rooms and stations in a preferred order with a pre-planned route.

0247In head mode, the user can draw a box or outline over a portion of the video feed to center the robot's head (top) on the box or an object within the box. In addition, the user can click on a position to change the orientation of the robot's head (top) and / or the entire robot. Various buttons and peripheral control toggles can be used to use the robot's base (bottom) and head (top) independently. For example, holding the shift key while in head mode allows the cursor to become a hand icon on the screen, allowing the user to grab and drag the head.

0248In some embodiments, the star icon can be used to control the navigation of the robot. The star icon can be displayed in any of the various views and can be selectively moved by the user to change the direction and / or speed of the robot. In addition to the star icon, you can also use an alternative icon.

0249Returning to FIG. 12, the virtual joystick window 680 can provide another input device for designating the desired path 652 or for manually controlling the robot 100. The virtual joystick window 680 can display the robot orientation indicator 682 and the navigation vector 684. The user can control the direction and speed of the robot 100 using the navigation vector 684. The virtual joystick facilitates control of the robot 100 by the user of a device such as a tablet computer, which generally does not have a mouse or a regular joystick.

0250A "stitch" video image can be displayed in the virtual joystick window 680. "Stitch" videos and images can be generated using the live downward cameras 320,450 at the front of Robot 100 and the live downward cameras at the rear of Robot 100. The user can grab and drag on the robot motion indicator 686 to specify the robot motion direction and drive speed (eg using a mouse or touch gesture). Driving the robot from the virtual joystick window 680 is advantageous over the mouse-based or remote video feed window 610 for driving the virtual joystick. In particular, this view is a perception based on lens distortion, lack of depth information, and rotation of the robot head that the user can experience while driving the robot with the video feed displayed in the remote video feed window 610. The problem can be reduced.

0251In addition to allowing the user to specify the desired route within the remote video feed window 610, the user can specify the robot route 652 on the map 622 displayed in the map window 620. Specifying the robot path 652 in the plan map 620 allows the robot 100 to navigate longer distances, thus allowing the user to be freed while the robot 100 is in motion to perform other tasks. it can. Various controls can also be provided for zooming and manipulating the display area of map 622 shown in map window 620. The slider 1210 can be used to specify the desired zoom, and the area pan control 1212 can be used to display the desired area.

0252Therefore, even a non-technical user can navigate from one position to another using any combination of various navigation methods and control devices. For example, in long-distance travel, the user can click on a destination on the floor plan map and the robot can autonomously navigate to the selected position. For medium-range travel, the user can select a destination within the video window of a location within the robot's field of view. In short-distance movement, the user can manually control the navigation path, rotation, head movement, etc. of the robot using a mouse, a joystick, a virtual joystick, or a meta-joystick.

0253FIG. 13 shows an exemplary user interface 605 of a telepresence software application 601 with a maximized remote video feed window 610 displaying a remote navigation view 612b, which is a user-selectable hypertag. Receive 1310 and / or context sensitive commands. User interface 605 includes a local video window 630 and a floor plan map 620 overlaid on the remote video feed window 610. In the illustrated example, the plan map 620 displays the 3D map 622c. The 3D map 622c can be used by the user to semi-autonomously navigate the robot 100 to the selected robot destination 618 on the 3D map 622c. In some embodiments, the virtual 3D grid 1302 is displayed in the remote navigation view 612b. Using the determined distortion between the plan map 810 and the robot map 820, the telepresence software application 601 can locate the floor in the live video feed and overlay it on the 3D map 622c. The user can select the square grid 1304 on the virtual grid 1302 as a robot and autonomously navigate the robot 100 to the selected square grid 1304. The virtual three-dimensional grid 1302 can improve the accuracy of positioning the robot 100.

0254In the illustrated example, various hypertags 1310 that provide context-sensitive behavior are displayed and made available to the user. Context-sensitive actions include approach command 1312 and follow command 1314. These context-sensitive actions can occur when a person 1330 is identified within the field of view 322,442,452 of Robot 100. The user can call the approach command 1312 to position the robot 100 in front of the person 1330. The approach command 1312 can cause the robot action system 510a to execute the approach action 512a (Fig. 5), whereby the robot 100 uses the sensor system 400 (eg, facial recognition) to identify and identify the person 1330. You can move in front of 1330. The user can call the follow command 1314 to have the robot 100 follow the person 1330 from 3 feet behind. The follow command 1314 can cause the robot action system 510a to execute the human follow-up action 512b, whereby the robot 100 uses the sensor system 400 (eg, facial recognition) to identify the person 1330 and move about the identified person 1330. To do. In some examples, Robot 100 can use facial recognition routines to detect individuals within field of sight 311,442,452. A label 1340 that identifies the person can be displayed. For example, the information can include name, title, affiliation, address, business address, email address, web page address, user annotations, and so on.

0255The telepresence software application 601 can determine the distortion between the displayed 2D map 622a and the personal video feed captured by the robot camera 320. Determining such distortion may include determining the coordinate transformation between the 2D map and the 3D "map". When the user places the tag 662 and / or the hypertag (which may contain the tag) 1310 on the remote view 612 of the remote video feed window 610 or on the 2D map 622a of the map window 620, the telepresence software application 601 distorts it. Is applied to the tag map coordinates associated with the tag 662 and / or hypertag 1310 to determine the corresponding video seat or plan map coordinates, respectively, and the determined video or map view coordinates are used to tag the tag 662 or hypertag. Tag annotations associated with 1310 can be overlaid to the displayed remote view 612 (ie, personal video feed) or map 622, respectively. In various embodiments, the 3D rendition of the tag annotation can be dynamically re-rendered based on the current location of the remote telepresence robot and the tag's perspective on the video feed. Thus, tag annotations can be dynamically re-rendered when the robot's position and / or perspective of the live video feed changes, for example when the robot's head (top) pans or tilts. For example, tag annotations corresponding to slopes can be overlaid in a video feed with respect to the floor. Similarly, tag annotations associated with an object on a wall can be overlaid with respect to the object or wall.

0256As described herein, the tag can include tag information that includes a robot behavior modifier. The tag can be interpreted by a robot operator, a local terminal or a remote telepresence robot to cause the robot to perform a predetermined action. For example, a robot behavior modifier must not invade a particular area, move at a slow speed in a particular area, move at a high speed in a particular area, take special care, and / or perform other actions. Can be instructed to the robot. Tags are generally any of a wide variety of information, such as the availability of wireless communication systems, the speed of movement of remote telepresence robots, the position of points of interest, the position of people, the position of docking stations, the position of rest areas, glass. It can include wall positions, slope positions, object positions, optimal routes for navigating narrow areas, optimal routes for navigating crowded areas, and actions to be performed by remote telepresence robots. Tags can be generated by the user, autonomously by the terminal, autonomously by the robot, and / or in response to historical data collected by the terminal and / or the robot.

0257The robot can include a tag identification system configured to identify tags having tag coordinates that meet along a navigation path. The robot may "encounter" this tag when its tag coordinates are within the robot's local perceptual space and / or when other group coordinates correspond to a planned navigation path or navigation path plan of interest. Thus, the tag identification system can "encounter" the tag along the navigation path even if the robot is not yet close to and / or cannot be close to the tag coordinates of the tag.

0258The robot and / or the remote terminal that determines the navigation route of the robot can consider tags that can influence the navigation route or possible tags. Thus, during navigation path determination, the tag identification system can be used to identify tags with tag coordinates projected along a possible navigation path. For example, several possible navigation routes can be used to reach the desired destination, and the choice of which navigation route to use can be determined by the tags associated with each of the possible navigation routes. it can. A robot that makes a selection from a large number of possible navigation routes can identify relevant tags to determine which navigation route provides the best wireless connectivity. Other factors such as tilt, elevator, distance, congestion, and objects can also be used.

0259In the illustrated exemplary user interface 605, the dashboard window 640 provides a robot outline with a battery charge status, a radio signal strength indicator, and a portion that may light up when a service is requested. The option window 690 allows the user to detach or dock the robot from the docking station and set software and / or robot options.

0260Referring to FIG. 14, in some embodiments, the robot 100 detects, tracks, and follows human 1330 while performing human follow-up action 512b. Since the robot 100 can pan and tilt the head 160 using the neck 150, the robot 100 can orient the second 3D image sensor 450b in a direction in which the corresponding field of view 452 is maintained by the person 1330. In addition, the head 160 can move relatively faster than the base 120 (using, for example, the drive system 200), and the head 160 (and the associated second 3D image sensor 450b) is the robot 100. You can track the person 1330 faster than if you rotate it on the fly. Robot 100 follows human 1330 in threshold range D_FCan move towards a person 1330 to stay within (eg, corresponding to the sensor field of view). In some examples, the robot 100 looks back towards the person / user 1330 while tracking the person 1330. Robot 100 can use speed and / or waypoint commands to track person 1330.

0261Additional details and features relating to human recognition and human follow-up can be found in the PCT application serial number PCT / US11 / 35488 filed May 6, 2011, which application is incorporated herein by reference in its entirety. Is done.

0262Figures 15A and 15B show another 3D map 622c and 2D map 622a that can be displayed within the floor plan map window 620, which contains hypertag 1310 associated with various information to identify the robot. It can be used to autonomously navigate to a destination. Hypertag 1310 can contain information about various locations or information about the patient. The user can add a label 1502 or a mark 1504, a personal memo, a shared memo, a sketch, a figure, and the like. The robot position 1510 can also be identified. The user can specify a robot destination 1512, such as a nurse station. Robot 100 can autonomously navigate to a designated robot destination.

0263The telepresence software application 601 can display information indicating the physical area of interest on the remote video view 612 and / or map 622. For example, a small arrow with a balloon reading "Pharma" can indicate the location of a pharmacy. Such a callout can include a tag. For example, this tag may have tag coordinates indicating where the word "Pharma" should be displayed, tag information such as related information related to a pharmacy, and a two-dimensional and / or three-dimensional graphic display of the word "Pharma". Tag annotations can be included. In some examples, the user can determine what information is available about a nearby room by hovering or gesturing the mouse over the area. Based on this information, the user can immediately choose to go to a destination (eg, a pharmacy) by selecting the robot destination 618 in the remote navigation view 612b (FIG. 12).

0264For example, according to one example, the robot or remote terminal can read the tag coordinates corresponding to the tags associated with the robot map. The robot position can be used to identify tags that are very close to the robot. Tags in the robot's field of view can be identified using the orientation of the robot's head (top). The robot and / or remote terminal then computes a set of coordinates for all pixels on the video screen and a perspective based on the perspective given by the robot's position and the robot's current head orientation (pan and / or tilt). Render the tag annotation associated with each tag in the direction (line of sight). According to some embodiments, the Denabit-Hartenberg parameter (DH parameter) can be used as a frame of reference for the spatial coupling between the video feed and the plan map.

0265To explain again with reference to FIG. 8E, the tagging view 660 of user interface 605 puts a tag on the floor plan map 810 to show the user where they are interested and / or an obstacle on the floor plan map 810. , Allows you to mark information such as preferred robot travel routes. Also referring to FIG. 13, the user can place the hypertag 1310 on the remote navigation view 612b to mark the location with context sensitive information. The map data source 1620 can store tags and hypertag information (eg, location, tag identification system, tag content) along with layout map and / or robot map information. The hypertags used herein can be embodied as tags, and data structures similar to those described herein can be used.

0266In addition to or instead of allowing the user to place tags 662 and hypertag 1310 in user interface 605, the user can enter user-specific hypertag 1310 during the operation of robot 100. The user can call a command that allows the insertion of hypertag 1310 at the robot's current position. Another command can allow the removal of hypertag 1310. Further, another user (eg, a nurse) can add a hypertag 1310 that can be shown to the user of the robot 100. The "nurse map application" can display a top-down map or tagging view 660 that allows the installation of a temporary hypertag 1310 to identify rooms of interest to doctors who can log in immediately, for example. In addition, some hypertags 1310 are user-specific and / or time-specific. For example, indoor stroke patients can show signs of exacerbation. The nurse calls the "Nurse Map Application", finds the room on the map, and enters the hypertag 1310. The nurse can enter the hypertag name = "stroke patient worsening", user-specific = "doctor Reynolds", and duration = 1 hour in the hypertag. Therefore, doctor Reynolds can log in within the next hour and see the hypertag 1310 associated with the patient's room on the map, which additionally indicates "stroke patient exacerbation." While approaching the building, doctors can also see a pop-up of Hypertag 1310 in the video stream pointing to the room labeled "Stroke Patient Deterioration." Other doctors cannot see these labels, only doctor Reynolds can see them during the first hour.

0267Physicians may set temporary bookmarks and reminder hypertags 1310 on local or remote station interfaces 606,608 to assist in their work planning. In some examples, the doctor can assign numbers to some patient rooms at the beginning of the work. Then during the activity, the doctor can see the numbers on the displayed map 622 and in the pop-up hypertag 1310 to remember the turn to visit patient 614. Physicians can add notes that can be seen through activity reminders or on the next return, such as "return at the end of work, or" write a prescription "or" retest at 4 pm "for a patient. ..

0268In addition, the "smart" hypertag 1310 can be displayed automatically. For example, a nurse can enter a picture of an inpatient 614 into a database that is cross-referenced with electronic medical records (eg, stored in local storage and / or cloud storage 722). The telepresence software application 601 can perform facial recognition algorithms on the video stream captured by the robot camera 320 to identify patients 614,1330 and cross-reference the identified patients with the database. During patient face recognition, the telepresence software application 601 can automatically retrieve and display the patient's electronic medical records.

0269Referring to FIG. 14, in some embodiments, each patient 614,1330 receives, for example, a radio frequency identification (RFID) chip 497 worn on a wristband. Robot 100 can include an RFID reader 498 that communicates with controller 500 as part of its sensor system 400 to recognize nearby patients via RFID chips. The telepresence software system 601 displays the corresponding hypertag when the patient enters the RFID range of Robot 100 (eg 6 feet). Since RFID is not directional, Hypertag 1310 appears to float in the air. Another hybrid method uses computer vision technology to confirm by face recognition that patients 614,1330 are within the field of view 322 of Robot 100, then confirm that RFID matching belongs to this patient, patient 614. Hypertag 1310 on, 1330 can be positioned.

0270Referring to FIG. 16A-16D, in some embodiments, the robot system 1600 includes one or more telepresence robots 100 communicating with the bridge 602, which is the local robot endpoint server 604a and the remote endpoint server 604b. Communicate with (eg, cloud computing service 720 (Figure 7)). The robot endpoint server 604a communicates with the local technician computing device 606 and the remote endpoint server 604b communicates with the remote operator computing device 608. The robot system 1600 also includes one or more data sources 1610 that store sensor data and / or user interaction data received from the local sensor system 400, such as information obtained from the user via webbad 310 and / or user interface 605. Can include. In the illustrated example, the robot system 1600 includes at least one data source 1610a for storing sensor data and at least one head data source 1610b for storing user interaction data. These data sources 1610 can reside on Robot 100, Cloud Storage 722 (Figure 7), Local Robot Endpoint Server 604a and / or Remote Endpoint Server 604b.

0271Map data source 1620, eg robot 100, cloud storage 722 (Figure 7), database stored in local robot endpoint server 604a and / or remote endpoint server 604b, is plan view map 810, robot map 820, tag information 662 , And / or the information of hypertag 1310 can be stored. The map data source 1620 can be a single database or a combination of multiple data sources 1610, such as a robot sensor data source 1610a and a head data source 1610b. Telepresence software application 601 and / or robot 100 (eg, controller 500) performs map data source 1620 for performing real-time or offline concordance processing, providing user interface feedback, performing navigation routines, rendering map 622, and so on. Can be accessed.

0272In some embodiments, the control system 510 running on the controller 500 acts by accessing one or more of data sources 1610 and / or map data sources 1620, such as robot sensor data source 1610a, head data source 1610b. Issue an event recognizable by system 510a. In response to the issued event, the action system 510a can perform one or more actions 512 that control the selection of commands executed by the remote control arbiter 560 on the robot resource 530 (Figure 5). In the example shown in Figure 16C, the robot control system 510 communicates with the map data source 1620 to access the Concordance Matrix / Database, which uses concordance process information such as real-time sensor / flag data 1622a, operator commands 1622b, and local perception. Spatial data 1622c (eg volumetric point cloud data received from 3D image sensor 450), occupied bitmap data 1622d (eg robot map 820), floor layout data 1622e (eg plan map 810), and end user A tag table 1622f (eg, stores x, y, z coordinates and tag fields) and / or a robot action tag table (eg, stores x, y, z coordinates and tag fields) can be stored. Also referring to FIG. 5, the action 512 of the action system 510a is a possible result of the robot action based on the issued event, eg, the sensor event from the sensor system 400 and the installation tags 662, stored in the tag tables 1622f, 1622g. Tag events issued by 1310 (eg, which can mimic sensor events) can be evaluated. Therefore, the motion selection engine 580 can select an executable robot motion that has the best result based on the behavior evaluation. As a result, the robot 100 can operate autonomously in consideration of the tags 662,1310 received by the telepresence software application 601.

0273Referring again to the example of the slope shown in FIG. 11C, when the robot 100 approaches the slope 1122, the robot control system 510 perceives the slope 1122 as an obstacle 1120 based on the sensor signal received from the sensor system 400. To distinguish between the perceived obstacle 1120a and the actual obstacle 1120b, the control system 510 needs access to a common data source, such as the map data source 1620, which stores robot data and user data. Using the map data source 1620, control system 510 determines that the detected slope 1122 is the perceived obstacle 1120a rather than the actual obstacle 1220b. Further, the control system 510 is for receiving user input as to whether the user perceives slope 1122 as an actual obstacle 1120b and / or for receiving alternative robot path 652 and / or for receiving alternative robot destination. In addition, it can communicate with the telepresence software application 601. Telepresence software application 601 uses map device source 1620 to distort between 2D maps 622a, 810 and 3D maps 622C to provide a hybrid map 622b (Figure 9B), live in remote view 612. The distortion between the video feed and the 2D and / or 3D maps 622a, 622c can be determined. In addition, the telepresence software application 601 can use the map device source 1620 to display the look-ahead view 612a in the plan map window 620 (FIG. 10C).

0274Referring again to FIG. 12, in an additional embodiment, the telepresence software application when the user selects a robot path on one of the 2D map 622a, the hybrid map 622b, the 3D map 622c and the remote view 610. The 601 uses the map data source 1620 to determine the distortion between one of the maps 622 and the remote view 612, attaches that distortion to the selected robot path, and executes the drive command to the destination 618. A sequence of corresponding robot path map coordinates can be determined on the robot map 820 used in the robot controller 500. In addition, the telepresence software application 601 can apply the determined distortion to determine a sequence of corresponding robot path coordinates for displaying the robot path 652 on any of the maps 622 and the remote view 612.

0275Therefore, the term "distortion" as used herein is broadly understood to relate to determining the difference in transformations from one coordinate system to another, including spatial coordinate errors, transformations between coordinate systems of different dimensions. Should be. For example, a robot and / or a remote terminal is between a 2D plan map and a 2D map generated at least partially by the robot, such as one generated using various robot sensors or laser scans. The distortion can be determined. In addition, the robot and / or remote terminal can determine the distortion between the 3D map or video feed and the 2D plan map. In addition, the distortion determination relates to coordinate transformations between first-person, third-person, floor plan, mixed map views, and / or between any two different coordinate systems or between different perspectives within the same coordinate system. It can be.

0276Referring again to FIG. 13, in some embodiments, when the user places the tag 662 or hypertag 1310 on top of the floor plan map 810 displayed as map 622 in the map window, the telepresence software application 601 maps. Determines the user-selected position on the electronic display displaying 622 and overlays tags 662,1310 and associated annotations on map 622. Telepresence software application 601 also determines the distortion between the plan map 810 and the remote view 610 (ie, the first-person video captured by the robot camera 320), and this distortion is determined by the tags 662,1310 on the plan map 810. It can be applied to the coordinates to determine the corresponding video coordinates of the remote view 610. Tag annotations associated with tags 662,1310 are stored in the map device source 1620 and can be displayed on the remote view 610 by the telepresence software application 601 with the determined video coordinates.

0277Referring to FIG. 17, in some embodiments, the user interface 605 is an enlarged overlay that makes the position of the head with respect to the base of the robot visible to the user, for example in the remote video feed window 610 and / or the map window 620. Provides 1710. The magnified overlay 1710 makes the robot sensor system 400's current field of view 322,442,452 (indicated by 1720 in the illustrated example) visible to the user for a total 360 degree field of view. This allows the user to select rotation (eg, head 160 and / or base 120) outside the current field of view 322,442,452.

0278The user can click within the zone defined by the first and second rings 1711 and 1724 to rotate the virtual head portion 1728 to the relevant point. When the user rotates the virtual head unit 1728, the robot head unit 160 moves in real time, and the telepresence software application 601 displays the live video feed from the robot camera 320 in the remote view 612 of the remote video feed window 610 in real time. Can be updated to. In the illustrated example, the enlarged overlay 1710 is attached to the virtual base portion 1726 corresponding to the robot base portion and the robot head portion 160 arranged at an angle and / or orientation corresponding to the current posture of the robot 100 with respect to the virtual base portion 1726. It has a corresponding virtual head unit 1728. In some embodiments, one of the virtual base section 1726 and the virtual head section 1728 is fixed, while the other is movable relative to the fixed one.

0279When the user clicks in the zone defined by the 1st and 2nd rings 1722 and 1724 to rotate the head 1728 out of the current field of view 1720, the robot 100 will move the head 160 and / or the base 120 to the user. Rotate to accomplish the command. In some examples, the head portion 160 can be rotated, then the base portion 120 can be rotated, and then the head portion 160 can be moved to the center position according to a user command. Such a change in position becomes a problem the next time the user attempts to reposition the robot based on its previous rotation. To alleviate this, certain embodiments can use a system that reduces the requirement for base rotation to accept head rotation. For example, the counter can be started when the virtual head unit 1728 is rotated by a certain angle. If the robot head 160 stays at that angle for a specified period of time, the system will slowly rotate the base 120 and at the same time rotate the head 160 in the opposite direction at the same speed in order to center the head 160 with respect to the base 120. Can be rotated with. As a result, the current object is kept in the field of view, the head portion 160 and the base portion 120 are aligned, and the forward reference system is determined by the direction in which the user is looking. Further, if the user wants to continue looking in that direction, the entire panning range of the head 160 is available.

0280FIG. 18 shows an exemplary sequence of robot events corresponding to user commands, for example in the telepresence software application 601. In the initial state, the robot 100 may receive a drive command to move from one position to another. This command can be an operator start, an action start (eg, an action performed by the controller's control system) or a planner start (eg, a pre-planned task or routine). In this example, the command can include a new direction of travel that moves in the opposite direction to the new direction. In response to this command, the robot 100 can turn its head 160 towards the panning limit (left or right). After reaching the panning limit, the robot 100 rotates the base 120 (holonomically in place) to allow the head 160 to move in order to rotate the head in a new direction of travel. The "panning limit" can be said to be the point where the upper part of the robot cannot physically rotate any more with respect to the lower part of the robot, the upper part is out of alignment with the lower part by a predetermined frequency, and / Alternatively, the "panning limit" can be a function of the frequency at which the top is misaligned with respect to the bottom and the length of time the top is misaligned with respect to the bottom.

0281In some embodiments, the robot 100 keeps rotating the base 120, so that the forward drive direction F coincides with the new direction of travel, so that the head 160 provides relatively equal left / right panning capabilities. When the robot 100 rotates the base portion, the head portion 160 can be rotated in the traveling direction at the same time, and if necessary, the visual fields 322,452 of the sensors 320,450,450b of the head portion 160 can be rotated so as to point to the new traveling direction. .. In some embodiments, the robot 100 rotates the base 120 and the head 160 together so that the head 160 points in a new direction of travel relatively faster. If the base 120 rotates too much, the head 160 can counterpan to restore alignment.

0282FIG. 19 shows an exemplary remote view 612 that overlays the screen indicator 1910 on a remote video feed received by the telepresence software application 601 from the robot 100. The screen indicator 1910 can be displayed near the mouse cursor 1912 to indicate the current head movement range. When the user moves the mouse cursor 1912 to the left or right side of the remote video view 612 (intended to click to move the head to its pointing point), the on-screen indicator 1910 moves over the cursor 1912. It can be displayed to indicate how long the movement of the head portion is maintained in that direction (for example, how much the remaining motion range of the head portion 160 can be used).

0283Highlight Box 1920 highlights the area of interest within Remote Video View 612. The user can drag and drop the box onto the screen and / or click to open the box around the area of interest to open the area of interest on a portion of the remote video view 612. A highlight box 1920 can be generated around. In response, the telepresence software application 601 can move the robot head 160 to the center of the highlight box 1920. In addition, the camera 320 can be zoomed in to match the dimensions of the highlight box 1920.

0284Referring to FIGS. 20A-20B, in some embodiments, if the robot 100 suddenly loses communication connectivity (eg, loss of radio signal), the robot 100 can stop or continue traveling to its destination. .. Communication may be interrupted as the telepresence robot moves through the environment, for example because the robot 100 moves transitions between various wireless access points and / or as a result of weak signal strength. This is because the data transmission is interrupted. By continuing autonomous navigation, communication can be restored by the time the robot reaches the desired destination.

0285When Robot 100 experiences a loss of communication connectivity, Robot 100 determines the last reliable position / attitude (stored locally by Controller 500) and / or now to continue sailing to the destination. The position / orientation given (based on the robot sensor system 400) can be referenced. If the robot route is a planned route, the robot 100 restarts the planned route to the destination. On the other hand, when the user remotely controls the robot 100 to an end, the robot 100 can follow the planned route to the nearest / last reliable position where there is communication connectivity (radio frequency and / or wireless). Alternatively, the robot 100 can travel along the shortest path (ie, the new path) to the nearest / last reliable position with communication connectivity.

0286After reaching the closest / last reliable position, Robot 100 determines if the communication connectivity has been reconfigured and, if so, whether the destination has been reached. If the communication connectivity is not reconfigured, the robot 100 moves to the next reliable position stored by the controller 500 in synchronization with its video recording (and any other sensor data in the sensor system 400). .. Furthermore, if the destination has not been reached but the communication connectivity has been reconfigured, the controller 500 will take evacuation port calls, which will continue to record sensor data and the robot side user interface (Webpad 310). Above) should indicate that the session has ended due to a loss of communication connectivity. Robot 100 can improve connectivity recovery rates by re-accessing and / or performing route planning to move to the last reliable location (using ODOA). Robot 100 can in some cases move its antennas 490a, 490b to obtain better communication reception. Robot 100 can use mobile ad hoc networks (MANETs), which are self-configuring infrastructureless networks of mobile devices connected by wireless links.

0287In some examples, Robot 100 can improve the integrity / accuracy of Robot Map 820 by marking locations where communication has been lost and where communication has been reconfigured. Robot 100 can use waypoint navigation to move to areas with known connectivity (eg WLAN warm zones or high signal zones). A waypoint is a set of coordinates that identifies a point in physical space. Robot 100 can use the waypoints set in robot map 820 to reach the destination.

0288Additional evacuation porting measures can include planning a route to the nearest minimum traffic area based on Robot Map 820, or moving to the nearest charging / docking station. In some examples, the robot 100 moves towards another robot 100 closest to it, which can have multiple antennas 490a, 490b for multi-input and multi-output (MIMO) to act as a Wi-Fi bridge 602. can do.

0289Various embodiments of the systems and techniques described herein are implemented with display electronics, integrated circuits, specially designed application specific integrated circuits (ASICs), computer hardware, firmware, and / or combinations thereof. it can. These various embodiments may be at least one programmable processor (dedicated or general purpose) coupled to receive and transmit data and instructions from and to a storage system, at least one input device and at least one output device. ) Can be implemented in one or more computer programs that can be run and / or interpreted on a programmable system.

0290These computer programs (also known as programs, software, applications or code) include machine instructions for programmable processors and are implemented in high-level procedures and / or object-oriented programming languages and / or assembly / machine languages. it can. As used herein, "machine readable medium" and "computer readable medium" are any computer program products, devices and / or used to provide machine instructions or data to a programmable processor. A device (eg, a magnetic disk, optical disk, memory, programming logic device (PLD)) that includes a machine-readable medium that receives a machine command as a machine-readable signal. A "machine-readable signal" is a machine command. And / or any signal used to provide data to a programmable processor.

0291The gist and functional operation embodiments described herein can be implemented in digital electronic circuits or computer software or hardware, including the structures disclosed herein and their equivalent structures or one or more combinations thereof. The examples of the gist described herein are for performing on or controlling the operation of one or more computer program products, i.e., encoded on a computer-readable medium, a data processor. It can be implemented as one or more computer program instruction modules. The computer-readable medium can be a machine-readable storage device, a machine-readable storage device, a memory device, or a composition that affects a machine-readable propagating signal. "Device processing device" includes any device, device and machine that processes data, including, for example, a programmable processor, a computer, or a plurality of programmable or computers. In addition to the hardware, the device can include code that creates an execution environment for the computer program in question, such as processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof. .. The propagating signal is an artificially generated signal, such as a machine-generated electrical signal, optical signal or electromagnetic signal generated to encode information to be transmitted to a suitable receiver.

0292Computer programs (also known as programs, software, applications, scripts or code) can be written in any programming language, such as a compiled or interpreted language, and deployed in any form. Can be deployed, for example, as a stand-alone program or as a module, component, subroutine or other unit suitable for a computing environment. Computer programs do not necessarily have to correspond to files in the file system. A program may be part of a file that holds other programs or data (eg, one or more scripts stored in a markup language document), a single file dedicated to the program in question, or multiple collaborative files (eg, multiple collaborative files). It can be stored in one or more modules, subprograms, or files that store parts of code). Computer programs can be deployed to run on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

0293The processes and logical flows described herein may be performed by one or more programmable processors running one or more computer programs in order to perform various functions by processing input data and producing outputs. it can. This process and logic flow can also be performed by a dedicated logic circuit, such as a field programmable gate array (FPGA) or ASIC, and the device can also be implemented as such a dedicated logic circuit.

0294Suitable processors for running computer programs can include, for example, both general purpose and dedicated microprocessors, and any one or more of digital computers of any kind. Generally, the processor receives instructions and data from read-only memory and / or random access memory. An essential element of a computer is a processor that executes instructions and one or more memory devices that store instructions and data. In general, computers also include one or more large capacity data storage devices such as magnetic disks, magnetic optical disks or optical disks, and are operably coupled to receive or transfer data from them. However, the computer need not include such a device. In addition, the computer can be incorporated into another device, such as a mobile phone, a personal digital assistant (PDA), a portable audio player, or a Global Positioning System (GPS) receiver, to name a few. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, for example semiconductor memory devices such as EPROM, EEPROM and flash memory devices. , Internal hard disks or magnetic disks such as removable disks, magnetic optical disks, and CDROM and DVD-ROM disks. The processor and memory can be complemented by a dedicated logic circuit and can also be incorporated into this circuit.

0295Embodiments of the gist described herein can be implemented in computing systems, which are described in back-end components such as data servers and middleware components such as application servers and front-end components such as those herein. It may include a client computer having a graphic user interface or web browser that interacts with an embodiment of the gist, or may include any combination of one or more such backend, middleware or frontend components. The components of this system can be interconnected by digital digital data communications of any form or medium, such as communication networks. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

0296Computing systems can include clients and servers. Clients and servers are generally remote to each other and generally interact over a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship with each other.

0297Although the specification includes many features, these features are described as features specific to a particular embodiment of the invention, limiting the scope of the invention or claims. Should not be interpreted as what it does. Some of the features described herein in relation to the individual embodiments can also be implemented in combination in a single embodiment. Conversely, the various features described in connection with a single embodiment can also be implemented individually or in any suitable subcombination in multiple embodiments. In addition, some features are described above as acting in several combinations and may first be stated in the claims, but in some cases one or more features from the combinations described in the claims. Can be removed from the combination, and the combination described in the claims may be directed to a partial combination or a variant of the partial combination.

0298Similarly, the operations are shown in the figure in a particular order, which means that these operations must be performed in the specific order shown or in a sequential order, and in order to achieve the desired result. It should not be understood that all the operations shown are required to be performed. In certain situations, multitasking and parallel processing are advantageous. Furthermore, the separation of the various system components in the above embodiments should not be construed as requiring such separation in all embodiments, and the program components and systems described are in a single software product. It can be integrated and packaged into a large number of software products.

0299A number of embodiments have been described. Nevertheless, it should be understood that various changes can be made without departing from the preliminary scope of the control system of the present disclosure. Therefore, other embodiments are also included in the claims below. For example, the operations described in the claims can be performed in different orders to achieve the desired result.