Interfacing with a mobile telepresence robot

Communicate with a mobile remote robot

[0001] The patent application for the present invention is a divisional application. The original application number is 201280006852.0, the filing date is 2012-127, and the name of the invention is: communicating with a mobile remote robot.

Technical field

[0002] The present invention relates to a movable remote robot.

Background technique

[0003] A robot generally refers to an electromechanical system machine manipulated by a computer or electronic program. Remote robots can move around their environment and are not fixed in a physical location. One example of the current widespread use of remote robots is the automated guided vehicle (AGV). Auto-guided vehicles usually refer to remote robots that navigate by following marks or wires on the ground, or using imaging systems or lasers. Long-range robots are used in industrial, military and security environments. They can also be used as consumer products for entertainment or to perform certain tasks like home assistants.

Summary of the invention

[0004] One aspect of the present invention provides a remote robot system including a local terminal and a remote control remote robot. The local terminal may include an electronic display, a processor, and a memory connected with the processor, and the memory contains instructions that can be executed by the processor. The executable instruction may be designed to cause the processor to read at least a part of the plan view, which represents the robot passable area of the robot operating surface; read at least one of a plurality of marks, the plurality of marks Each of them includes marker coordinates and marker information describing the relevant positioning of the marker, and the marker information may include marker annotations; receiving video input from a remote-controlled remote robot imaging system; receiving positioning information; displaying information from the remote control The video input of the remote robot imaging system; displaying the plane view so that the current position of the remote robot is indicated on the plane view; by using the marker coordinates at least in the plane view and the video input A recurrence of the mark annotation on which the at least one mark is displayed; and one or more commands are transmitted to the remote control remote robot.

[0005] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to determine the plan view and the video input received from the remote control remote robot imaging system (For example, the coordinate conversion between a two-dimensional coordinate system and a three-dimensional coordinate system); applying the distortion to the mark coordinates of the at least one mark to determine that the at least one mark is relative to the The corresponding video coordinates and projection data of the positioning and projection of the video input;

[0006] And use the mark video coordinates to display a three-dimensional recurrence of the mark annotation covering the at least one mark of the video input.

[0007] In some embodiments, based on the current position of the remote control remote robot and the projection of at least one marker relative to the video input, the three-dimensional reproduction of the marker annotation may be dynamically provided again.

[0008] In some embodiments, the three-dimensional rendition of the mark annotation may cover the video input related to the target detected in the video input.

[0009] In some embodiments, the three-dimensional recurrence of the mark annotation may be detected along the video input

CN 104898652 Β

Wall covering.

[0010] In some embodiments, the tag information of the at least one tag includes a remote robot motion correction factor, and the robot motion correction factor may be designed to provide an execution instruction for the remote robot control system to respond The first action is performed when the remote robot is located within a predetermined range of the mark coordinates of the at least one mark.

[0011] In some embodiments, when the remote robot is within a predetermined range of the marker coordinates of the at least one marker, the instructions that can be executed by the processor are further designed to cause the processing The controller transmits the execution instructions to the control system of the remote robot

[0012] In some embodiments, the robot motion correction factor further includes an instruction related to one of time and positioning on the plan view, so that the control system of the remote robot executes the first motion .

[0013] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to receive a series of coordinates that form a path relative to the plane view, and the remote control remote robot has moved along Travel along the path; store the series of coordinates forming the path as path markers including marker coordinates and marker information, the marker information may include marker annotations; when the remote control remote robot reaches within a predetermined distance of the marker coordinates When, read the path mark; and use the mark coordinates to display the recurrence of the mark annotation of the path mark on at least one of the plane view and the video input.

[0014] In some embodiments, the local terminal of the remote robot system further includes at least one user input device and the series of coordinates that can be provided by the user input device to form the path.

[0015] In some embodiments, the series of coordinates forming the path may be provided by the remote control remote robot.

[0016] In some embodiments, the remote robot system further includes a communication system designed to facilitate communication between the local terminal of the remote robot system and the remote control remote robot.

[0017] In some embodiments, the local terminal further includes at least one user input device, and the user input device may be designed to enable the user to provide a view in the plane and the imaging system from the remote control remote robot At least one of the above indicates an expected destination of the remote control remote robot; and the command transmitted to the remote control remote robot includes the expected destination.

[0018] In some embodiments, the series of coordinates forming the robot path may be based at least in part on marker information associated with the at least one marker.

[0019] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to determine a series of coordinates relative to the plan view, so as to be in the current position of the remote control remote robot. A robot path is generated between the remote control remote robot and the expected destination, and the command transmitted to the remote control remote robot includes the series of coordinates forming the robot path.

[0020] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to display a series of coordinates that form a robot path that covers the plan view.

[0021] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to determine the plan view and the video input received from the remote-controlled remote robot imaging system (For example, the coordinate conversion between a two-dimensional coordinate system and a three-dimensional coordinate system); the distortion is applied to the series of coordinates forming the robot path to determine that the series of coordinates are relative to the The corresponding video coordinates and projection data of the positioning and projection of the video input; and display the series of seats that form the robot path covering the video input

CN 104898652 Β

Three-dimensional reproduction of the target.

[0022] In some embodiments, with respect to the ground detected in the video input, the three-dimensional reproduction of the series of coordinates forming the robot path may cover the video input.

[0023] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to receive a series of coordinates from the remote control remote robot relative to the plane view, the series The coordinates form a robot path between the current position of the remote control remote robot and the expected destination of the remote control remote robot; and display a series of coordinates forming the robot path covering the plane view.

[0024] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to determine the plan view and the video input received from the remote-controlled remote robot imaging system (For example, the coordinate conversion between a two-dimensional coordinate system and a three-dimensional coordinate system); the distortion is applied to the series of coordinates forming the robot path to determine that the series of coordinates are relative to the The corresponding video coordinates and projection data of the positioning and projection of the video input; and display the three-dimensional reproduction of the series of coordinates that form the robot path covering the video input.

[0025] In some embodiments, with respect to the ground detected in the video input, the three-dimensional reproduction of the series of coordinates forming the robot path may cover the video input.

[0026] In some embodiments, the tag information includes information about one of the following: the availability of wireless communication signals, the speed at which the remote-controlled remote robot should travel, the location of the target point, the location of a person, and the location of a docking station , The positioning of the rest area, the positioning of the glass wall, the positioning of the ramp, the positioning of the target, the best route to pass through confined areas, the best route to pass through crowded areas, and the actions that the remote control remote robot should perform.

[0027] In some embodiments, the marking information may relate to a position, a path, and/or a volume, and the control system may be designed to perform actions related to the position, the path, and/or the volume.

[0028] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to receive the coordinates on the plane view of the obstacle detected by the sensor system of the remote control remote robot .

[0029] In some embodiments, the plan view and the plurality of markers are stored remotely.

[0030] In some embodiments, the plan view and the plurality of marks are stored in the remote control remote robot.

[0031] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to determine the plan view and the video input received from the remote control remote robot imaging system Distortion (for example, coordinate conversion between a two-dimensional coordinate system and a three-dimensional coordinate system); and generate a hybrid map view including a hybrid view of the plane view and the video input from the remote-controlled remote robot imaging system.

[0032] In some embodiments, wherein the hybrid map view includes a three-dimensional representation of a plan view overlaying the video input.

[0033] In some embodiments, the local terminal of the remote robot system further includes at least one user input device, and the instruction design that can be executed by the processor is further used to cause the processor to input via at least one The device receives the provided requirements for predicting the actual positioning of the remote control remote robot on the plane view; determines the distortion between the plane view and the video input received from the remote control remote robot imaging system (for example, two-dimensional coordinates) Coordinate conversion between the system and the three-dimensional coordinate system); generate actual three-dimensional video input based on the actual positioning of the remote control remote robot; and display the actual three-dimensional video input based on the actual location of the remote control remote robot.

[0034] In some embodiments, the mark information of the at least one mark includes a set of relative

CN 104898652 Β

Based on the coordinates of the plan view, and the mark annotation of the at least one mark can be designed to indicate the existence of a protected area.

[0035] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to receive a request for generating a new mark; and associate the mark coordinates describing the relative positioning of the new mark with the mark Information, the mark information may include a mark annotation of a new mark; and use the mark coordinates to display a recurrence of the mark annotation of the new mark on at least one of the plane view and the video input.

[0036] In some embodiments, the request for generating a new mark may be generated by the remote control remote robot.

[0037] In some embodiments, the requirement to generate a new marker can be automatically generated based on the detected target in the video input.

[0038] In some embodiments, the new marker may be a temporary marker designed to terminate once the detected target no longer exists in the video input.

[0039] In some embodiments, the target may be a human, and the newly marked marking information includes identification information related to the human.

[0040] In some embodiments, the target may be a person, and the newly tagged tag information includes potential actions that the remote control remote robot can perform on the person.

[0041] In some embodiments, the request for generating a new mark may be generated by a user input device connected to a local terminal of the remote robot system.

[0042] In some embodiments, the requirement to generate a new mark is made relative to the video input.

[0043] In some embodiments, the requirement to generate a new mark proceeds relative to the plan view.

[0044] In some embodiments, the request to generate a new mark is performed relative to the current position of the remote control remote robot.

[0045] In some embodiments, the tag information includes information about one of the following: the availability of wireless communication signals, the speed at which the remote-controlled remote robot should travel, the location of the target point, the location of a person, and the location of a docking station , The positioning of the rest area, the positioning of the glass wall, the positioning of the ramp, the positioning of the target, the best route to pass through confined areas, the best route to pass through crowded areas, and the actions that the remote control remote robot should perform.

[0046] In other embodiments, the remote robot can communicate with the remote control terminal. The remote robot may include: a driving system designed to make the remote robot move according to driving instructions; a control system communicating with the driving system, the control system being designed to generate The driving instructions for the movement of the robot; the imaging system connected with the control system; the drawing module connected with the imaging system, the drawing module is designed to access a map data source, and the map data source includes a representation of the operating surface of the robot A plan view of the robot's passable area; and a plurality of markers, each of which is a data structure including marker coordinates and marker information describing the relative positioning of the marker; a positioning system connected to the control system, the positioning system design For providing the current position relative to the plan view; a mark identification system, the mark identification system is designed to identify at least one mark related to the navigation path of the remote robot; and a communication system, the communication system is designed for To facilitate the communication between the control system and the remote control terminal; and wherein the control system is designed to perform actions based on the identified mark, and the mark information of the identified mark includes a remote robot motion correction factor.

[0047] In some embodiments, the mark information of the authenticated mark includes an instruction regarding one of time and positioning on a plan view that the control system should perform the action.

[0048] In some embodiments, the control system may be designed to pass video input from the imaging system through all

CN 104898652 Β

The communication system is transmitted to the remote control terminal, and the control system may be designed to receive an indication of an expected destination on the plan view from the remote control terminal via the communication system.

[0049] In some embodiments, the remote robot may further include: a plurality of sensors designed to identify obstacles in the vicinity of the remote robot; and communicating with the plurality of sensors and The obstacle avoidance system connected with the control system, wherein the control system may be further designed to generate another driving command to avoid obstacles in the vicinity of the remote robot.

[0050] In some embodiments, the plurality of sensors includes at least one of a progress sensor, a contact sensor, a distance measurement sensor, and a three-dimensional image sensor.

[0051] In some embodiments, the plurality of sensors may include a three-dimensional image sensor forming a point cloud including a three-dimensional occupation area of an obstacle, and the driving instruction may be designed to avoid the obstacle The three-dimensional occupancy area of the object.

[0052] In some embodiments, the remote robot may further include: a map generation system connected with the control system, the map generation system is designed to automatically generate a plane view of the robot operating surface, wherein the control system A driving instruction that causes the remote robot to move on the entire robot operating surface and obtain a plurality of measurement results is generated, and the map generation system uses the plurality of measurement results to generate a plan view.

[0053] In some embodiments, the remote robot may further include a navigation system, the navigation system is designed to generate a navigation path, the navigation path includes from the current position on the plan view to the plan view The series coordinates of the intended destination.

[0054] In some embodiments, the remote robot may transmit coordinates relative to the plane view of the detected obstacle to the remote control terminal via the communication system.

[0055] In some embodiments, the series of coordinates that form the passage path may be based at least in part on marking information associated with the authenticated marking.

[0056] In some embodiments, the navigation system is designed to generate a travel path by selecting a travel path from a plurality of potential travel paths, and combine the mark on the travel path of the remote robot with the plurality of potential travel paths. The travel path is associated, and the navigation system is designed to select the travel path based at least in part on the identified relevant markers.

[0057] In some embodiments, the series of coordinates forming the passage path are transmitted to the remote control terminal via the communication system.

[0058] In some embodiments, the remote robot may be designed to generate a new mark using a series of coordinates forming the passage path, so that the new mark includes the series of coordinates and marking information related to the passage path And a mark annotation related to the passage.

[0059] In some embodiments, the tag information of each of the plurality of tags includes information about one of the following: the availability of wireless communication signals, the speed at which the remote-controlled remote robot should travel, the location of the target point, the human Positioning, positioning of docking stations, positioning of rest areas, positioning of glass walls, positioning of ramps, positioning of targets, best routes for passing confined areas, best routes for passing crowded areas, and actions to be performed by remote control remote robots .

[0060] In some embodiments, the control system may be further designed to receive a travel path from the current position on the plan view to the intended destination on the plan view, and the control system may be further designed for A driving instruction that causes the driving system to move the remote robot to a desired terminal based on the passage path is generated.

[0061] In some embodiments, the communication system may be designed to detect the interruption of communication between the remote robot and the remote control terminal, and wherein the control system may be further designed to continue to generate and cause the remote robot to be in place. Narrate

CN 104898652 Β

A drive command that automatically moves to the expected destination during a communication interruption.

[0062] In some embodiments, the map data source can be stored remotely, so that the measurement module can be designed to access the map data via the traffic system.

[0063] In some embodiments, the map data source can be stored in the remote robot, so that the measurement module can be designed to access internal map data sources.

[0064] In some embodiments, the internal map data source may be synchronized with a remotely stored map data source.

[0065] In some embodiments, the positioning system may be further designed to improve the robot posture relative to the plan view.

[0066] In some embodiments, the remote robot may be designed to generate a new mark through a process that describes the new mark relative to one of the plan view and the video input generated by the imaging system The relative positioning of the marker coordinates is associated; the marker information is associated with the new marker; and the marker annotation is associated with the new marker.

[0067] In some embodiments, the new marker may be generated in response to a remote robot detecting a target in the video input.

[0068] In some embodiments, the target may be a human, and the newly marked marking information includes identification information related to the human.

[0069] In some embodiments, the target may be a person, and the newly tagged tag information includes potential actions that the remote control remote robot can perform on the person.

[0070] In some embodiments, the tag information includes information about one of the following: the availability of wireless communication signals, the speed at which the remote-controlled remote robot should travel, the location of the target point, the location of a person, and the location of a docking station , The positioning of the rest area, the positioning of the glass wall, the positioning of the ramp, the positioning of the target, the best route to pass through confined areas, the best route to pass through crowded areas, and the actions that the remote control remote robot should perform.

[0071] In some embodiments, the remote robot may further include: an RFID reader communicating with the positioning system, wherein the positioning system associates multiple RFID chips with corresponding multiple coordinates on the plan view. And the positioning system may be designed to determine the current position of the remote robot based at least in part on the positioning of one or more RFID chips within the range of the RFID reader.

[0072] A variety of control methods can be used in current systems and methods. For example, the local terminal of the remote robot system may include: an electronic display; a processor communicating with the electronic display interface; a memory communicating with the processor, the memory including instructions that can be executed by the processor, the The instructions are designed to cause the processor to perform the following operations: read at least a part of the plan view, the at least part of the plan view represents the robot passable area of the robot operating surface; receive at the first projection from the remote control remote robot imaging system Video input;

[0073] Receive the current position relative to the plane view from the positioning system of the remote control remote robot; display the video input from the imaging system of the remote control remote robot; display the plane view so that the current position of the remote robot The position indication is on the plane view; the command is transmitted to the remote control remote robot; and the user input device communicated with the processor, the user input device is designed to enable the user to select movement for the remote control remote robot, so The selected movement includes selected from the destination of the remote control remote robot relative to the video input and relative to the plane view; and through one of at least four possible directions relative to the current position of the remote control remote robot Gradually advance the remote control remote robot.

[0074] In some embodiments, the selection of the movement includes selecting an alternative projection of the video input by selecting a point in the video input. This mode may be used to achieve the mid-range positioning within the view of the video input

CN 104898652 Β

from.

[0075] In some embodiments, the selection of the movement includes selecting an alternative projection of the video input by selecting a point in the video input. This mode should be able to be used to reach longer distances that are not positioned within the view of the video input (for example, under aisles, between rooms, etc.). In some embodiments, the choice of movement includes the use of a joystick or a manually controlled metal joystick. This mode should be able to be used for fine adjustment/fine adjustment, for example in a room in the immediate vicinity of a human/patient.

[0076] In some embodiments, the selection of the movement includes selecting an alternative projection of the video input by incrementally shifting or tilting the imaging system while the remote control remote robot maintains the current position.

[0077] In some embodiments, wherein the selection of the movement may involve rotating one of the lower part of the remote-controlled remote robot and the upper part of the remote-controlled remote robot.

[0078] In some embodiments, there will be a way to switch between various modes, such as a multi-mode user interface, one of which can choose to control the movement of the head/imaging system or remotely control the movement of the bottom/lower part of the robot.

[0079] In some implementations that select the movement control of the head/imaging system, there may be a selection of position-based box-zoom head motion via a mouse or a position-based box-zoom head motion via a mouse. Selection of velocity-based head motion.

[0080] In some implementations for selecting remote control of the bottom/lower part of the current robot, there may be a choice selected from one of the following: click-on-map, that is, up and down map view, And tap the target destination or selected from the list of destinations; (2) Click-on-video, that is, click-on-video, which can click the position-based control of positioning in the video; (3) Joystick or metal joystick, For example, the speed-based control of the mouse or the arrow clearly forward, left, right, etc.

[0081] In some embodiments, the required functionality/information that can be read by the user at any time when the robot base is moving includes: (1) Remote view, that is, the robot travels sufficiently to provide the user with a meaningful safe operation Visual information;

(2) For the monitoring control mode, there is a potential demand for human control capabilities that can cancel/suspend operations as needed.

[0082] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to receive a destination selection of the remote control robot from the user input device; A series of coordinates of the plan view to generate a passage path between the current position of the remote control remote robot and the selected destination of the remote control remote robot; and transmit a command to the remote control remote robot, the command command includes forming The series of coordinates of the transit path.

[0083] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to display a series of coordinates that form a traffic path covering the plan view.

[0084] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to perform the following operations: determining the plan view and the remote robot imaging system received Distortion between the video inputs (for example, coordinate conversion between a two-dimensional coordinate system and a three-dimensional coordinate system); applying the distortion to the series of coordinates forming the passage path to determine the series of coordinates Corresponding video coordinates and projection data relative to the positioning and projection of the video input; and displaying the three-dimensional reproduction of the series of coordinates that form the traffic path covering the video input.

[0085] In some embodiments, with respect to the ground detected in the video input, the three-dimensional reproduction of the series of coordinates forming the robot path may cover the video input.

[0086] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to receive the destination selection of the remote control robot from the user input device; Destination

CN 104898652 Β

The coordinates are transmitted to the remote control remote robot, and the destination coordinates correspond to the selected destination; a series of coordinates relative to the plane view is received from the navigation system of the remote control remote robot, and the series of coordinates are in the remote control A travel path is formed between the current position of the remote robot and the expected destination of the remote control remote robot; and a series of coordinates forming the travel path covering the plane view is displayed.

[0087] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to determine the plan view and the video input received from the remote-controlled remote robot imaging system (For example, the coordinate conversion between a two-dimensional coordinate system and a three-dimensional coordinate system); the distortion is applied to the series of coordinates forming the passage to determine that the series of coordinates are relative to the Corresponding video coordinates and projection data of the positioning and projection of the video input; and display the three-dimensional reproduction of the series of coordinates that form the traffic path covering the video input.

[0088] In some embodiments, with respect to the ground detected in the video input, the three-dimensional reproduction of the series of coordinates forming the passage path may cover the video input.

[0089] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to receive the coordinates on the plan view of the obstacle detected by the sensor system of the remote control remote robot .

[0090] In some embodiments, the plan view is stored remotely.

[0091] In some embodiments, the plan view is stored in the remote control remote robot.

[0092] In some embodiments, the instructions that can be executed by the processor are further designed to cause the processor to determine the plan view and the video input received from the remote-controlled remote robot imaging system Distortion (for example, coordinate conversion between a two-dimensional coordinate system and a three-dimensional coordinate system); and generate a hybrid map view that includes a hybrid view of the plane view and the video input from the remote control remote robot imaging system.

[0093] In some implementations, the hybrid map view includes a three-dimensional illustration that overlays a plan view of the video input.

[0094] In some embodiments, the instruction design that can be executed by the processor is further configured to cause the processor to receive via an input device the actual positioning of the remote control robot on the plan view provided by the processor. The requirements; determine the distortion between the plan view and the video input received from the remote control remote robot imaging system (for example, the coordinate conversion between a two-dimensional coordinate system and a three-dimensional coordinate system); generate based on the remote control remote robot The actual three-dimensional video input of the actual positioning; and the actual three-dimensional video input based on the actual positioning of the remote control remote robot is displayed.

[0095] In some embodiments, the robot may be designed to unfold and/or control the upper and lower parts independently so as to behave like a human. For example, the remote robot may include: an upper part; a lower part rotatably connected to the upper part; a driving system designed to move the remote robot according to driving instructions; a control system communicating with the driving system, the control system Designed to generate a driving command that causes the driving system to move the remote robot; a rotation system designed to make the robot travel from a first direction of travel to a second direction by independently rotating the upper and lower parts Direction rotation.

[0096] In some embodiments, the rotation system may be designed to rotate the robot toward the second direction of travel by turning the upper part of the robot toward the second direction of travel; detecting that the upper part of the robot has reached the The translation limit of the upper part of the robot relative to the lower part of the robot; starting at the translation limit of the upper part of the robot, the lower part of the robot is turned toward the second direction of travel; detecting that the upper part of the robot has reached the A second direction of travel; and while simultaneously turning the upper part of the robot in the reverse direction, continue to make the lower part of the robot face the

The second traveling direction rotates so that the upper part of the robot maintains the second traveling direction.

[0097] In some embodiments, the translation limit may be reached when the upper part cannot be physically rotated relative to the lower part of the robot.

[0098] In some embodiments, the translation limit may be reached when the upper part is offset by a predetermined number of degrees of rotation relative to the lower part.

[0099] In some embodiments, the translation limit may be a function of a degree and a length of time, the degree is the degree by which the upper part is offset relative to the lower part, and the length of time is the degree that the upper part has been relative to the lower part. The length of time for the lower offset.

[0100] In some embodiments, the rotation system may be designed to rotate the robot toward the second traveling direction by rotating the upper part of the robot toward the second traveling direction at a first rotation rate. Rotating the lower part of the robot toward the second direction of travel at a second rotation rate; detecting that the upper part of the robot has reached the second direction of travel; and when simultaneously rotating the upper part of the robot in reverse, The lower part of the robot is continuously rotated toward the second traveling direction, so that the upper part of the robot maintains the second traveling direction.

[0101] In some embodiments, the remote robot may further include: an imaging system connected with the control system; and a positioning system connected with the control system, the positioning system designed to provide the robot with The current position in the plan view and the current orientation of the upper part relative to the plan view; wherein the control system is designed to input video input from the imaging system, the current position of the robot, and the The current orientation of the upper part is transmitted to the remote control terminal, so that the remote control terminal can determine the distortion between the plan view and the video input received from the remote control remote robot imaging system (for example, between the two-dimensional coordinate system and the three-dimensional coordinate system). Coordinate conversion); applying the distortion to a marker having coordinates associated with the plan view, thereby determining the corresponding video coordinates and projection data describing the positioning and projection of the marker relative to the video input; and using The video coordinates show that the mark covers the three-dimensional reproduction of the video input.

[0102] The above embodiments are illustrated by projections of robots and/or local terminals. It is obvious to those skilled in the art that the foregoing implementation manners can be implemented as a system, adjusted to a method implemented by the system, or embodied in a computer-readable medium that can be executed by the system. For example, the method of changing the traveling direction of the robot may include transmitting the traveling direction to the control system of the robot, and the control system of the robot connected with the driving system is designed to make the robot move according to the driving instruction, and make the control system of the robot move. The upper part independently rotates from the lower part of the robot toward the traveling direction.

[0103] In some embodiments, the method of controlling a remote-controlled remote robot may include: reading at least a part of a plan view, the part of the plan view representing a robot passable area of the robot operating surface; reading at least one of a plurality of marks, Each of the plurality of markers includes marker coordinates and marker information describing the relative positioning of the marker; receiving video input from the imaging system of the remote control remote robot; receiving information associated with the current position of the remote control remote robot Positioning information; display the video input from the imaging system of the remote control remote robot through an electronic display; use the marker coordinates to display the reproduction of at least one marker on the video input through the electronic display; and transmit the command to The remote control remote robot. A method for controlling a remote robot includes: reading at least a part of a plan view; reading at least one of a plurality of marks, each mark is a data structure including mark coordinates and mark information describing the relative positioning of the mark; The current position of the plan view; at least one mark identifying the plurality of marks with respect to the travel path of the remote robot; performing an action based on the identified mark, and the mark information of the identified mark includes a remote action correction factor.

[0104] In some embodiments, the method of controlling a remote robot may include: reading at least a part of a plan view,

CN 104898652 Β

This part of the plan view represents the robot passable area of the robot operating surface; receiving video input from the imaging system of the remote control remote robot at the first projection; receiving positioning data associated with the current position of the remote control remote robot; displaying from The video input of the imaging system of the remote control remote robot; the transmission of commands to the remote control remote robot; and accepts a variety of movement options from the movement of the user input device, and the movement selection is performed as follows: (1) With respect to the Video input; (2) relative to the plan view; and/or (3) advance the remote control remote robot incrementally in a direction relative to the current position of the remote control remote robot.

[0105] The details of one or more embodiments of the present invention are set forth in the drawings and the following description. Other aspects, features and advantages of the present invention will become apparent from the accompanying drawings and claims.

Description of the drawings

[0106] FIG. 1 is a perspective view of an exemplary remote robot.

[0107] FIG. 2 is an elevated perspective view of an exemplary remote robot.

[0108] FIGS. 3A-3C are schematic diagrams of exemplary remote robots.

[0109] FIG. 4A is a front perspective view of an exemplary base of a mobile human interface robot.

[0110] FIG. 4B is a rear perspective view of the base shown in FIG. 4A.

[0111] FIG. 4C is a top view of the base shown in FIG. 4A.

[0112] FIG. 4D is a top schematic view of an exemplary base of a remote robot.

[0113] FIG. 4E is a bottom perspective view of an exemplary drive system of the remote robot.

[0114] FIG. 4F is a top perspective view of the drive system shown in FIG. 4E.

[0115] FIG. 5 is a schematic diagram of an exemplary control system executed by a controller of a remote robot.

[0116] FIG. 6A provides a schematic diagram of an exemplary robot system including a plurality of robots communicating with a robot terminal server.

[0117] FIG. 6B shows a remote software application executed by a robot or terminal.

[0118] FIG. 6C shows an embodiment of a screenshot of a user interface for controlling the travel of a semi-automatic remote robot.

[0119] FIG. 6D shows a screenshot in which the relative area of the screen for the map window is increased.

[0120] FIG. 7 is a schematic diagram of an exemplary robot system configuration.

[0121] FIG. 8A is a schematic diagram of an exemplary occupancy diagram.

[0122] FIG. 8B is a schematic diagram of the field of view of the scene that the remote robot has in the working area.

[0123] FIG. 8C is a schematic diagram of an exemplary layout diagram.

[0124] FIG. 8D is a schematic diagram of an exemplary robot diagram corresponding to the layout diagram shown in FIG. 8C.

[0125] FIG. 8E provides an exemplary arrangement of operations for operating a remote robot to move around the environment using a layout diagram and a robot diagram.

[0126] FIG. 8F shows a method of using robot positioning and projection to determine the deviation between the video input and the plan view.

[0127] FIG. 9A is a schematic diagram of an exemplary remote control video field of view from a robot placed in an aisle.

[0128] FIG. 9B is a schematic diagram of an exemplary mixed graph in which the remote control video field of view shown in FIG. 9A is mixed with the graph indicating the room number.

[0129] FIG. 10A provides an exemplary remote control view of the remote control video window of the remote software application.

[0130] FIG. 10B is a schematic diagram of an exemplary diagram of an area shown by the remote control field of view of FIG. 10A.

CN 104898652 Β

[0131] FIG. 10C is a schematic diagram of an exemplary foreseeable field of view of a remote software application.

[0132] FIG. 10D is a schematic diagram of the diagram shown in FIG. 10B with a robot icon and a corresponding imaging area of the field of view thereon.

[0133] FIG. 10E is a schematic diagram of an exemplary foreseeable field of view of a remote software application.

[0134] FIG. 10F is a schematic diagram of the diagram shown in FIG. 10B with a robot icon and a corresponding imaging area of the field of view thereon.

[0135] FIG. 10G provides an exemplary operational arrangement of the foreseen route of the remote software application.

[0136] FIG. 11A is a schematic diagram of an exemplary user interface that allows the user to specify the destination of the robot within the authenticated passage area.

[0137] FIG. 11B provides an exemplary operation arrangement of a method of passing a robot to a destination.

[0138] FIG. 11C is a schematic diagram of an exemplary user interface prompting the user that a ramp has been selected as the destination of the robot.

[0139] FIG. 11D is a schematic diagram of an exemplary user interface prompting the user that an obstacle has been selected as the destination of the robot.

[0140] FIG. 12 is a schematic diagram of an exemplary user interface that allows the user to clarify the driving path of the robot in the identified passable area.

[0141] FIG. 13 is a schematic diagram of an exemplary user interface that mixes numerous markings and environment-sensitive commands.

[0142] FIG. 14 is a perspective view of an exemplary remote robot maintaining a sensor field of view on a human body.

[0143] FIG. 15A is a schematic diagram of an exemplary three-dimensional map field of view including numerous markers.

[0144] FIG. 15B is a schematic diagram of an exemplary two-dimensional view field including numerous marks.

[0145] FIG. 16A is a schematic diagram of an exemplary robotic system.

[0146] FIG. 16B is a schematic diagram of an exemplary interaction with a graph data source.

[0147] FIG. 16C is a schematic diagram of an exemplary interaction between the robot control system and the graph data source.

[0148] FIG. 16D is a schematic diagram of an exemplary robotic system.

[0149] FIG. 17 is a schematic diagram of an exemplary user interface including an enhanced overlay corresponding to a remote robot.

[0150] FIG. 18 is a schematic diagram of an exemplary series of robot actions.

[0151] FIG. 19 is a schematic diagram of an exemplary user interface with a screen indicator covering a remote control video input received from a remote robot.

[0152] FIGS. 20A-20C provide an exemplary arrangement of operations to repair a loss of communication with the robot.

[0153] The same reference numerals in different drawings indicate the same elements.

Detailed ways

[0154] Remote robots can interact or interact with humans to provide many services, for example, physicians or health workers provide remote medical consultation, family assistance, business assistance, and so on. In the case of home assistance, remote robots can assist the elderly in daily tasks, including but not limited to maintaining medication management; mobility assistance; communication assistance (for example, video conferencing, telecommunications, Internet access, etc.); home or place monitoring (Inside and/or outside); personal monitoring; and/or provision of a personal emergency response system (PERS). In terms of business assistance, remote robots can provide video conferencing (for example, in a hospital environment); point of sale terminal; interactive information/point of sale terminal, etc.

[0155] In some specific embodiments, referring to FIGS. 1-3B, the remote robot 100 includes a robot body 110 (or a chassis) that defines a forward driving direction F. The robot 100 also includes a driving system 200 (FIG. 4D ), an interface module 300 and a sensor system 400, which are all supported by the robot body 110 and communicate with a controller 500 (FIG. 5) that coordinates the operation and movement of the robot 100. The power source 105 (for example, a battery) can be carried by the robot main body 110 and, if necessary, is electrically connected to these elements and transmits power to them.

CN 104898652 Β

[0156] In the illustrated example, the robot body 110 includes a base 120, at least one leg 130 extending upward from the base 120, and a torso 140 supported by the at least one leg 130. The base 120 can support the driving system 200. The robot body (lower part) 110 also includes a neck 150 supported by the torso 140. The neck 150 supports a head (upper part) 160, and the head 160 supports at least a part of the interface module 300. The base 120 includes sufficient weight (for example, by supporting the power supply 105 (battery)) to maintain the low center of gravity CGb of the base 120 and the low overall center of gravity CGr of the robot 100 to maintain mechanical stability.

[0157] In some embodiments, referring to FIGS. 2 and 4A-4C, the base 120 defines a shape having three equal sides (for example, a triangle in the top view). For example, the base 120 may include a base frame 122 that supports a base body 124 having first, second, and third base body portions 124a, 124b, and 124c (for example, See Figure 4A). Each base body portion 124a, 124b, and 124c can be movably supported by the chassis frame 122 so as to independently move relative to the chassis frame 122 in response to contact with a target. The equal shape of the three sides of the base 120 allows 360° collision detection around the robot 100. Each base body part 124a, 124b, and 124c may have an associated contact sensor (such as a capacitance sensor, a reading converter, etc.), which detects the corresponding base body part 124a, 124b, and 124c relative to the base frame 122 mobile.

[0158] In some embodiments, the drive system 200 provides omnidirectional and/or complete mechanical motion control of the robot 100. The term "omnidirectional" as used in the text refers to the ability to move in any basic plane direction, that is, left and right (lateral), forward/backward, and rotation. Here, these directions are usually referred to as x, y, 0, S, respectively. In addition, the term "holonomic" is used in a manner that is basically consistent with the literature application of the term, and refers to the direction in the plane Three plane degrees of freedom, that is, the ability to move two translations and one rotation. Therefore, a robot with complete mechanics has the ability to move in the plane direction at a speed consisting of three plane speeds (lateral, forward/backward, and rotation) in essentially any ratio, and the ability to change these ratios in a substantially continuous manner .

[0159] The robot 100 utilizes wheeled mobility to operate in a human environment. In some embodiments, the drive system 200 includes first, second, and third drive wheels 210a, 210b, and 210c (ie, the three sides are equal) that are equidistant (for example, 120 degrees apart) about the longitudinal axis Z; however, other arrangements It is also possible. 4D, the driving wheels 210a, 210b, and 210c may define a laterally arc-shaped rolling surface (ie, a curved profile in a direction transverse to or perpendicular to the rolling direction DR), which may promote the mobility of the complete driving system 200. Each driving wheel 210a, 210b, and 210c is connected to a respective driving motor 220a, 220b, and 220c, which can drive the driving wheel 210a, 210b in a forward and/or backward direction independent of the other driving motors 220a, 220b, and 220c And 210c. Each drive motor 220a-c can have its own encoder, which provides wheel rotation feedback to the controller 500. In some examples, each of the driving wheels 210a, 210b, and 210c are mounted on or near three points of an equilateral triangle, and has a driving direction (forward direction and backward direction) perpendicular to the angle bisector of the corresponding triangle end Direction). Drive the complete mechanical base 120 with three equal sides in the forward drive direction F so that the robot 100 can be converted to a non-forward drive Direction to escape from restrictions or chaos, and turn and/or change to drive along the forward drive direction F after the escape has been resolved.

[0160] In some embodiments, referring to FIGS. 4E and 4F, the drive system 200 includes first, second, third, and third drive wheels 210a-d, which are arranged in a square or rectangular shape from a top view. In the configuration (for example, equidistant from the Z axis). The drive system 200 can be operated in a complete mechanics mode, allowing sweeping. Each driving wheel 210a-d is connected to a respective driving motor 220a-d, and the motor can drive the driving wheel 210ad in a forward and/or backward direction independently of the other driving motors 220a-d. Each drive motor 220a-d can have its own encoder, which provides wheel rotation feedback to the controller 500. The chassis frame 122 supports the drive motors 220a-d and the corresponding drive wheels 210a-d connected thereto.

[0161] As shown in FIG. 2, in some examples, the first driving wheel 210a is arranged as a guide along the forward driving direction F.

CN 104898652 Β

Guide driving wheels, and the remaining two driving wheels 210b and 210c follow behind. To drive forward in this arrangement, the controller 500 can issue a drive command that causes the second and third drive wheels 210b and 210c to drive in the forward rolling direction at the same speed, while the first drive wheel 210a moves along Slide in the forward drive direction F. In addition, this drive wheel arrangement allows the robot 100 to stop suddenly (for example, subject to rapid negative acceleration opposite to the forward drive direction F). This is due to the normal dynamic instability of the three-wheel design. If the forward drive direction F is along the bisector of the angle between the two forward drive wheels, a sudden stop will generate a torque that forces the robot 100 to rotate around its two "front" wheels, causing it to decelerate. On the contrary, if a quick stop is required, a driving wheel 210a is allowed to move forward naturally to support or prevent the robot 100 from falling forward. However, when starting acceleration from a stop, the controller 500 needs to consider its moment of inertia I from the overall center of gravity CGr of the robot 100.

[0162] In some specific embodiments of the drive system 200, each of the drive wheels 210a, 210b, and 210c has a rolling direction Dr radially aligned with the longitudinal axis Z, which is perpendicular to the X and Y axes of the robot 100. The first driving wheel 210a can be arranged to guide the driving wheel along the forward driving direction F, while the remaining two driving wheels 210b and 210c follow behind. In this arrangement, in order to drive forward, the controller 500 may issue a drive command that causes the first drive wheel 210a to be driven in a forward rolling direction, and the second and third drive wheels 210b and 210c are in contact with the first drive wheel 210a. A driving wheel 210a is driven at the same speed but in the opposite direction.

[0163] In other specific embodiments, the drive system 200 can be arranged such that the first and second drive wheels 210a and 210b are positioned such that the angular bisector of the angle between the two drive wheels 210a and 210b is equal to that of the robot 100. The forward driving direction F is aligned. In this arrangement, in order to drive forward, the controller 500 may issue a drive command that causes the first and second drive wheels 210a and 210b to drive in the forward rolling direction and at the same speed, while the third drive wheel 210c is driven or held in the opposite direction, and drags behind the first and second driving wheels 210a and 210b. In order to turn left or right when driving forward, the controller 500 may issue a command that causes the corresponding first or second driving wheels 210a and 210b to drive at a relatively fast/slow speed. Other drive system arrangements can also be used. The driving wheels 210a, 210b, and 210c may define cylindrical, circular, elliptical, or polygonal shapes.

[0164] Referring again to FIGS. 1-3B, the base 120 supports at least one leg 130 extending upward from the base 120 in the Z direction. The legs 130 may be designed to have a variable height for raising and lowering the torso 140 relative to the base 120. In some embodiments, each leg 130 includes first and second legs 132, 134 that move relative to each other (eg, telescopic movement, linear movement, and/or angular movement). Rather than telescoping the protrusions with gradually smaller diameters to move in and out of each other, and to move out the relatively larger base protrusions, in the example shown, the second leg 134 telescopes and moves on the first leg 132, thus making Other elements can be placed along the second leg 134 and may move to a relatively close vicinity of the base 120 as the second leg 134 moves. The leg 130 may include an actuator assembly that moves the second leg 134 relative to the first leg 132. The actuator assembly 136 may include a motor driver in communication with a hoist motor and an encoder, and the encoder provides position feedback to the controller.

[0165] Generally, the telescopic arrangement includes a protrusion that gradually decreases in diameter, and the protrusion moves up and out of the base 120 telescopically and moves out of a relatively large protrusion, so that the center of gravity CG1 of the entire leg 130 is as low as possible. In addition, stronger and/or larger elements can be placed on the bottom to cope with the greater torque experienced by the base 120 when the legs 130 are fully extended. However, this method has two problems. First, when a relatively small element is placed on the top of the leg 130, any rainwater, dust, or other particles are likely to flow or fall into the protrusions and penetrate into the spaces between the protrusions, thus preventing the protrusions from nesting. This creates a very difficult sealing problem while still trying to maintain the complete movability/engagement of the leg 130. Second, loads or accessories can be installed on the robot 100. A common location for mounting accessories is the top of the torso 140. If the second leg 134 is telescopically moved in and out of the first leg, accessories and components can only be installed on the entire second leg 134, if they need to follow

CN 104898652 Β

When the torso 140 moves. Otherwise, any element installed on the second leg 134 will limit the telescopic work of the leg 130.

[0166] By making the second leg 134 telescopically move on the first leg 132, the second leg 134 provides an additional load attachment point that can move vertically relative to the base 120. This type of arrangement causes particles in the water or air to flow down the torso 140 on the outside of each leg 132 and 134 (eg, protrusion) without entering the space between the legs 132 and 134. This greatly simplifies the sealing of any connecting portion of the leg 130. In addition, regardless of how the legs 130 are stretched, the load/accessory mounting features of the torso 140 and/or the second leg 134 are always publicly available and available.

[0167] Referring to FIG. 2, the legs 130 support a torso 140, which may have shoulders 142 extending above and above the base 120. In the example shown, the torso 140 has a downwardly facing surface or bottom surface 144 (for example, toward the base) forming at least a portion of the shoulder 142 and an opposite upwardly facing surface or top surface 146 (FIG. 3A), with side surfaces 148 between them. Between stretches. The torso 140 can define different shapes or geometric shapes, such as a circle or an ellipse, which have a central portion 141 supported by the legs 130 and a peripheral free portion 143 that extends laterally beyond the lateral limits of the legs 130, thus providing a defined face downward An overhanging portion of the surface 144. In some examples, the torso 140 defines polygons or other complex shapes that define shoulders, which provide overhangs of the legs 130 that extend above the base 120.

[0168] The robot 100 may include one or more accessory ports 170 (eg, mechanical and/or electrical connection points) for receiving loads. The accessory port 170 can be positioned such that the received load does not block or interfere with the sensors of the sensor system 400 (for example, on the bottom and/or top surfaces 144, 146 of the torso 140, etc.).

[0169] The outer surface of the torso 140 may be sensitive to the user's contact or touch in order to receive the user's touch command. For example, when the user touches the top surface 146 of the torso 140, the robot 100 will respond by lowering the height of the torso relative to the ground (for example, by lowering the height of the legs 130 supporting the torso 140). Similarly, when the user touches the bottom surface 144 of the torso 140, the robot 100 responds by raising the torso 140 relative to the ground (for example, by raising the height of the legs 130 that support the torso 140). In addition, when a user touch is received on the front part, rear part, right part, or left part of the side surface 148 of the torso 140, the robot 100 uses the received touch command (for example, backward, forward, and backward, respectively). Left and right) corresponding to the direction of movement. The outer surface of the torso 140 may include a capacitive sensor in communication with a controller that detects user contact.

[0170] Referring again to FIGS. 1-3B, the torso 140 supports the neck 150, which provides translation and tilting of the head 160 relative to the torso 140. In the example shown, the neck 150 includes a rotating device 152 and a tilter 154. The rotation device 152 can provide an angular movement range between about 90° and about 360° (for example, around the Z axis). Other ranges are also possible. In addition, in some examples, the rotating device 152 includes electrical connectors or contacts, which enable the head 160 to continuously rotate 360° with respect to the torso 140 at an unlimited number of rotations, while maintaining the gap between the head 160 and the rest of the robot 100 Telecom. The tilting device 154 may include the same or similar electrical connectors or contacts, which can realize the rotation of the head 160 relative to the torso 140 while maintaining electrical communication between the head 160 of the robot 100 and the rest. The rotating device 152 may include a rotating device motor connected to or engaged with an annular object (for example, a zigzag ring frame). The tilting device 154 can move the head independently of the rotating device 152 relative to the torso 140 at an angle of 0x (for example, around the Y axis). In some examples, the tilt device 154 includes a tilt device motor that moves the head 160 relative to the Z axis between an angle θτ±90°. Other ranges are also available Line, for example, soil 45. Wait. The robot 100 may be set so that the legs 130, the torso 140, the neck 150, and the head 160 are kept within the boundary of the base 120 to maintain the stable mobility of the robot 100.

[0171] The head 160 may be sensitive to the user's contact or touch in order to receive the user's touch command. For example, when the user pulls the head 160 forward, the head 160 negatively resists and tilts forward, and then maintains the position. In addition, if the user pushes/pulls the head 160 vertically downwards, the torso 140 can lower the head 160 (by reducing the length of the legs 130). The head 160 and/or neck 150 may include a deformation detector and/or a contact sensor 165 that senses user contact or manipulation.

CN 104898652 Β

[0172] In some embodiments, the head 160 supports one or more parts of the interface module 300. The head 160 may include a dock 302 for releasably receiving one or more computing tablets 310. The computing tablets 310 are also called web pads or tablet computers. There may be a touch screen 312. The network board 310 may be oriented forward, backward, or upward. In some embodiments, network board 310 includes a touch touchscreen, optional 1/0 (e.g., buttons and / or connectors, as micro-USB, etc.), a processor and a memory and a processor Unicom. An exemplary web board 310 includes Apple's Apple iPad. In some instances, the function of the network board 310 is the same as that of the controller 500, or assists the controller 500 to control the robot 100. The touch screen can detect, monitor and/or reproduce the point the user touches on it to receive user input and provide a graphical user interface for contact interaction. In some instances, the web board 310 includes a touch screen calling program, and when it is removed from the robot 100, the touch screen calling program allows the user to retrieve it.

[0173] The interface module 300 may include a camera 320 (for example, see FIG. 3A) disposed on the head 160, and the camera 320 is used to capture video from an elevated vantage point of the head 160 (for example, for video conferencing). In the example shown in FIG. 2, the camera 320 is arranged on the neck 150. In some instances, the camera 320 can only operate after the network board 310 is detached or removed from the head 160. When the network board 310 is attached or parked on the base 302 of the head 160 (and optionally covers the camera 320), the robot can use the camera of the network board 310 to capture video. In this instance, the camera 320 may be configured behind the parked network board 310 and enter the active state after the network board 310 is separated or removed from the head 160, while the network board 310 is attached or parked on the head 160 When up, it is in standby state.

[0174] The robot 100 can provide a video conference (for example, 24 frames per second or higher) through the interface module 300 (for example, using a network board 310, a camera 320, a loudspeaker 330, and/or a speaker 340). Video conferences can be conducted by multiple parties. The robot 100 can provide eye contact between both parties in the video conference by manipulating the head 160 to face the user. In addition, the robot 100 can have V5. The viewing angle (for example, the angle deviating from the axis orthogonal to the outward surface of the head 160). At least one three-dimensional image sensor 450 and/or camera 320 on the robot 100 can capture images of actual size including body language. The controller 500 can synchronize audio and video (for example, a difference of V50ms). The camera 320 can move freely within at least 1° independently of the network board 310. The head 160 may include one or more speakers 340 to have a sound source from the head 160 near the web board 310 displaying the video conference.

[0175] The interface module 300 may include a loudspeaker 330 (or microphone group) for receiving sound input and one or more speakers 340 configured on the robot body 110 for transmitting sound output.

[0176] Referring to FIGS. 1-3C, in order to achieve reliable and stable autonomous movement, the sensor system 400 may include several different types of sensors, which can be used to interact with other sensors to generate sufficient perception of the robot environment. The robot 100 can make intelligent decisions about actions in its environment. The sensor system 400 may include one or more types of sensors supported by the robot body 110, and these sensors may include obstacle detection and avoidance (OD0A) sensors, communication sensors, navigation sensors, and the like. For example, these sensors may include, but are not limited to, progress sensors, contact sensors, three-dimensional imaging/depth map sensors, cameras (such as visible light and/or infrared cameras), sonar, radar, light detection and ranging (LIDAR, It can cause the performance of detecting scattered light to find the range of far-end targets and/or other information (optical remote sensing), laser detection and ranging (LADAR), etc. In some embodiments, the sensor system 400 includes ranging sonar sensors 410 (for example, nine around the perimeter of the base 120), a proximity detector 420, a contact sensor 430 (FIG. 4A), a laser scanner 440, One or more three-dimensional imaging/depth sensors 450 and imaging sonar 460.

[0177] In some embodiments, the sensor system 400 includes a group or a row of progressive sensors 410 and 420, which are in communication with the controller 500 and are arranged in one or more areas or parts of the robot 100 (for example, arranged in The base body parts 124a, 124b, and 124c of the robot 110 are used to detect any nearby or intrusive obstacles. process

CN 104898652 Β

The integrated sensors 410 and 420 may be convergent infrared (IR) emitting sensor elements, sonar sensors, ultrasonic sensors, and/or imaging sensors (e.g., three-dimensional depth sensors) that provide signals to the controller 500 when the object is within a given range of the robot 100. Figure of the image sensor).

[0178] In the example shown in FIGS. 4A-4C, the robot 100 includes a row of sonar-type progress sensors 410 (for example, substantially equidistant) disposed around the base 124 of the main body 120 and whose field of view is set upward. The first, second, and third sonar progress sensors 410a, 410b, and 410c are arranged on or near the first (forward) base body portion 124a, and at least one sonar progress sensor is located on the diameter of the first base 124a of the main body 120. To the vicinity of the outermost edge 125a. The fourth, fifth, and sixth sonar progression sensors 410d, 410e, and 410f are arranged on or near the second (right) base body portion 124b, and at least one sonar progression sensor is located on the diameter of the second base 124b of the main body 120. To the vicinity of the outermost edge 125b. The seventh, eighth, and ninth sonar progression sensors 410g, 410h, and 410i are arranged on or near the third (left) base body portion 124c, and at least one sonar progression sensor is located on the diameter of the third base 124c of the main body 120. To the vicinity of the outermost edge 125c. This design provides at least three detection areas.

[0179] In some examples, a set of sonar progression sensors 410 (for example, 410a-410i) disposed around the base 124 of the main body 120 are arranged to point upward (for example, substantially in the Z direction), and can optionally deviate from the Z axis outward An angle is formed, and thus a detection curtain 412 around the robot 100 is created. Each sonar progress sensor 410a410i may have a shield or launch guide 414, which can guide the sonar to launch upward or at least not toward other parts of the robot body 110 (for example, so as not to detect the movement of the robot body 110 relative to itself). The emission guide 414 may define a shell shape or a half shell shape. In the example shown, the base 124 of the main body 120 extends laterally beyond the leg 130, and the sonar progress sensor 410 (e.g., 410a-410i) is disposed on the base 124 of the main body 120 surrounding the leg 130 (e.g., substantially along The boundary of the base 124 of the main body 120). In addition, the upward-pointing sonar progression sensor 410 is spaced apart to create a continuous or substantially continuous sonar detection curtain 412 around the leg 130. The sonar detection curtain 412 can be used to detect obstacles with elevated lateral protrusions, such as desktops, shelves, and the like.

[0180] The upward-looking sonar process sensor 410 provides the ability to see objects mainly on a horizontal surface, such as a desktop. These objects may be missed by other sensors of the sensor system due to their aspect ratio, such as the laser scanner 440 or the imaging sensor 450, and this may cause problems for the robot 100. The upward-looking sonar progression sensor 410 arranged around the boundary of the base 120 provides a way to see or detect this type of object/obstacle. In addition, the sonar process sensor 410 can be placed around the widest point of the base boundary and angled slightly outward so as not to be blocked or hindered by the torso 140 or the head 160 of the robot 100, and therefore does not cause damage to the robot 100 itself. Perceived false positives. In some embodiments, the sonar progression sensor 410 is arranged (upward and outward) outside the field of view of the sonar progression sensor 410 to leave space around the torso 140 and therefore can freely receive installed loads or accessories, such as basket 360 . The sonar process sensor 410 can be recessed into the base body 124 to provide visual concealment, and has no appearance features to hook or hit obstacles.

[0181] The sensor system 400 may include one or more sonar progressive sensors 410 (for example, post-progressive sensors 410j) facing backward (for example, opposite to the forward driving direction F) for detecting obstacles when retreating. The rear sonar process sensor 410 j may include a transmission guide 414 for detecting its sonar detection area 412. In addition, the rear sonar progress sensor 410 j can be used for distance measurement to measure the distance between the robot 100 and an object detected within the field of view of the rear sonar progress sensor 410 j (for example, as a "backward alarm"). In some examples, the rear sonar process sensor 410j is installed to be recessed into the base 124 of the main body 120 so as not to provide any visual or functional irregularities in the form of a cover.

2 and 4B, in some embodiments, the robot 100 includes a cliff progress sensor 420, which

CN 104898652 Β

They are installed near or around the driving wheels 210a, 210b, and 210c to allow cliff detection before the driving wheels 210a, 210b, and 210c enter the cliff (eg, stairs). For example, the cliff progress sensor 420 can be positioned on or near the radially outermost edge 125a-c of each base unit 124a-c, and at a position between them. In some cases, cliff sensing is implemented by using infrared (IR) proximity or actual range sensing, using infrared emitters 422 and infrared detectors 424 that form angles toward each other to have overlapping emission and detection fields, and therefore have The detection area at a predictable position on the ground. IR proximity sensing can have a relatively narrow field of view, which may depend on the reliability of the surface albedo, and can have different ranges of accuracy from surface to surface. As a result, multiple scattered sensors can be placed on the boundary of the robot 100 to sufficiently detect cliffs from multiple points on the robot 100. In addition, sensors based on IR proximity usually cannot distinguish between cliffs and safety events, for example, just after the robot 100 has crossed a threshold.

[0183] The cliff progress sensor 420 can detect when the robot 100 encounters a falling edge of the ground, for example, when it encounters a set of stairs. The controller 500 (execution control system) may perform behaviors that cause the robot 100 to take actions, such as changing its forward direction when an edge is detected. In some embodiments, the sensor system 400 includes one or more secondary cliff sensors (for example, other sensors are designed for cliff sensing and can select other types of sensing). The cliff detection progress sensor 420 can be deployed to provide early detection of cliffs, provide data for distinguishing actual cliffs from safety events (such as climbing over thresholds), and be positioned downward and outward so that their field of view includes robots At least a part of the main body 110 and a certain area away from the main body 110 of the robot. In some embodiments, the controller 500 performs identification and inspection of cliffs that support the edge of the working surface (for example, the ground), the distance across the edge of the working surface, and/or the increase in the distance between the robot body 110 and the working surface. Testing procedures. This specific implementation achieves: 1) Early detection of potential cliffs (this can achieve faster mobility in unknown environments); 2) Increase the reliability of autonomous mobility, because the controller 500 can receive cliff images from the cliff detection process sensor 420 Information to understand whether the cliff event is indeed unsafe or whether it can be safely crossed (such as climbing and crossing the threshold); 3) the reduction of cliff false alarms (for example, due to the use of edge detection compared to multiple scattered with narrow field of view Regional IR process sensor). Additional sensors arranged as "wheel drop sensors" can be used for redundancy and for detecting certain types of cliffs that cannot be reliably detected by range sensing cameras.

[0184] Threshold and step detection allows the robot 100 to effectively plan to cross a climbable threshold or avoid excessively high steps. This is the same for whether the robot 100 can safely pass any object on the working surface. For those obstacles or thresholds that the robot 100 judges to be able to climb, knowing their height allows the robot 100 (if deemed necessary) to decelerate appropriately to achieve a smooth transition, in order to maximize stability and minimize any sudden acceleration. Instability. In some embodiments, threshold and step detection is based on object height and geometric shape recognition on the working surface (for example, to distinguish thresholds or wires from larger objects with no certain shape, such as a sock). The threshold can be identified by edge detection. The controller 500 may receive imaging data from the cliff detection progress sensor 420 (or another imaging sensor on the robot 100), execute an edge detection program, and issue a driving command based on the edge detection program result. The controller 500 may also use pattern recognition to recognize objects. Threshold detection allows the robot 100 to change its orientation relative to the threshold to maximize the ability to climb stairs smoothly.

[0185] The progress sensors 410 and 420 may function alone, or alternatively, may also function in combination with one or more contact sensors 430 (for example, a collision switch). For example, one or more contact or collision sensors 430 on the robot body 110 can detect whether the robot 100 physically encounters an obstacle. Such sensors may use physical properties or physical displacements in the robot 100, such as capacitors, to determine whether an obstacle has been encountered. In some embodiments, each of the base body parts 124a, 124b, and 124c of the base 120 has an associated contact sensor 430 (for example, capacitive

CN 104898652 Β

Sensors, reading converters, etc.), which detect the movement of the corresponding base body parts 124a, 124b, and 124c relative to the base frame 122 (as shown in FIG. 4A). For example, each base 124a of the main body 120-c can move radially relative to the Z axis of the base frame 122 to provide three-way collision detection.

[0186] Referring again to FIGS. 1-4C, in some embodiments, the sensor system 400 includes a laser scanner 440 installed on the front of the robot body 110 and communicating with the controller 500. In the illustrated example, the laser scanner 440 is mounted on or above the first base 124a of the main body 120 (for example, along the driving direction F of the robot with the largest imaging coverage) facing forward (for example, along the The field of view in the forward driving direction F) on the base 124. In addition, the placement of the laser scanner on or near the front end of the triangular base 120 means that the outer angle of the robot base (for example, 300°) is larger than the field of view area 442 (for example, -285°) of the laser scanner 440, thus preventing the base 120 from blocking Or hinder the field of view detection area 442 of the laser scanner 440. The laser scanner 440 can be recessed into the base body 124 as much as possible without blocking its field of view, so as to minimize the laser scanner over any protruding part of the base body 124 (for example, for aesthetics and minimizing hooking) Obstacles) ο

[0187] The laser scanner 440 scans the area of the robot 100 and the controller 500 using the signal received from the laser scanner 440, and generates an environment map or object map of the scanned area. The controller 500 may use the object map to navigate, detect obstacles, and avoid obstacles. In addition, the controller 500 may use sensory input from other sensors of the sensor system 400 to generate an object map and/or for navigation.

[0188] In some examples, the laser scanner 440 is a scanning LIDAR, which can use a laser that quickly scans a one-dimensional area as the "main" scanning route, and use phase difference or similar techniques to allocate the number of pixels generated on the route. Depth transit time imaging element (two-dimensional depth route back to the scan plane). In order to generate a three-dimensional image, IDAR can perform an "assisted" scan in a second direction (for example, by "swinging" the scanner). This mechanical scanning technology, if not supplemented, can be supplemented by technology, such as the "flash" LIDAR/LADAR and the "Swiss Ranger" type focal plane imaging principle sensor, which uses semiconductor stacking to allow the complete two-dimensional matrix transit time of the pixel The calculated technique provides the depth at each pixel, or even a series of depths at each pixel (by encoding a light emitter or emitting a laser).

[0189] The sensor system 400 may include one or more three-dimensional image sensors 450 in communication with the controller 500. If the field of view of the 3D image sensor 450 is limited, the controller 500 or the sensor system 400 can drive the 3D image sensor 450a to generate a relatively wide field of view in a side-to-side scanning manner to perform rough obstacle detection/obstacle avoidance (ODOA) ο Referring to FIGS. 1-3B, in some embodiments, the robot 100 includes a scanning three-dimensional image sensor 450a installed in the front of the robot body 110, and its field of view is along the forward driving direction F (for example, along The driving direction F of the robot has the largest imaging range). The scanning three-dimensional image sensor 450a can be mainly used for ODOA. In the illustrated example, the scanning 3D image sensor 450a is mounted on the torso 140 or the bottom surface 144 below the shoulder 142, and is recessed in the torso 140 to prevent the user from contacting the scanning 3D image sensor 450a, as shown in FIG. 2, for example. The scanning three-dimensional image sensor 450 can be installed to aim to be substantially downward and away from the robot body 110 so that there is a downward field of view for ODOA in front of the robot 100 (for example, obstruction of the base 120 or other parts of the robot body 110). On or near the front edge of the torso 140 The placement of the scanning three-dimensional image sensor 450a can make the field of view area of the three-dimensional image sensor 450 (for example, -285.) relative to the three-dimensional image sensor 450 narrower than the angle of the outer surface of the torso 140 (for example, 300°), thus preventing the torso 140 from blocking Or it prevents scanning of the detection field of view area 452 of the three-dimensional image sensor 450a. In addition, the scanning 3D image sensor 450a (and the associated actuator) can be installed to be recessed into the torso 140 as much as possible without blocking its field of view (for example, also for aesthetics and minimizing hooking of obstacles). The scattered scanning movement of the scanning three-dimensional image sensor 450a is invisible to the user, resulting in less scattered interaction experience. Unlike protruding sensors or faces, concave scanning three-dimensional image sensing

CN 104898652 Β

The device 450a does not unconsciously interact with the environment (hooking people, obstacles, etc.), especially when moving or scanning, because almost no moving part extends beyond the shell of the torso 140.

[0190] In some embodiments, the sensor system 400 includes an additional three-dimensional image sensor 450 disposed on the base 124, the legs 130, the neck 150, and/or the head 160 of the main body 120. In the example shown in FIG. 1, the robot 100 includes a three-dimensional image sensor 450 on the base 124, the torso 140 and the head 160 of the main body 120. In the example shown in FIG. 3A, the robot 100 includes a three-dimensional image sensor 450 on the base 124, the torso 140 and the head 160 of the main body 120. In the example shown in FIG. 3B, the robot 100 includes three-dimensional image sensors 450 on the legs 130, the torso 140 and the neck 150. Other configurations are also possible. One three-dimensional image sensor 450 (for example, on the neck 150 and head 160) can be used for person recognition, motion recognition, and/or video conferencing, while another three-dimensional image sensor 450 (for example, on the base 120 and/or leg 130) can be used for person recognition, motion recognition, and/or video conferencing. Above) can be used for navigation and/or obstacle detection and obstacle avoidance.

[0191] The forward-facing three-dimensional image sensor 450 disposed on the neck 150 and/or the head 160 can be used to recognize people, faces, and/or postures of people around the robot 100. For example, using the signal input from the three-dimensional image sensor 450 on the head 160, the controller 500 can recognize the user by the following method, generate a three-dimensional image of the users face seen/captured, and compare the generated three-dimensional image with a known face of a person. Three-dimensional image, and determine the match with a known three-dimensional facial image. Facial recognition can be used to confirm that the user is a legitimate user of the robot 100. In addition, one or more three-dimensional image sensors 450 can be used to confirm the posture of the person seen by the robot 100, and react at will based on the confirmed posture (for example, hand pointing, waving, and/or gesture). For example, the controller 500 may issue a driving command in response to the recognized finger pointing in a specific direction.

[0192] The three-dimensional image sensor 450 can generate the following types of data: (i) a depth map. (Ii) Intensity map based on reflectance, and/or (iii) Regular intensity map. The three-dimensional image sensor 450 can obtain the data through image pattern matching, and measure the transit time and/or the phase delay shift between the light emitted from the source and the light reflected by the target.

[0193] In some embodiments, the thinking or control software that can be executed on the processor (eg, the robot controller 500) uses a combination of computing software, which is executed using various data types generated by the sensor system 400. Thinking software processes the data collected from the sensor system 400 and outputs the data to make navigation decisions, for example, where the robot 100 can move without colliding with obstacles. Through the accumulation of image data of the robot environment over time, the thinking software can use effective methods to select segments of the perceived image to improve the depth measurement of the three-dimensional image sensor 450. This can include the use of appropriate temporal and spatial averaging techniques.

[0194] The reliability of performing collision-free robot movements can be based on: (i) credibility established by high-level thinking over time and (ii) depth perception sensors that accumulate three main types of data for analysis : (A) Depth map, (b) Active lighting map and (c) Ambient lighting map. The calculation program recognition for different types of data can be performed on each image obtained by the depth-sensing imaging sensor 450. Collected data can increase credibility compared to systems that use only one type of data.

[0195] The three-dimensional image sensor 450 may obtain an image including depth and brightness data from a scene including one or more objects around the robot 100 (for example, a sensor field of view part of a room or a work area). The controller 500 may be designed to confirm the occupied area data of the object based on the captured light reflected from the scene. In addition, in some instances, the controller 500 issues a driving command to the driving system 200 to surround an obstacle (ie, an object in the scene) based at least in part on the occupied area data. The three-dimensional image sensor 450 can repeatedly capture the scene depth map for the controller 500 to make a real-time decision to navigate the robot 100 around the scene and prevent it from colliding with any objects in the scene. For example, the speed or frequency at which the depth map data can be obtained by the three-dimensional image sensor 450 can be controlled by the shutter speed of the three-dimensional image sensor 450. In addition, the controller 500 may receive an event trigger (for example, from another sensor element of the sensor system 400, such as process sensors 410 and 420,

CN 104898652 Β

Report nearby objects or dangers to the controller 500). In response to the event trigger, the controller 500 can cause the three-dimensional image sensor 450 to increase the frequency, thereby capturing the depth map and obtaining occupancy information.

[0196] In some embodiments, the robot includes a sonar scanner 460 for acoustic imaging of the area around the robot 100. In the example shown in FIGS. 1 and 2, the sonar scanner 460 is disposed on the front of the base 124 of the main body 120.

[0197] Referring to FIGS. 1-3B, in some specific embodiments, the robot 100 uses a laser scanner or a laser rangefinder 440 for redundant sensing, and uses a rearward-facing sonar process sensor 410j to ensure safety, both Orientation is parallel to the ground G. The robot 100 may include first and second three-dimensional image sensors 450a and 450b (depth cameras) that provide a rough perception of the surrounding environment of the robot 100. The first three-dimensional image sensor 450a is installed on the torso 140 and directed downward at a fixed angle to the ground G. By making the first three-dimensional image sensor 450a form an angle downward, the robot 100 receives a dense sensor range in the area where the robot 100 is directly advancing or adjacent, which is of great significance to the short-term travel of the robot 100 in the advancing direction. The rear-facing sonar device 410j provides object detection when the robot is traveling backward. If traveling backward is unique to the robot 100, the robot 100 may include a third 3D imaging sensor 450 facing downward and backward to provide a dense sensor range in the immediate rearward or adjacent area of the robot 100.

[0198] The second three-dimensional image sensor 450b is installed on the head 160, which can be translated and tilted through the neck 150. The second three-dimensional image sensor 450b can be used for remote driving because it enables the human operator to see where the robot is headed. The neck 150 enables the operator to tilt and/or translate the second three-dimensional image sensor 450b to view near and far objects. Translating the second three-dimensional image sensor 450b increases the relevant horizontal field of view area. During fast travel, the robot 100 can slightly tilt the second three-dimensional image sensor 450b downward to increase the total or combined field of view of the three-dimensional image sensors 450a and 450b, and give the robot 100 sufficient time to avoid obstacles (because of relatively Faster speed usually means less time to react to obstacles). At a slower speed, the robot 100 can tilt the second three-dimensional image sensor 450b upward or make it substantially parallel to the ground G to monitor the person that the robot 100 should be following. In addition, when driven at a relatively slow speed, the robot 100 can translate the second three-dimensional image sensor 450b to increase its field of view around the robot 100. The first three-dimensional image sensor 450a can remain fixed (for example, does not move relative to the base 120) when the robot is driven to extend its sensing range. In addition and/or alternatively, the first three-dimensional image sensor 450a can be used for inspection while moving. The potential obstacles around the measuring robot are scanned at a slow speed. In some examples, the height of the first three-dimensional image sensor 450a can be adjusted upward in order to optimize the field of view of the first three-dimensional sensor 450a, for example, through the use of a Z-lift.

[0199] In some embodiments, the at least one three-dimensional image sensor 450 can be a volumetric point cloud imaging device (such as a spot or time-of-flight camera), which is 1 or 2 feet above the ground (or at a height of 1 or 2 feet). (Approximately 1 or 2 feet above the ground) is installed on the robot 100 and directed to obtain a point cloud from the volume of space including the ground plane in the direction in which the robot moves (through the omnidirectional drive system 200). In the example shown in FIGS. 1 and 3, the first three-dimensional image sensor 450a can be mounted on the base at a height of 1 or 2 feet higher than the ground, and is aimed along the forward driving direction F to capture the An image of the volume of the ground (for example, for obstacle detection and obstacle avoidance) (for example, a point cloud for measuring the volume). The second three-dimensional image sensor 450b is shown as being installed on the head 160 (for example, at a height of about 3 or 4 feet above the ground), thereby obtaining frame recognition and defining points from the volume of space in close proximity to the robot 100. The controller 500 can execute the framework/digital recognition software to analyze the captured volumetric point cloud data.

3A-4C, the sensor system 400 may include an inertial measurement unit (IMU) 470 communicating with the controller 500, which is used to measure and monitor the moment of inertia of the robot 100 relative to the total center of gravity CGr of the robot 100.

CN 104898652 Β

[0201] The controller 500 can monitor the feedback of the IMU 470 for any deviation from the threshold signal corresponding to normal unhindered operation. For example, if the robot starts to tilt from a vertical position, it may "fall over" or unless it is blocked, or someone May have a sudden increase in heavy load. In these instances, it is necessary to take emergency actions (including but not limited to maneuver evasion, recalibration, and/or issuing audible/visual warnings) to ensure the safe operation of the robot 100.

[0202] Because the robot 100 can operate in a human environment, it can interact with humans and can operate in a space designed for humans (and without considering robot limitations). The robot 100 can limit its driving speed and acceleration when it is in a crowded, restricted or highly dynamic environment, such as in a cocktail party or a busy hospital. However, the robot 100 may encounter situations where it can be safely driven relatively quickly, such as in a long empty corridor, but can also suddenly decelerate, such as when something passes through the moving route of the robot.

[0203] When accelerating from a stop, the controller 500 may consider the moment of inertia from the total center of gravity CGr of the robot 100 to prevent the robot from tipping over. The controller 500 may use a model of its posture, including its current moment of inertia. When supporting a load, the controller 500 can measure the influence of the load on the total center of gravity CGr and monitor the momentary movement of the robot inertia. For example, the torso 140 and/or the neck 150 may include a deformation measurer to measure deformation. If this is not possible, the controller 500 may apply a test torque command to the driving wheel 210 and use the IMU 470 to measure the actual linear and angular acceleration of the robot to confirm the safety limit through experiments.

[0204] At the time of sudden deceleration, the specified load of the second and third driving wheels 210b and 210c (rear wheels) is reduced, while the first driving wheel 210a (front wheel) slides in the forward driving direction and supports the robot 100. If the loads on the second and third driving wheels 210b and 210c (rear wheels) are not equal, the robot can "yaw", which will reduce dynamic stability_oThe IMU 470 (eg, gyroscope) can be used to detect yaw and command the second and third drive wheels 210b and 210c to reorient the robot 100.

5, in some specific embodiments, the controller 500 executes a control system 510, which includes a behavior system 510a and an interconnected control arbitration system 510b. The control arbitration system 510b allows applications 520 to be dynamically added and removed from the control system 510, and facilitates allowing applications to control the robot 100 individually without needing to know any other applications 520. In other words, the control arbitration system 510b provides a simple priority processing control mechanism between the application 520 and the resource 530 of the robot 100. The resource 530 may include the drive system 200, the sensor system 400, and/or any load or controllable device connected with the controller 500.

[0206] The application program 520 can be stored in the memory of the robot 100 or communicated with it to run simultaneously (for example, a processor) and control the robot 100 at the same time. The application 520 can obtain the behavior 512 of the behavior system 510a. The independently deployed application programs 520 are dynamically combined at runtime and share the robot resources 530 of the robot 100 (for example, the driving system 200, the arm, the head, etc.). The execution of the low-level strategy is to dynamically share the robot resources 530 in the application 520 at runtime. This strategy confirms that the application 520 has the control right of the robot resource 530 specified by the application 520 (for example, it creates a priority level between the application 520). The application program 520 can be dynamically started and stopped and run completely independently of each other. The control system 510 can also implement complex behaviors 512, which can be combined together to assist each other.

[0207] The control arbitration system 510b includes one or more resource controllers 540, a robot manager 550, and one or more control arbiters 560. These elements do not need to be in a common process or computer, and do not require any specific commands to start. The resource controller 540 element provides an interface for the application 520 and the control arbitration system 510b. There are instances of such elements for each application 520. The resource controller 540 extracts and summarizes the complexity of authentication, the allocated resource control arbiter, command buffer, and so on. The robot manager 550 coordinates the priority of the application program 520, and has dedicated control over any robot resource 530 at any specific time by controlling the application program 520. Since this is the central coordinator of information, there is only one instance of each robot's robot manager 550. The robot manager 550 executes the priority strategy, which has

CN 104898652 Β

There is a linear priority of the resource controller 540 and the monitoring of the resource control arbiter 560 that provides hardware control is maintained. The control arbiter 560 receives commands from each application 520, generates a signal command based on the priority of the application, and publishes it for related resources 530. The control arbiter 560 also receives situation feedback from the relevant resource 530 and returns it to the application program 520. The robot resource 530 may be a functional module network with one or more hardware controllers (for example, actuators, driving systems, and combinations thereof). The command to control the arbiter 560 is specific to the resource 530 that performs a specific action.

[0208] The dynamic model 570 executable on the controller 500 can be designed to calculate the cross product of the center of gravity (CG), the moment of inertia, and the inertia of various parts of the robot to determine the current situation of the robot. The dynamic model 570 can also model the shape, gravity, and/or moments of inertia of these elements. In some examples, the dynamic model 570 is in multi-part communication with the IMU 470 or one (for example, an accelerator and/or a gyroscope) that is configured on the robot 100 and communicates with the controller 500 for calculating various centers of gravity of the robot 100. The dynamic model 570 can be used by the controller 500 to confirm the operating shell of the robot 100 and its components together with other programs 520 or behaviors 512.

[0209] Each application 520 has an action selection motor 580, a resource controller 540, one or more actions 512 associated with the work selection motor 580, and one or more action models associated with the action selection motor 580 590. The behavior system 510a provides predictive modeling and allows the behavior 512 to collaboratively determine the robot's actions by assessing the possible outcomes of the robot's actions. In some instances, behavior 512 is a plug-in program element that provides a hierarchical, comprehensive evaluation function that combines perceptual feedback from a composite source with a priori restrictions and information input to the robots proper motion evaluation feedback. Behaviors 512 can be inserted into the application 520 (for example, exist inside or outside the application 520), and they can be removed and added without modifying the application 520 or any other part of the control system 510. Each behavior 512 is an independent strategy. In order to make the behavior 512 more efficient, the output of the complex behavior 512 can be attached to other inputs to have a complex mixing function. The behavior 512 means an easy-to-handle part of the overall recognition of the execution robot 100.

[0210] The action selection motor 580 is a coordinating element of the control system 510 and runs a fast and optimized action selection loop (prediction/modification loop) to search for the most suitable action for the input of all actions 512. The action selection motor 580 has three stages: appointment, action selection search and completion. During the appointment phase, each behavior 512 is informed that the action selection cycle has been started and is provided with the limitations of the cycle start time, the current situation, and the robot actuator space. Based on internal strategy or external input, each behavior 512 determines whether it should participate in this action selection cycle. At this stage, a list of valid behavior primitives will be generated, the input of which will affect the selection of commands to be executed on the robot 100.

[0211] In the action selection search stage, the action selection motor 580 generates feasible results from the space of effective actions, which also involves, for example, the action space. Because the action of simulating each command at different times varies with the future time range, the action selection motor 580 uses the dynamic model 590 to provide a batch of feasible commands (within the limit) and corresponding results.

[0212] In the completion stage, the commands corresponding to the most suitable result of the collaboration are mixed together as the overall command, which is presented to the resource controller 540 for execution on the robot resource 530. The most suitable result is provided as feedback to the effective behavior 512 , For future evaluation cycles.

[0213] The sensor signal received from the sensor system 400 can cause interaction with one or more behaviors 512 to perform an action. For example, by using the control system 510, the controller 500 selects the work (or movement) for each machine element (for example, a motor or an actuator) from the corresponding action space (for example, for the collection of possible actions or movements of a specific element). Command) to perform the coordinated actions of each machine element in an effective manner, so as to avoid itself and any objects around the robot 100 that have been detected by the robot 100. The controller 500 can issue a collaboration command that crosses a robot network such as the EtherlO network, as described in US No. 61/305,069 proposed on February 16, 2010, the entire content of which is by reference

Merged here.

[0214] FIG. 6A provides a schematic diagram of an exemplary robot system 600 having one or more remote robots 100 in contact with a bridge 602, which is associated with a local robot terminal server 604a and a remote terminal server 604b (e.g., cloud computing service 720 ( Figure 7)) There is a connection. The local robot terminal server 604a contacts the local technician computing device 606, and the remote terminal server 604b contacts the remote operator computing device 608.

[0215] Referring to FIGS. 2 and 4C, in some embodiments, the robot 100 includes a multi-element antenna. In the example shown, the robot 100 includes a first antenna 490a and a second antenna 490b that are both configured on the base 120 (but the antennas can be configured on any other part of the robot 200, such as the legs 130, the torso 140, the neck 150, and the / Or head 160). The use of multi-element antennas provides reliable signal reception and transmission. The use of multi-element antennas provides the robot 100 with multiple inputs and multiple outputs (MIMO), that is, the use of multi-element antennas of the transmitter and/or receiver improves the communication performance. MIMO provides a significant increase in data throughput and link range without additional bandwidth or transmit power. It achieves this increase through higher spectral efficiency (bandwidth with more bits per second per hertz) and link reliability or diversity (reduce fading). Due to these characteristics, ΜΙΜΟ is an important part of modern wireless communication standards, such as IEEE802.11n (WIFI), 4G, 3GPP long-term evolution, WiMAX and HSPA+. In addition, the robot 100 can act as a Wi-Fi bridge for other nearby electronic devices. , Hub or hotspot. The mobility of the robot 100 and the use of MIMO can allow the robot to serve as a relatively reliable Wi-Fi bridge 602.

6A and 6B, the remote software application 601 is executed on at least one of the following devices, the robot controller 500, the local robot terminal server 604a, the remote terminal server 604b, the local technician computing device 606, and the remote operator computing device 608. In some instances, a portion of the remote software application 601 is executed on one or more of the aforementioned devices. The remote software application 601 allows one or more users to interact with the robot (for example, driving the robot 100) and/or remotely and other people or objects adjacent to the robot 100 through the remote characteristics of the robot 100.

[0217] FIG. 6C provides a schematic diagram of an exemplary user interface 605 of the remote software application 601, which may be presented on the display of the touch screen 312 of, for example, the network board 310 and/or the remote operator computing device 608 for controlling navigation, remote control Monitoring and/or other aspects of the robot 100. The user interface 605 includes a remote video input window 610 that displays a remote video 612, such as a video input of a patient 614. The video input can be generated by one of the cameras 320 and 450 of the robot 100. The user interface 605 can display a planning video map window 620 that contains a map 622 of the local area where the robot 100 is operated. In the illustrated example, the map 622 displayed in the planning video map window 620 is a two-dimensional, top-down map 622a (FIG. 6D); however, other types of maps are also feasible. The user interface 605 may also include a local video window 630 that displays a local video 632, such as a user's video input (for example, from the remote of the robot 100). The video input displayed in the local video window 630 can be transmitted to the robot 100 and displayed to the patient 614 using a display device, such as the network board 310 on the robot 100.

[0218] The dashboard 640 may provide information about the direction of the robot 100, an indication of the battery charger of the robot, an indication of wireless data signal strength, and/or an indication of network quality. The direction of the robot 100 may be indicated by an icon 642 showing the direction of the head 160 of the robot 100 relative to the torso 140 or the base 120. Such instructions can assist the user in determining the direction of the robot 100 to observe the item of interest. The range of movement of the head 160 of the robot may be restricted. Therefore, some executions may display the indication of the rotation position of the head 160 and the movement range of the head 160.

[0219] The media control 647 may allow the user to use various types of media to interact with the patient 614 and obtain and store media that records the interaction between the user and the patient 614. For example, the media control 647 may allow the user to play audio and/or video clips that can be used to train the patient 614 about the medical situation or procedure. In order to record still photos of various situations, the cameras 320 and 450 of the robot 100 can be used to obtain them. Further, the robot 100 may obtain audio (for example, using the speaker 330) or video (for example, using the camera 320) that records the interaction between the user and the patient 614, and may optionally store the obtained

CN 104898652 Β

The audio/video is stored in the memory of the controller 500 and/or the obtained audio/video is transmitted to a remote device or cloud service.

[0220] In some embodiments, the media control 647 allows the user to resolve temporary connectivity issues. For example, a video recording can be started when a meeting is unexpectedly disconnected. The robot 100 can continuously record videos and store them in a local memory, such as the memory of the controller 500. When accidentally disconnected, the robot can display a message, such as "The meeting is terminated-the button below the video recording continues can also display the subtitle "Stop recording". The nurse next to the robot can touch the "Stop recording" button (for example, on the touch screen 312) And terminate the local recording. Otherwise, the recording can continue at a specific time interval. If the same user returns to the robot 100 to record within the specific time interval, the recording button of the remote station can display that the recording is in progress. When the local recording of the robot is completed, it can Start transferring video files to remote stations or other locations that can be approached by the disconnected user. Therefore, the user may see events that occur during the disconnection of the conference.

[0221] In the example shown in FIG. 6C, the remote video input window 610 occupies a relatively large portion of the display area. The user interface 605 may have a remote video input window with a pixel resolution of 640×480 and a local video window 630 with a pixel resolution of 610.320×240 and a planned video window 620 with a pixel resolution of 530×200. Therefore, this video may be most appropriate when the user contacts the patient 614 and/or manually drives the robot 100. The layout of the default user interface 605a shown in FIG. 6C may allow the user to replace the content of the planned video map window 620 with the remote video input window 610. For example, the video can be replaced by double-clicking the graph window 620. The window can be replaced later by double-clicking the remote video input window 610.

[0222] The selectable floor plan may be displayed to the user as long as it is applicable to the task performed by the user. In the example shown in FIG. 6D, it is predicted that the semi-automatic navigation is used to guide the robot 100 to move from one position to another, and the size of the planning video map window 620 is increased. For example, the user interfaces 605 and 605a as shown in FIG. 6C can be the default conditions for patient interaction, and the user interfaces 605 and 605b as shown in FIG. 6D can be alternate conditions for robot navigation.

[0223] The image-to-video conversion button 645 of the user interface may allow the user to refer to an optional user interface 605b including a relatively large image window 620. For example, the user interface 605b as shown in FIG. 6D may be mainly used to manually drive or automatically navigate the robot 100 to a desired target. Clicking the video conversion button 645 again will bring the user back to the default user interface 605a, which can be used when performing medical consultations effectively. Therefore, the user can plan the video image window with or without emphasis (for example, maximize or minimize) as desired. Certain windows shown in the optional user interface 605b are also displayed, such as the remote video input window 610 and the local video window 630. In some examples, the planning video window 620 can be displayed with a pixel resolution of 880 X 700, the remote video input window 610 can be displayed with a pixel resolution of 320 X 240, and the local video window 630 can be displayed with a pixel resolution of 160 XI20. rate.

[0224] Referring to FIG. 6D, the planning video map window 620 may provide a robot position icon 650 indicating the position of the robot 100 in the local environment. In order to cause the robot to semi-automatically or automatically navigate to the selected point, the user may click or touch the point on the displayed map 622. In some examples, when the cursor is on the planning video window 620, the user can use the mouse wheel to zoom, or when it is displayed on the touch screen 312, the user can use touch gestures to zoom.

[0225] The real-time positioning and mapping (SLAM) technology can use a laser ranging scanner, an odometer, a sonic rangefinder, or all of these instruments to map the local environment and place the robot 100 on the map. The images recorded by the robot 100 (for example, by the camera 320 or the three-dimensional image sensor 450) can be stored in an internal database (for example, the controller

500 database) and/or remote database (for example, cloud service). When the robot 100 reacquires the image in the current database, the calculation program resets the current position of the robot to the position recorded when the road sign first entered the database. This method helps offset the inherent drift of the wheel encoder odometer. The system can also utilize RFID clipping and/or triangulation of wireless access points. Further, the name or identification number of a specific room can be linked to the location on the map. Image data can be accumulated over time

CN 104898652 Β

In order to save costs or save space, the robot 100 may each use remote data storage and/or remote processing to store and/or process image data. For example, the RFID reader program can detect the RFID clip linked to the coordinates on the planning video map to identify the current position of the robot. "RFID clip" can include RFID devices or RFID "tags" as understood by those skilled in the industry. RFID clip Can be expressed as passive, active or battery-assisted passive (BAP) RFID clips.

[0226] FIG. 7 provides a schematic diagram of an exemplary robot system structure 700, which may include the robot 100 (or a part thereof, such as the controller 500 or the drive system 200), and a computing device 310 (for example, detachable or fixed attached to Head 160), cloud 720 (ie, cloud computing service), and portal 730. The computing device 310 can execute one or more robotic applications 710, which can include software applications such as safety, medication compliance, remote monitoring, behavior training, social networks, active alerts, and family management (e.g., stored in memory and in processing Executed on the device). The computing device 310 may provide communication capabilities (for example, secure wireless connection and/or cellular communication), refined application development tools, voice recognition, and person or object recognition capabilities. In some instances, the computing device 310 utilizes the interaction/COMS feature operating system, such as Android provided by Google Inc., iOS provided by Apple Inc., or other smart phone operating systems, or a dedicated robot operating system, such as RSS A2_O

[0227] Cloud 720 provides cloud computing and/or cloud storage capabilities. Cloud computing can provide Internet-based computing and provide resources, software, and data to computers and other required devices through shared servers. For example, the cloud 720 may be a cloud computing service including at least one server computing device, and the server computing device may include a service abstraction layer and a hypertext transfer protocol package on a server virtual machine instantiated thereon. The server computing device can be designed to analyze HTTP requests and issue HTTP responses. Cloud computing can be a technology that uses the Internet and a central remote server to maintain data and applications. Cloud computing can allow users to access and use the application 710 without installation and access personal files on any computer that can be connected to the Internet. Cloud computing allows relatively more efficient calculations through storage, storage, processing, and bandwidth under central control. Cloud 720 can provide scalable, on-demand computing power, storage, and bandwidth, while reducing robot hardware requirements (for example, by freeing up CPU and memory usage). The connectivity of the robot to the cloud 720 allows automatic collection of data on the operation and usage history of the robot without requiring the robot 100 to return to the base station. In addition, continuous data collection over time can produce large amounts of data that can be used for sales, product development, and support.

[0228] The cloud storage 722 can be a model of networked computer data storage, where the data is stored in multiple virtual servers usually owned by a third party. By providing communication between the robot 100 and the cloud 720, the information gathered by the robot 100 can be safely seen by authorized users through a web-based information portal.

[0229] The portal 730 may be a web-based user portal used to aggregate and/or provide information, such as personal information, family status information, and robot status information. The information can be combined with third-party information to provide users and/or robots 100 with additional functions and resources. The robotic system structure 700 can facilitate active data collection. For example, the application 710 executing on the computing device 310 may collect data and report actions performed by the robot 100 and/or humans or the environment seen by the robot 100 (using the sensor system 400). Such data can be a unique feature of the robot 100.

[0230] This involves comparing "dense data" with respect to the spatial data set versus "sparse data", and "dense characteristics" versus "sparse characteristics". It does not limit or narrow how those skilled in the art should understand the meaning of these terms. "Dense" vs. "sparse" generally refers to many data points represented by each space compared to few data points, and can especially refer to:

[0231] (i) In the context of two-dimensional image data or three-dimensional "images" that include two-dimensional data and ranges, "dense" image data includes image data that is substantially completely filled with pixels, or that can capture the original image without Loss and/or processed (including basically uncompressed, unprocessed, or lossless compressed images) rasterized image data, while "sparse" images are images that are quantized, sampled, lossy compressed, vectorized, and separated (E.g. become superpixels, nodes, edges, tables

CN 104898652 Β

Faces, points of interest, voxels) images, or the fidelity of the original capture is substantially reduced, or the pixels must be tampered with in order to reproduce the image;

[0232] (ii) In the content of two-dimensional or three-dimensional characteristics, "dense characteristics" can be occupied in a substantially unrestricted manner, resolution regarding detection orientation, all characteristics that can be detected and recorded, and/or determined by The recognition of the detector is to collect the characteristics of many characteristics (H0G, microwave) on the sub-image; the "sparse characteristics" can be purposefully limited in numbers, with characteristic input, side suppression, and/or characteristic selection numbers, and / Or the recognition that can be recognized by the detector to identify limited outliers on the image (Harris corners, edges, Shi-Tomasi).

[0233] With respect to the three-dimensional environment structure, the robot 100 can obtain an image of the scene 10 around the robot 100 when it is operating on the work surface 5, such as a dense image 701. In some embodiments, the robot 100 uses a camera 320 and/or an imaging sensor 450 (for example, a point-shaped imaging device that measures volume) to obtain a dense image 701. The controller 500 associated with the camera 320 and/or the imaging sensor 450 may associate information with the dense image 701 (for example, using data elevation or marking the dense image 701), such as accelerometer data tracing, rangefinder data, and/or Other data from the sensor system 400 along with the time stamp. In some examples, the robot 100 captures the dense image 701 of the flow sequence and uses the elevation data to mark the dense image sequence to provide the elevation dense image sequence. The cloud service 720 may process the received image data 701 and return the processed data set to the robot controller 500, which may issue a driving command to the driving system 200 to operate around the scene 10 based on the received processed data set.

[0234] The cloud service 720 can execute one of many offline methods to process the stored image data set 703 into a dense 3D map or model 705 of the scene 10 (environment), and then simplify this dense 3D map or model 705 into a 2D The height map 707 can be a two-dimensional map with height data at each point (for example, similar to a two-dimensional topographic map). In some examples, the two-dimensional height map 707 is a topographic map with X and Y coordinates and Z data. Each X and Y coordinate may have one or more Z points (ie, height data). Unlike a dense three-dimensional map where each X and Y coordinate can have a large number of Z points (for example, hundreds or thousands of Z points), each X and Y coordinate of the two-dimensional height map 707 can have less than the threshold number. Z point, for example, between 2 and 20 points (for example, 10). The two-dimensional height map 707 derived from the three-dimensional map of the table in the room may show the first Z point of the bottom surface of the desktop and the second Z point of the top surface of the desktop along each X and Y coordinates of the table. This information allows the robot 100 to confirm whether it can pass under the desktop. By reducing for each X and Y coordinates from a dense data set of Z points in a continuous range to a sparse data set of selected numbers indicating the Z point of the detection object 12, the robot 100 can receive a three-dimensional map that is higher than that used by cloud computing 720. There is a two-dimensional height map 707 of relatively small size. The robot 100 receives the two-dimensional height map 707 from the cloud 720, and the cloud 720 is the robot 100 and The related controller 500 provides navigation data for future work in scenario 10.

[0235] Other methods and features of 3D map data compression are described in R. Triebeb P. Pfaff and W. Burgard's "MultiLevel Surface Maps for Outdoor Terrain Mapping and Loop Closing"⁷⁷ ; IEEE/RSJ International Conference on Intelligent Robots and Systems, 2006 is disclosed, which is incorporated herein by reference in its entirety.

[0236] The cloud 720 provides the robot 100 with an on-demand proportion of resources that may not be otherwise practical or cost-effective on the robot 100 (eg, calculated, processed, stored, etc.). For example, the cloud 720 can provide an upgradeable cloud storage 722, which is upgraded to the first size to store and/or process a relatively large amount of data 701, which can only be used for a short time and is discarded afterwards, and then downgraded to The second size. In addition, the cloud 720 can provide computer processing capabilities for executing relatively complex calculations or "powerful" calculation programs that may not otherwise be feasible on a robot. By moving the computer processing power and memory to the scalable cloud 720, the robot 100 can use the controller 500 with relatively small computing power and memory, thus providing a cost-effective solution. In addition, the robot 100 performs real-time tasks such as obstacle avoidance (in control

CN 104898652 Β

On the controller 500 or the network board 310), however, tasks that are not real-time or time-insensitive are sent to the cloud 720 for processing and then recovered.

[0237] Cloud 720 can execute one or more screening programs (for example, optimized adjustment algorithm, RANSAC, expectation maximization method, SAM or other 3D structure evaluation calculation program) to process the stored image data set 703 into 3D display . Once the dense 3D map 705 is processed and generated or updated, the image data set 703 can be discarded from the cloud storage 722, freeing up resources and allowing the cloud 720 to upgrade accordingly. Therefore, the robot 100 does not require either on-board storage or processing of the operating storage and processing of the image data set 703 because of the use of cloud-based resources. The cloud 720 may return the processed navigation data 701 or map 707 (for example, a compressed two-dimensional height map) to the robot 100 so that it can be used for relatively simple positioning and navigation processing later.

[0238] The additional methods and characteristics of three-dimensional reproduction were published in the 5th International Conference on Three-dimensional Digital Image and Model in 2005, 3D Models from Extended Uncalibrated Video Sequences of J. Repko and M. Pollefeys: Addressing Key-frame Selection and Projective Drift'', the entire article is hereby used as a reference.

[0239] Referring to FIGS. 8A and 8B, in some cases, the robot 100 receives the occupancy map 800 of the object 12 in the scene 10 and/or the work surface 5, or the robot controller 500 generates (and can be updated) based on the imaging sensor 450 (For example, the second three-dimensional image sensor 450b) the received image data and/or characteristic depth data occupancy map 800o SLAM is a technology, which can be used by the robot 100 in an unknown environment or scene 10 (no a priori Knowledge) construct the occupancy map 800, or update the occupancy map 800 in a known environment (with prior knowledge from the given map), while maintaining the monitoring of its current position.

[0240] The controller 500 may display the diagram 622 on the user interface 605 in connection with the occupancy diagram 800 and the remote software application 601. The user interface map 622 may be partially or fully derived from the occupancy map 800. In addition, referring also to FIG. 7, the remote software application 601 may receive regular updates of the occupancy map 800 through the cloud service 720. For example, the cloud service 720 may provide the remote software application 601 with a dense 3D map or model 705 of the scene 10 around the robot 100 and/or a simplified 2D height map 707 used to generate the user interface map 622. In another example, the cloud service 720 provides a user interface map 622 to a remote software application 601 based on a dense three-dimensional map or model 705 or a two-dimensional height map 707.

[0241] Referring again to FIGS. 8A and 8B, the diagram 800 can be used to confirm a location within the environment 10 and describe the environment for planning and navigation. The graph 800 supports the assessment of the actual location by recording the information obtained from a perception method and comparing it with the current perception. The advantage of the graph 800 is that it increases the assistance for the location assessment and reduces the accuracy and quality of the current perception. The graph 800 generally shows the situation when the graph 800 is provided or generated. This does not need to be consistent with the environmental conditions when the map 800 is used. Other positioning technologies include monocular visual SLAM (MonoSLAM) and specific implementations using extended Kalman filter (EKF) for MonoSLAM solutions.

[0242] The controller 500 may perform Scale Invariant Feature Transform (SIFT) to detect and describe the local characteristics of the captured image. For any object 12 in the image, the points of interest on the object 12 can be extracted to provide a "characteristic description" of the object 12. This description extracted from the training image can later be used to identify the object 12 when trying to locate the object 12 in a test image containing many other objects. To perform reliable recognition, it is important that the features extracted from the training image are detectable even under changes in image scale, noise, and lighting. These points are usually in high-contrast areas of the image, such as the edges of objects. For object recognition and detection, the robot 100 can use SIFT to find unique key points, which are invariant to position, scale, and rotation, and are resistant to affine transformations (changes in scale, rotation, shear, and position) and lighting The change is robust. In some specific embodiments, the robot 100 captures the scene 10 or the object 12 (for example, in a different environment, from a different angle, etc.)

CN 104898652 Β

Multiple images (using camera 320 and/or imaging sensor 450) and store the images, for example into a matrix. The robot 100 can enter the stored images to identify new images through comparison, screening, and so on. For example, the SIFT feature is obtained from the input image and matched with the SIFT feature database obtained from the training image (captured in advance). Feature matching can be performed by the nearest neighbor method based on Euclidean distance. The Hough transform can be used to increase object recognition by gathering features that belong to the same object and discarding matches that were missed in the gathering process. Speeded up robust feature (SURF) can be a rough image detector and descriptor.

[0243] In addition to the positioning of the robot 100 in the scene 100 (for example, the environment around the robot 100), the robot may use the sensor system 400 to travel to other points within the connected space (for example, the work surface 5). The robot 100 may include a short-range type imaging sensor 450a (for example, installed on the bottom surface of the torso 140 as shown in FIGS. 1 and 3), which is used to survey the vicinity of the robot 100 and identify relatively close objects 12. And a remote type imaging sensor 450b (for example, installed on the head 160 as shown in FIG. 1 and FIG. 3), which is used to survey a relatively large area around the robot 100 and to identify relatively distant objects 12. The robot 100 can use the occupancy map 800 to recognize the known object 12 in the scene 10 and the occlusion 16 (for example, when the object 12 should or should not, but cannot be confirmed from the current vantage point). The robot 100 can record the occlusion 16 or a new object 12 in the scene 10 and attempt to circumnavigate the occlusion 16 or the new object 12 to determine the position of the new object 12 or any object 12 in the occlusion 16. In addition, by using the occupancy map 800, the robot 100 can confirm and monitor the movement of the object 12 in the scene 10. For example, the imaging sensors 450, 450a, and 450b can detect the new position of the object 12 in the scene 10 without detecting the object 12 in the scene 10. The location of the mapping. The robot 100 can record the position of the old object 12 like the occlusion 16 and try to circumnavigate the occlusion 16 to determine the position of the object 12. The robot 100 can compare the new image depth data with the previous image depth data (eg, graph 800) and assign the reliability of the position of the object 12 in the scene 10. The reliability of the position of the object 12 in the scene 10 can expire after the threshold of time. The sensor system 400 can update the position credibility of each object 12 after each imaging cycle of the sensor system 400. In some instances, a new occlusion 16 detected during occlusion detection (e.g., less than 10 seconds) (e.g., an object 12 missing from the occupancy map 800) may be indicative of a "living" object 12 in the scene 10 (e.g., moving The object 12).

[0244] In some specific embodiments, the second object of interest 12b located behind the detected first object 12a in the scene 10 may be regarded as the occlusion 16 in the scene 10 that was not detected at first. The occlusion 16 can be an area in the scene 10 that is not easily detected or seen by the imaging sensors 450, 450a, and 450b. In the example shown, the sensor system 400 of the robot 100 (for example, or a part of it, such as imaging sensors 450, 450a and 450b) has a field of view of the leaf (which can be any angle between 0° and 360°) Area 452 to view scene 10. In some examples, the imaging sensor 450 includes an omnidirectional optical device with a 360° viewing angle of 0 γ, while in other examples, the imaging sensors 450, 450a, and 450b have a viewing angle of less than 360° (for example, between about 45° and 180° between). In an example, where the viewing angle Oγ is less than 360°, the imaging sensors 450, 450a, and 450b can be rotated relative to the robot body 110 to achieve a 360° viewing angle. In some embodiments, the imaging sensors 450, 450a, and 450b or a part thereof can move relative to the robot body 110 and/or the driving system 200. In addition, in order to detect the second object 12b, the robot 100 can move along one or more directions around the scene 10. The imaging sensors 450, 450a, and 450b are moved by driving (for example, by translation and/or rotation on the work surface 5) to obtain the advantage of allowing detection of the second object 12b. The independent movement of robot movement or imaging sensor 450, 450a, 450b or a part of it can also solve monocular difficulties.

[0245] The credibility can be assigned to detect the position of the object 12 in the work area 5 or to monitor its movement. For example, when the occupancy map 800 is generated or updated, the controller 500 may assign credibility to each object 12 on the map 800. The credibility can be proportional to the likelihood that the object 12 is actually located in the work area 5 as indicated on the graph 800. The credibility can be confirmed by many factors, such as the number and type of sensors used to detect the object 12. For example, the contact sensor 430 can provide the highest degree of confidence, because

CN 104898652 Β

The contact sensor 430 senses the actual contact between the robot 100 and the object 12. The imaging sensor 450 can provide different credibility, which can be more credible than the progress sensor 430. The data received from more than one sensor of the sensor system 400 can be aggregated or accumulated to provide a relatively higher degree of confidence than any single sensor.

[0246] Odometer uses movement data from actuators to assess changes in position (distance traveled) over time. In some instances, an encoder is installed on the drive system 200 to measure wheel rotation and therefore the distance traveled by the robot. The controller 500 may use the odometer to assess the reliability of the position of the object. In some embodiments, the sensor system 400 includes an odometer and/or angular velocity sensor (for example, a gyroscope or IMU470) for sensing the distance traveled by the robot 100. A gyroscope is a device used to measure or maintain orientation, which is based on the principle of conservation of angular momentum. The controller 500 may each use the odometer and/or gyroscope signals received from the odometer and/or angular velocity sensor to confirm the position of the robot 100 in the work area 5 and/or on the occupancy map 800. In some instances, the controller 500 uses dead reckoning. Dead reckoning is the process of assessing the current position based on the previously confirmed position, advancing the position based on the known or assessed speed over time, and guiding the course. By knowing the position of the robot in the work area 5 (for example, through an odometer, gyroscope) and the perceived position of one or more objects 12 in the work area 5 (through the sensor system 400), the controller 500 can check the occupancy map The position or movement of the object 12 on the 800 and in the work area 5 is evaluated with relatively high credibility (compared to not using an odometer or gyroscope).

[0247] The odometer based on wheel movement has electrical noise. The controller 500 can be connected or replaced with the wheel odometer by scanning matching. The use of scan matching can provide accuracy and/or reduce the amount of calculation. In this specific embodiment, two partial images obtained using LIDAR and/or other surveying and mapping methods can be combined with a single image. Two or more partial images can be merged using known scanning positions. Alternatively, two or more partial images can be combined using the geometric characteristics of partial scans. The controller 500 can receive data from the imaging sensor 450 of the environment around the robot 100 or the scene 10 to calculate the robot movement, independently of the odometer of the wheel-based drive system 200, and through the visual odometer. The visual odometer can derive the use of optical flow to confirm the movement of the imaging sensor 450. The controller 500 can use the calculated movement based on the imaging data of the imaging sensor 450 to correct any wheel-based odometer errors, thus allowing for improved mapping and movement control. If the imaging sensor 450 cannot monitor characteristics within the captured image, the visual odometer may have the limitations of low-texture or low-light scene 10.

[0248] Other details and features of the odometer and imaging systems that can be combined with those described herein are described in U.S. Patent 7,158,317 (description of the "depth of field" imaging system) and U.S. Patent 7,115,849 (description of the wavefront coded interferometric phase contrast imaging system) Found in and its entire contents are incorporated here by reference.

[0249] Referring to FIGS. 8C and 8D, when the robot enters a new building where it will work, the robot may need to lead a tour around or provide a building map (for example, room and corridor locations) for braking navigation. For example, in a hospital, the robot may need to know the location of each patient's room, nurse's station, etc. In some embodiments, the robot 100 receives, for example, the plan view 810 shown in FIG. 8C, and can be trained to remember the plan view 810. For example, when the robot 100 is guided to surround a building, the robot 100 may record a specific position corresponding to the position on the plan view 810. The robot 100 can display a plan view 810 on the networking board 310 and when the user leads the robot 100 to a specific position, the user can mark the position on the plan view 810 (for example, using a touch screen or other positioning device of the networking board 310). The user can choose to enter a label for the marked location, like a room name or room number. When marking, the robot 100 can store the markers, the points used on the plan view 810 and the corresponding points on the robot map 820, for example, as shown in FIG. 8D. For example, the robot map 820 may be a two-dimensional plan view similar to the plan view 810. In an alternative embodiment, the robot diagram 820 may be a three-dimensional diagram including a ground plane, and the ground plane corresponds to a two-dimensional plan view similar to the plan view 810.

CN 104898652 Β

[0250] Using the sensor system 400, the robot 100 can construct a robot map 820 as it moves around. For example, the sensor system 400 can provide information about the moving distance and the traveling direction of the robot 100. The robot map 820 may include fixed obstacles other than the wall provided by the plan view 810. The robot 100 may use the robot map 820 to perform automatic navigation. For example, in the robot map 820, the "wall" may not appear to be completely vertical because of the corresponding corridors and/or the inspected packing boxes of the furniture inside the inspected various compartments along the wall. In addition, rotation and resolution differences may exist between the plan view 810 and the robot map 820.

[0251] Referring to FIG. 8E, in some embodiments, the remote software application 601 displays a mark view 660 that allows the user to place a mark 662 on the plan view 810. The plan view 810 may be the same diagram displayed in the plan view window 620, or may be a different diagram from that used internally for navigation purposes.

[0252] The user, the remote terminal, and/or the robot may add a mark on the specific position of the plan view 810 and/or the robot map 820 to mark the map position with information, such as driving obstacle area, obstacle, robot assistance, and so on. For example, the user can drag and drop the marker 662 to a specific position in the plan view 810. As described herein, the marker may include marker coordinates related to a point or area, marker information for a clear purpose of marking, the type of marker, the nature of the marker, an explanation of the connection between the user and/or the robot and the marker, and/or other related information. The information related to the mark, and finally, the mark may include a mark annotation composed of two-dimensional and/or three-dimensional diagrams or text corresponding to the mark. An example of the mark annotation is that the octagonal red stop mark related to the mark contains mark information indicating an area in which the robot cannot enter. Mark annotations can be judged by humans and/or machines. The marking coordinates can be points, lines, planes, surfaces, rolls, and/or 2.5D or mixed surfaces. The tag can be formed by a data structure with any number of additional fields and/or parameters. For example, the tag includes fields related to time, schedule, spatial coordinates, and/or triggers of predetermined functions.

[0253] As used herein, the term annotation includes text, passwords, or other verbal expressions. Therefore, the mark annotations can be pictures, graphic images, charts, hi ero annotations, and non-verbal signs. In addition, the mark annotation can be in the form of words, letters, phrases, or other textual forms. For example, the mark related to the nurse's station may include a mark annotation that contains a text representation of the nurse's name. The textual representation of the name of the nurse can be a two-dimensional text, or can be a three-dimensional text. As an option, the mark annotation related to the nurse station can be a capital letter N, or the sign of the nurse station (for example, a nurse hat or a nursing sign).

[0254] The mark 662 may include a wireless local area network (WLAN) hot mark 662a indicating an area with relatively good signal reception, and a WLAN cold mark 662b indicating an area with relatively weak signal reception. The robot 100 can use this information to navigate from one location to another, passing through areas with relatively good wireless signal reception while avoiding areas with relatively weak wireless signal reception.

[0255] The low traffic volume mark 662c indicates an area with relatively low traffic volume (people and/or robots). The robot 100 may select a travel route through an area with relatively low traffic volume instead of passing through an area with relatively high traffic volume. In addition, if the robot 100 must travel through an area with high traffic volume, the robot 100 may perform one or more specific object detection obstacle avoidance (ODOA) behaviors to successfully pass the area without collision.

[0256] The parking mark 662d indicates the location of the robot docking station. The low battery event may send a signal to the controller 500 to seek charging. The robot 100 can use the map position marked with the docking mark 662d to locate the robot docking station for charging. For example, by applying the application to resolve the deformation between the plan view 810 and the robot map 820 (FIGS. 8C and 8D), the robot 100 can confirm the corresponding robot map position 824 corresponding to the marked layout plan position 814 to navigate to the marked position and the robot insert. Docking station. Resolving the distortion may include confirming the distortion between two graphs that use the same coordinate system. The robot drawing 820 and the plan view 810 may both be two-dimensional and the same, and confirming the deformation may not require confirming the coordinate conversion between different dimensions.

[0257] Some markers 662 can be used to indicate obstacles or specific areas to cross. For example, the glass mark 662e indicates that the glass

CN 104898652 Β

The location of walls, windows, or doors. The robot 100 can use this information to avoid the marked glass structure, which may not be detected by the infrared process sensor. The ramp mark 662f indicates the location of the ground ramp. From a long distance, the robot 100 can detect that the ramp is an obstacle because it seems to have a vertical height higher than the threshold traverse height. When approaching the marked ramp, the robot 100 may perform a ramp or traversal behavior to successfully pass the ramp. The dense mark 662g indicates the location of a relatively narrow corridor or expressway. The robot 100 can avoid the area in order to avoid any restrictive situations.

[0258] The slow mark 662h indicates a position or area where the robot drives relatively slowly. The location or area may coincide with a high-traffic area. The avoidance mark 662i indicates a position or area that the robot 100 should avoid (ie, not drive through). In some embodiments, the avoidance mark 622i may depend on the mode of operation. For example, the avoidance mark 622i may only be applicable when the robot is operating in a fully automatic mode. During remote monitoring, the avoidance mark 622i can be effectively ignored by the robot. The operating room user interface (ORUI) mark 622 j indicates the location or area of the hospital operating room. The robot 100 can use the marker to find the operating room to provide remote support and/or display a specific user interface (for example, the ORUI interface) when entering the OR area. The training mark 622k can be used to mark general locations such as corridors and rooms to train the robot 100 to remember its environment 10.

[0259] The manual elevator mark 6221 indicates that the robot 100 should allow the user to assist the robot in traversing the position of the elevator entering/exiting the elevator. The smooth passage of manual elevators can be based on remote user navigation or robot-local user navigation. Regarding the remote user piloting, the remote user provides a driving command to the robot 100 (for example, using a joystick). Regarding robot-local user navigation, a person adjacent to the robot 100 can physically touch the robot 100, and in response to those touches, the robot 100 moves accordingly. The characteristics of the robot's responsiveness to user touches and those described here can be found in the application number 13/032,390 filed on February 22, 2011, which is incorporated herein by reference in its entirety.

[0260] The escalator mark 622m indicates the position of an elevator through which the robot 100 can automatically pass (in and out). The robot 100 may perform a threshold traversal behavior 512d (FIG. 5) to enter and exit the elevator in order to avoid tipping. The characteristics of the robot's responsiveness to the user's touch and those described here can be found in the application number PCT/US11/59910 filed on 2011.11.9, which is incorporated herein by reference in its entirety.

[0261] The keep right mark 622n indicates that the robot should keep the map position or area to the right. Users can place this marker along certain corridors, such as high-traffic corridors. In response to keeping the mark 622n to the right, the robot 100 may perform a wall-following behavior while the mark area is driven to stay along the wall.

[0262] After the map training, when the user needs to let the robot 100 go to a location, the user can either refer to the label/mark 622 (for example, enter a label or mark in the position text box displayed on the network board 310) or the robot 100 can A plan view 810 is displayed on the networking board 310 to the user, and the user can select a position on the plan view 810. If the user selects the marked layout drawing position 814, the robot 100 can easily confirm the corresponding robot drawing position 824 on the robot drawing 820 and can navigate forward to the selected position 814.

[0263] In some specific embodiments, the robot controller 500 may perform the first behavior 512 when operating around the first area, and then perform the second behavior when operating around the second area with the mark of the relevant robot behavior correction factor. 512. For example, when performing a human following behavior 512b, the robot controller can either terminate the execution of the behavior 512b when it reaches a map location marked with a ramp mark 662f or an escalator mark 622m, or simultaneously perform a threshold traversal behavior 512d.

[0264] If the selected position on the plan view 810 is not the marked position 814, the robot 100 confirms the corresponding position 824 on the robot map 820. In some embodiments, the robot 100 uses the existing marked positions to calculate the plan view 810 The scale size, origin mapping, and rotation between the robot map 820 and the robot map 820, and the calculation parameters are applied to confirm the robot map position (for example, using affine transformation or coordinates).

[0265] The robot map 820 may have a different orientation and scale from the plan view 810. In addition, the layout drawing may not be to scale and

CN 104898652 Β

And can have different deformations with the map area. For example, the plan view 810 produced by scanning fire evacuation maps commonly seen in hotels, offices, and hospitals is usually not drawn to scale and can even have different scales in different areas of the map. The robot map 820 may have its own errors. For example, the position on the robot map 820 can be calculated by counting wheel rotations like measuring distance, and if the ground is slightly slippery or the corners cause additional wheel rotation, inaccurate rotation calculations can cause the robot 100 to confirm the inaccurate position of the mapped object .

[0266] The way of mapping the point 814 given on the plan view 810 to the corresponding point 824 on the robot map 820 may include using the existing marked points 812 to calculate the layout point between the plan view 810 and the robot map 820. The local deformation (in the same two-dimensional coordinate system) of the area (for example, within a threshold radius). The method further includes applying deformation calculations to the layout drawing point 814 to find the corresponding robot drawing point 824. The reverse is also possible, if you start with the point given on the robot drawing 820 and want to find the corresponding point on the plan view 810, For example, to ask the robot its current location.

[0267] Any kind of marking mode and data structure can be used. For example, the tag may include characteristics in the form of key-value pairs, which may specify the purpose of the tag, the tag parameters, and the tag characteristics (usually "tag information"). Table 1 below provides a specific example.

[0268]

<td>Area name</td><td>type of data</td><td>description</td>

<td>Tag ID</td><td>Integer</td><td>Mark the ID that entered the mark table above</td>

<td>name</td><td>text</td><td>parameter name</td>

<td>Numerical value</td><td>text</td><td>Parameter value</td>

[0269] Table 1

[0270] Region-related markers may have characteristics associated with them, which specify their purpose and parameters that affect region-related behavior. These key-value pairs can be stored using a data structure similar to the example in Table 2 below:

<td>[0271]</td><td>Area name</td><td>type of data</td><td>description</td>

<td></td><td>Area ID</td><td>Integer</td><td>Enter the ID of the region table above</td>

<td>[0272]</td><td>name</td><td>text</td><td>parameter name</td>

<td></td><td>Numerical value</td><td>text</td><td>Parameter value</td>

[0273] Table 2

[0274] The data structure of each mark may include mark coordinates and mark information, and the mark information may include mark annotations (for example, two-dimensional and/or three-dimensional illustrations of the mark). Table 3 below provides a specific example of a tagged data structure.

[0275]

<td>Area name</td><td>type of data</td><td>description</td>

<td>ID</td><td>Integer</td><td>A globally unique identifier marked in the database</td>

<td>Figure ID</td><td>Integer</td><td>Mark the identifier of the robot map to which it belongs</td>

<td>Timestamp</td><td>float</td><td>Timestamp indicating when the marker was created</td>

<td>Posture X</td><td>float</td><td>Mark the X coordinate in the robot map coordinate system</td>

<td>Posture Y</td><td>float</td><td>Mark the Y coordinate in the coordinate system of the robot graph</td>

<td>Posture Z</td><td>float</td><td>Mark the Z coordinate in the robot map coordinate system</td>

CN 104898652 Β

<td>Posture Xr</td><td>float</td><td>Mark the rotation about the X axis</td>

<td>Posture Yr</td><td>float</td><td>Mark the rotation about the Y axis</td>

<td>Posture Zr</td><td>float</td><td>Mark the rotation about the Z axis</td>

<td>name</td><td>Takemoto</td><td>Human readable identifier of the tag</td>

<td>annotation</td><td>image</td><td>2D and/or 3D graphics</td>

[0276] Table 3

[0277] As described herein, markers may be related to regions on the map, rather than specific points. The relationship between the tag information and the tag can be many-to-one. Table 4 below provides a specific example of the data structure of a region-related tag.

<td>ID</td><td>Integer</td><td>The globally unique identifier of the region in the database</td>

<td>Figure ID</td><td>Integer</td><td>Identifier of the robot map to which the region belongs</td>

<td>Hour-Η</td><td>float</td><td>Indicatively, the timestamp of the time when the zone was created,</td>

<td>Posture X</td><td>float</td><td>The X coordinate of the area centroid in the same coordinate system as the robot figure</td>

<td>Posture Υ</td><td>, Q-L</td><td>The centroid of the area is in the Y coordinate of the same coordinate system as the robot map</td>

<td>Posture ζ</td><td>float</td><td>The area centroid is in the Z coordinate of the same coordinate system as the robot map</td>

<td>Posture Xr</td><td>float.</td><td>Rotation of the region on the X axis:</td>

<td>Posture Yr</td><td>float</td><td>Rotation of the area on the Y axis</td>

<td>Posture Zr</td><td>float</td><td>Region about the rotation of the Z axis</td>

<td>name</td><td>text.</td><td>Human readable identifier of the region</td>

<td>annotation</td><td>image</td><td>2D and/or 3D graphics</td>

[0280] Table 4

[0281] In some instances, the geometries of regions can be broken down into the centroids and offsets of their components to allow rapid updates of the position and rotation of many objects. When CPU resources allow, the limit box of the final coordinates (the face point about the center of mass, converted from the pose of the center of mass to the coordinate system of the graph) can be indexed using R*-tree or similar data structures to quickly retrieve based on geometric constraints . The points of the surface containing the area can be stored clockwise (or counterclockwise) to facilitate point-to-surface testing based on the ray-tracing calculation program.

[0282] As an example, the mark indicating the area of the slow-traveling zone may have a data structure as provided in Table 5 below.

[0283]

<td>name</td><td>Numerical value</td>

<td>Types of</td><td>Speed limit zone</td>

<td>Subtype</td><td>obvious</td>

<td>Max X speed</td><td>0.75</td>

<td>Maximum Y speed</td><td>0</td>

<td>Maximum 0 speed</td><td>0.75</td>

CN 104898652 Β

[0284] Table 5

[0285] Table 5 shows an example of a slow-traveling zone mark, in which a speed limit is set based on a standard limitation related to the area itself. As an option, it may be an area defined in the manner described above, and the robot can understand that the area is a slow-moving area. For example, in Table 6 below, the robot can understand that the area defined as the intersection is a slow-moving area and reduce its speed to a predetermined speed.

[0286]

<td>name</td><td>Numerical value</td>

<td>Types of</td><td>Speed limit zone</td>

<td>Subtype</td><td>intersect</td>

[0287] Table 6

[0288] Marks related to points or regions on the map may include mark coordinates, mark information, and any kind of mark annotations. In addition, the mark can be supplemented with any data type, including those shown in Table 1-6 above. The terms labeling information and labeling annotations involved here are separate components. However, according to various embodiments, the mark annotation may be a part of the mark information. To be precise, the data structure may or may not include explicit mark annotation fields. In contrast, the tag information field of the data structure may contain tag comments.

[0289] FIG. 8F provides an exemplary arrangement 800f for manipulating the robot 100 to navigate around the environment using a plan view 810 and a robot map 820. This operation includes accepting the plan view 810 corresponding to the environment of the robot 100 at 802f, moving the robot 100 in the environment of 804f to the layout position 812 on the plan view 810, and recording the robot map 806f corresponding to the environment and generated by the robot 100. The robot map position 822 on the 820, use the recorded robot map position 822 and the corresponding layout plan position 812 to confirm the deformation (two-dimensional) between the 808f robot map 820 and the plan view 810, and apply the target layout confirmed by 810f The deformation of the map position 814 confirms the corresponding surface robot map position 824, thus allowing the robot to navigate to the selected position 814 on the plan view 810. In some embodiments, this operation includes using the marked position to confirm the scale size, origin mapping, And the rotation between the plan view 810 and the robot map 820, and determine the robot map position corresponding to the selected target layout map position 814. This operation may include applying an affine transformation to the confirmed scale size, origin mapping, and rotation to determine the position of the robot figure. Any of the above operations can be repeated any number of times in order to increase accuracy and/or efficiency. For example, moving the robot 100 in the 804f environment and recording the 806f robot map position can be repeated Many times come to gather enough relevant points for the conversion and calculation between the subsequent layout drawing and the robot drawing.

[0290] Other details and features incorporated herein are found in the PCT application number: PCT/US11/60935 filed on November 16, 2011, which is incorporated herein by reference in its entirety.

[0291] Referring to FIGS. 9A and 9B, in some specific embodiments, the remote software application 601 displays the hybrid three-dimensional imaging map 622b (hybrid map) in the map window 620. The mixed image 622b may be a mixture of the remote view 612 displayed by the remote video input window 610 and the two-dimensional, top-down plan view 810 of the image 622a (FIG. 6D) displayed on the plan view window 620, for example. Figure 9A shows a remote video view that the user can see when the robot is in the corridor. 9B shows a hybrid view 622b on which the plan view 810 is partially covered and modified to fit the remote view 612, indicating the room number and/or room type of the area within the field of view of the robot 100. When watching real-time video input, the user can place the cursor on the window and start moving the turbine upwards. During the transition process, the projected video view (coming from the camera 320 on the robot 100) gradually transitions between the remote video view 612 and the diagram 622. The graph 622 is completely deformed at the beginning of the transition to the surveying and mapping projection remote video view 612, and gradually returns to its undeformed view at the end of the transition. So if the mouse wheel is 30% scroll up, then the user will see an unremoved image containing 70% video and 30%> picture, and the video part is

CN 104898652 Β

30% >Undeformed, and the picture shows 70% deformation. This specific embodiment allows a single video to smoothly project real-time remote video views 612 and 622.

[0292] To provide the hybrid map 622b, the remote software application 601 can use the robot map position 822 and the corresponding layout map position 812 on the recorded robot map 820 to confirm the deformation between the remote view 612 and the plan view 810 (two-dimensional Between the coordinates and the three-dimensional coordinates), and apply the confirmed deformation to fit the plan view 810 to the remote view 612. In some embodiments, confirming the deformation includes confirming the scale size, origin mapping, and rotation between the plan view 810 and the remote view 612, for example, confirming the scale size, origin mapping, and rotation by applying an affine transformation. Confirming the deformation between the two-dimensional plan view and the three-dimensional video input may include confirming the coordinate conversion between different coordinate systems.

[0293] Referring to FIGS. 6D and 10A-10E, in some embodiments, the user interface 605 provides a foresee command 624, which causes the foresee view 612a to be displayed in the image window 620, a dedicated independent window, or some other window. When driving the robot 100, the user can invoke the foresight command 624 that causes the robot 100 to stop physically moving, and the remote software application 601 generates and displays the foresight view 612a, which provides a projected view suggesting the driving route of the robot, or the robot 100 will continue to move along. Move along its driving route. This can be done by using map data such as the position of the wall and constructing a projected "virtual reality" view based on the actual position of the robot 100. For example, the remote software application 601 may use the plan view 810, the robot map 820, and/or the stored image data 701 to construct the foresight view 612a. Regarding the use of the cloud computing service 720 in the example shown in FIG. 7 for the robot system, the remote soft magnetic application 601 and the random robot 100 can be connected to the cloud computing service 720, so that the stored image data 701, three-dimensional map 705, And/or the foresee view 612a of the two-dimensional height map 707 (or a 2.5D hybrid map as an option), and then provide the foresight view 612a to be presented on the map window 620. This embodiment allows the remote software application 601 to utilize cloud computing's scalable computer processing and data Storage capabilities (for example, cloud computing 720 can be upgraded at will to process data and later downgraded), thus executing remote software applications 601 for computing devices to reduce processing and memory requirements.

[0294] FIG. 10A shows an exemplary remote view 612 of the remote video input window 610 of the remote software application 601. FIG. 10B shows a complementary diagram 622 displayed on the diagram window 620. The graph 622 provides the current position of the robot 100 as indicated by the robot icon 650 following the camera field of view area 322 of the robot camera 320. 10C and 10E provide an exemplary foresee view 612a displayed on the remote video input window 610. The remote video input window 610 may continue to display the remote view 612 from the robot camera 320 in the picture-in-picture window, for example, placed in the corner of the remote video input window 610. 10D and 10F provide an exemplary diagram 622 displayed on the diagram window 620. When the foreseeing command is executed, the remote software application 601 may present the robot icon 650 on the current position of the robot along with the robot camera field of view area 322. In addition or as another As an alternative, the remote software application 601 may move along the foreseen route along with the virtual robot icon 650a presented in the expected foreseeable camera field of view area 322a on the plan view window 620.

[0295] In some embodiments, when the user drives the robot 100 along the corridor using the joystick associated with the remote software application 601, the user can invoke the foreseen command 624 (for example, by selecting the corresponding button on the user interface 605 Or joystick). For example, at a position 50 feet away from the turn of the corridor, the user may invoke the foresee command 624, which causes the generation of the foresight view 612a and stops further movement of the robot 100 along the corridor. However, the user can continue to virtualize the mobile robot 100 in the foreseeable mode. The user interface 605 can display the foreseeable view 612a of the same corridor in the same location (for example, in a three-dimensional mode). When the user drives forward in the foresight mode, continues for 50 feet, turns left, and continues to drive, the user can see the location of the room and other corridors along the route of the three-dimensional mode/foresight view 612a. In some instances, for the first 30 feet of the "virtual" drive, the remote software application 601 can display the actual view (from a fixed physical robot, further magnified and projected-deformed to match the virtual position) and 3D mode/predicted view Mix of 612a.

CN 104898652 Β

[0296] FIG. 10G provides an exemplary arrangement 1000 for performing the operation of the remote software application 601. The operation includes turning on 1002 foresight mode (also known as flight mode) and checking 1004 the actual positioning of the robot 100, such as the current position of the robot 100 Posture and/or coordinates. The robot 100 can confirm its position based on the sensor signal received from its sensor system 400 and then send the position to the remote software application 601 and/or cloud computing service 720. This operation further includes generating 1006 a virtual position of the robot 100 and/or posture. The remote software application 601 or the cloud computing service 720 can use the dynamic mode 570 (Figure 5) linked to the robot 100 and the image data 701 (Figure 7) (for example, point image data for measuring the volume) to generate virtual positioning and/or posture. This operation may include accessing 1008 the three-dimensional rendering data corresponding to the confirmed virtual robot positioning and generating 1010 the three-dimensional rendering of the robot 100 and/or the local environment around the robot 100. This may result in access to the locally or remotely stored image data 701 (for example, point-shaped image data for measuring volume) and/or three-dimensional map 705 stored in the cloud storage 722 to form a local three-dimensional model/forecast view 612a, which can be applied by remote software The program 601 is displayed on the remote video input window 610. In addition, this can cause the generated three-dimensional model of the robot 100 to be generated by the virtual machine The robot icon 650a and the anticipated camera shooting area 322a are shown on the image window 620. This operation may include updating the foresight view 612a or the first person view (POV) displayed at 1012 and updating the virtual positioning/posture of the robot 100 when the robot 100 is virtually operating in the foresight mode. Steps 1008-1014 can be repeated (e.g., on a period basis) until 1016 foresee/airplane mode is terminated.

[0297] Referring to FIG. 11A, in some specific embodiments, the user interface 605 of the remote software application 610 displays a remote navigation view 612b on the remote video input window 610 (or other windows). The remote navigation view 612b allows the navigable area 616 to be presented on the real-time video input of the remote view 612. The user can switch between the remote view 612 and the remote navigation view 612b. The navigable area 616 may be confirmed based on the plan view 810 and/or the robot map 820. The navigable area 616 can be represented as a closed area with the field of view of the robot cameras 320 and 450 except for obstacles. In addition, the navigable area 616 may be filled with colors or other signals that communicate with the user, and the navigable area is free of obstructions or other obstructions.

[0298] The navigable area on the layout plan can be emphasized based on the information of the robot's internal obstacle map. In a specific embodiment, the navigable area can be recognized as white pixels on the image. The robot can return its position on the robot map and the position and orientation of the 3D depth camera. The processor can use the position of the robot and the movement of the head camera (for example, the pan and tilt angle) to confirm the pixels on the video screen that represent the ground plane. In other words, the processor can use the projection of the robot position and the video input to calculate the coordinates for each ground level pixel on the video input. The navigable area marked by the white pixels can later be overlaid on the ground plane pixels of the video input. Therefore, the robot and/or the user controlling the robot can recognize the navigation area by following the white pixels. In alternative embodiments, pixels of any color or other identifying marks can also be used. Alternative data structures or tags can be used to replace white pixels. Specifically, the coordinates of the navigable ground plane pixels of the POV from the robot can be marked by any method, as long as the robot is designed to recognize them.

[0299] The user can select the robot target 618 in the navigable area 616, which causes the remote software application 601 to issue a driving command to the robot 100 to move to a position corresponding to the selected robot target 618. In the example shown, the remote video input window 610 of the user interface 605 provides a remote navigation view 612b of the robot 100 in the hospital room. The remote software application 601 can use the plan view 810, the robot map 820, the 3D map 705, the 2D (or 2.5D hybrid) height map 707, and/or the stored image data 701 to determine the selected robot on the remote navigation view 612b The deformation between the target 618 and the corresponding robot map position 824 (within the same dimension and/or between different dimensions). The remote software application 601 can then issue a driving command to the robot 100 to automatically or semi-automatically manipulate to the robot map position 824, and use its sensor system 400 and behavior system 510a to avoid any obstacles, such as moving people.

[0300] In one example, the graph may be returned from the robot API schedule as an image, such as a PNG, JPG, or TIFF image. machine

CN 104898652 Β

The robot can process the image to detect pixels (for example, black pixels) that form the outline of the obstacle in the image. A curve suitable for the calculation program can be used to process the pixels that form the outline of the obstacle. The resulting curve can later be used to generate obstacle maps. Additional processing can be done to further improve the obstacle detection and/or improve the accuracy of the contour of the obstacle that the curve is suitable for detection. For example, if the curve is closed and forms a shape similar to a circle, the obstacle diagram can simply use a circle instead. Similar ideas can be applied to shapes like rectangles or ellipses, people, faces, and/or objects that approach a database of known object shapes from various perspectives.

[0301] The user interface 605 may provide a progressive sensor window 670 displaying approaching obstacles in the sensor field of view areas 442 and 452 (eg, in the three-dimensional image sensor field of view area 452 and/or the laser scanner field of view area 442).

[0302] In some embodiments, the user may use the avoidance marks 662 and 662i to mark the protected area/area on the remote video input window 610 and/or the plan view window 620 (not shown). The protected area can be handled by the robot 100 as the object 12, and therefore, the protected area can be avoided during automatic navigation. Protected areas can be used to help create a safe distance around vulnerable equipment, or to ensure that the robot avoids other areas. The user can place the avoidance marks 662 and 662i on the mark view 660 on the plan view 810 or the remote navigation view 612b. In addition, the user can place other marks 662 on the remote navigation view 612b. The remote software application 601 can determine the deformation between the remote navigation view 612b and the plan view 810 and/or the robot map 820 and update the robot map 820 accordingly.

[0303] For example, confirming the deformation between the plan view and the video input may include generating conversion measurements between coordinate points of any navigation view 612b, plan view 810, and/or robot map. Similar to the coverage of restricted areas on the ground plane of the video input, the ground plane of the two-dimensional map can effectively map the coordinates to the detected ground plane of the video input provided by the robot.

[0304] FIG. 11B shows a flowchart of an exemplary arrangement 1100 of method operations of a robot navigating (eg, semi-automatically) to a selected robot target 618. The method includes identifying 1102 the navigable area 616 within the field of view areas 322, 442, and 452 of the robot 100. The identification of the navigable area 616 is performed using the sensor system 400 of the robot 100. The method also includes visually indicating 1104 the navigable area 616 on the user interface 605, for example by displaying an enclosed area (eg, emphasized border, filled with color or pattern) on the remote navigation view 612b. The method may include receiving 1106 a user's selection of the robot target 618 and confirming 1108 whether the robot target 618 is within the recognized navigable area 616. If the robot target is outside the recognized navigable area 616, the method includes prompting 1110 that the user is a valid robot target 618 in the navigable area 616. If the robot target 618 is within the recognized navigable area 616, the method may include confirming 1112 The route to the robot target 618. This can cause confirmation of the deformation between the remote navigation view 612b and the robot map 820 and later determine the robot map position 824 corresponding to the selected robot target 618. The method includes allowing the 1114 robot 100 to navigate (automatically or semi-automatically) to the robot target 618.

[0305] FIGS. 11C and 11D show an exemplary remote navigation view 612 in which the user selects a robot target 618 (actual or perceived by the robot 100) beyond the navigable area 616 or on the obstacle 1120. In the example shown in FIG. 11C, the user selects the robot target 618 in the perceived obstacle 1120a and the ramp 1122±. From a long distance, the robot sensor system 400 can recognize the ramp 1122 as an obstacle, because it comes from a long distance The viewing ramp 1122 may have a perceived height above the threshold traversal height of the robot 100. In addition, the robot behavior system 510a can execute the OD0A behavior 512c to respond to the sensor event. The reason is that the sensor signal of the sensor system 400 indicates that the obstacle has a height higher than the threshold traverse height. Using the plan view 810 and/or the robot map 820, the robot 100 can confirm.

[0306] Although the ramp 1122 is within the navigable area 616, the remote software application 601 may determine that the robot target 618 on the ramp 1122 is an unsafe stop position. The remote software application 601 can display a warning dialog box 1130, indicating

CN 104898652 Β

The selected robot target is an unsafe stop position. As shown in the example, the warning dialog 1130 indicates that the user has selected the ramp 1122 as the robot target 618 and provides an alternative robot target 619 just in front of the ramp 1122. Stopping the robot 100 on the ramp 1122± can be dangerous to both the people near the robot 100 and itself, if the robot overturns or rolls down the ramp 1122. By deciding that the robot target 618 is on the ramp 1122, the remote software application 601 Such a robot target 618 can be prohibited and/or a safe alternative target 619 can be provided, in this case in front of the ramp 1122.

[0307] Referring to FIG. 11D, when the user selects an actual obstacle 1120b, the remote software application 601 may display a warning dialog box 1130, indicating that the selected robot target 618 is outside the navigable area 616 or is an obstacle 1120. As shown in the example , The warning dialog box 1130 indicates that the user has selected the obstacle 1120b as the robot target 618 and provides an alternative robot target 619 just in front of the obstacle 1120b.

12, in some embodiments, the remote navigation view 612b displayed by the user interface 605 of the remote software application 601 allows the user to specify the robot path 652 of the selected robot target 618 within the navigable area 616. The user can use a variety of input devices to specify the robot path 652. For example, on the touch screen display, the user can drag his/her finger or pen tip from the robot icon 650 indicating the current robot position to the robot target 618. In another example, the user can drag the robot icon 650 (for example, using a mouse or touch gesture) to the robot target 618 along a specific robot path 652. As shown in the example, the user can select the set path button 1202 on the user interface 605 to allow the user to indicate that the gesture performed in the navigable area 616 should be understood as the robot path 652. The user can track the robot path 652 in the remote video input window 610. Similarly, the user can select the robot path 652 in the plan view 810 to be displayed like the two-dimensional image 622a in the image window 620. After setting the robot path 652, the user can press the start button 1204 to make the robot 100 move. Similarly, the stop button 1208 can be used to stop the activity of the robot 100. The clear path button 1206 can remove or eliminate the robot path 652 set from the remote navigation view 612b.

[0309] The display window may include a fly-out icon panel displayed by a mouse swiping over. For example, the icon panel can "fly" out from the upper left of the window. The icon panel allows the user to select manual drive, click drive, and head activity icons. In a specific embodiment, the user can use the space bar to switch icons. Manual driving may allow the user to click on the target and/or click and drag the path. After drawing the path on the map, the user can right-click and select "Save Path" from the pop-up menu. They can name the path. After that, the user can "load the path", and the robot will navigate to the starting point of the path, and then follow a specific path to the target. The path can be stored as a marker data structure, including marker coordinates and marker annotations. The marked path includes multiple stops along the road. When drawing a path, the user can indicate stops along the path. In some embodiments, the site may be represented by a mark annotation including a stop sign. Later, when traversing the path, once the station is reached, the robot can flash a "Stop" button and the path can be changed brighter and/or illuminated. At this point, the user can perform a consultation and do a partial drive, and then click "Go" to return to the path. Therefore, the doctor can save a path for his night tour, patrolling all rooms and stations in a preferred sequence and pre-arranged route.

[0310] In the head mode, the user can draw a frame or outline in a part of the video input to focus the head (upper part) of the robot to the center of the frame or objects within the frame. In addition, the user can click on the position to change the heading (upper part) of the robot and/or the heading of the entire robot. Various buttons and peripheral control switch keys can be used independently to control the base (lower) and head (upper) of the robot. For example, holding down the shift key in the head mode can change the cursor on the screen to a hand icon and allow the user to grab and drag the view of the head.

[0311] In some embodiments, the star icon can be used to control the navigation of the robot. The star icon can be displayed in any view and can be selected by the user to change the direction and/or speed of the robot. Alternative icons other than the star icon are also available.

[0312] Returning to FIG. 12, the virtual joystick window 680 can provide other input devices to specify the ideal path 652 or manually control the robot 100. The virtual joystick window 680 can display the robot orientation indicator 682 and navigation guidance 684. The user can use navigation Guide 684 to control the speed of the direction of the robot 100. The virtual joystick can assist the control of the robot 100 by using a device that may not have a mouse or a commonly used joystick, such as a tablet computer.

[0313] The "stitched" video image may be displayed on the virtual joystick window 680. The "stitching" video and images can be generated using real-time cameras 320, 450 facing down in front of the robot 100 and real-time cameras facing down in the rear of the robot 100. The user can grab (for example, use a mouse or touch gesture) and drag on the robot activity indicator 686 to specify the direction and driving speed of the robot activity. The advantages of driving the robot 100 from the virtual joystick window 680 are more than using the remote video input window 610 to drive based on a mouse or a virtual joystick. In particular, the community can reduce lens distortion, lack of depth information, and perception problems based on the rotation of the robot head 160, which can be experienced by the user when using the video input displayed on the remote video input window 610 to drive the robot 100.

[0314] In addition to allowing the user to specify an ideal path within the remote video input window 610, the user may specify the route path displayed on the map 622 of the map window 620. Specifying the ideal path 652 in the plane view window 620 may allow the robot 100 to travel a longer distance, and thus allow the user to perform other tasks freely when the robot 100 is navigating. Various controls may also be provided to manipulate the zoom and display area of the graph 622 shown in the graph window 620. The ideal zoom can be specified using the slider 1120, and the ideal area can be displayed using the region pan control 1212.

[0315] Therefore, non-professional users may be able to use any combination of navigation methods and controls to navigate from one location to another. For example, in a long journey, the user can click on the target on the plan view and the robot can automatically navigate to the selected location. In the middle distance travel, the user can select the position within the robot's field of view in the video window as the target. During a short journey, the user can use a mouse, joystick, virtual joystick, or meta joystick to manually control the robot's navigation path, rotation, head movement, and the like.

[0316] FIG. 13 shows an exemplary user interface 605 of the remote software application 601 with a maximized remote video input window 610, which displays the remote video input window 610 that receives the hyper-mark 1310 and/or the environment-sensitive commands that can be selected by the user. Long-range navigation view 612b. The user interface 605 includes a partial video window 630 and a plane view window 620 covering the remote video input window 610. As shown in the example, the plan view window 620 displays a three-dimensional (three-dimensional) image 622c. The three-dimensional map 622c can be utilized by the user to cause the robot 100 to semi-automatically navigate to the robot target 618 selected on the three-dimensional map 612c. In some embodiments, the virtual three-dimensional grid 1302 is displayed on the remote navigation view 612b. Using the determined deformation between the plan view 810 and the robot map 820, the remote software application 601 can determine the position of the ground surface input by the real-time video to overlay the three-dimensional map 622c. The user can select the square 1304 as the robot target 618 on the virtual grid 1302 to cause the robot 100 to automatically navigate to the selected square 1304. The virtual three-dimensional grid 1302 may allow for improved accuracy in positioning the robot 100.

[0317] As the example shows, many hyper-marks (marks) 1310 provide environment-sensitive actions that are displayed and available to the user. The environment-sensitive actions include an approach command 1312 and a follow command 1314. These environment-sensitive actions can be generated when a person is recognized in the field of view areas 322, 442, and 452 of the robot 100. The user can refer to the approach command 1312 to place the robot 100 in front of the person 1330. The proximity command 1312 can be used by the robot behavior system 510a to cause the execution of the proximity behavior 512a (FIG. 5), whereby the robot 100 uses its sensor system 400 (for example, using facial recognition) to recognize the person 1330 and drives to face the recognized person 1330 The user can invoke the follow command 1314 to drive the robot 100 behind the person 1330 and follow at a distance of 3 feet. The follow command 1314 can cause the follow human behavior 512b (FIG. 5) to be executed by the robot behavior system 510a, thereby using its sensor system 400 (for example, using facial recognition) to recognize the person 1330 and drive to follow the recognized person

CN 104898652 Β

1330. In some examples, the robot 100 may use facial recognition programs to detect people in its field of view areas 322, 442, and 452. The label 1340 may be displayed to identify the person. For example, the information may include name, job title, occupation, address, business address, email address, web page address, user manual, and so on.

[0318] The remote software application 601 may determine the deformation between the displayed two-dimensional image 622a and the first-person video input captured by the robot camera 320. Determining the deformation may include determining the coordinate transformation between a two-dimensional map and a three-dimensional "map." When the user places the mark 662 and/or the super mark (which may include the mark) 1310 on the remote view 612 of the remote video input window 610 or the two-dimensional image 622a of the graph window 620, the remote software application 601 can match the marks 662 and/ Or the mark image coordinates related to the super mark 1310 are deformed to determine the respective corresponding video coordinates or plan view coordinates, and the determined video or graph view coordinates are used to respectively overwrite the mark annotations related to the mark 662 or the super mark 1310 to the remote display View 612 (ie, first-person video input) or diagram 622. In many specific embodiments, the three-dimensional reproduction of the mark annotation can be dynamically reproduced based on the current position of the remote robot and the projection of the mark on the video input. Therefore, when the position of the robot and/or the projection of the real-time video input changes, for example, when the head (upper part) of the robot is translated or tilted, the mark annotation can be reproduced dynamically. For example, marker annotations corresponding to ramps can be overlaid relative to the ground in the video input. Similarly, the marking annotations related to the object on the wall can be covered with respect to the object or the wall.

[0319] As described herein, the tag may include tag information including a robot motion correction factor. The mark can be understood by a robot manipulator, a local terminal, or a remote robot and causes the robot to perform a predetermined action. For example, the robot motion correction factor may instruct the robot not to enter a specific area, travel through certain areas slowly, travel through certain areas quickly, be extra careful, and/or perform other actions. General signs can include any kind of information, such as the availability of wireless communication signals, the speed at which the remote robot should travel, the location of the point of interest, the location of the person, the location of the docking station, the location of the rest area, the location of the glass wall, the ramp The position of the object, the position of the object, the optimal route through the dense area, the optimal route through the crowded area, and the actions that the remote robot should perform. Marks can be created by the user, automatically created by the terminal, created automatically by the robot, and/or created in response to historical data collected by the terminal and/or robot.

[0320] The robot may include a marker recognition system designed to recognize markers with marker coordinates encountered along the navigation path. The robot can "encount" the marker when the marker coordinates are in the robot's local perception space and/or the marker coordinates involve an objective, planned navigation path or a navigation path plan. Therefore, the marker recognition system can "encount" the marker along the navigation path, even if the robot has not yet approached and/or may never approach the marker coordinate of the marker.

[0321] The robot and/or the remote terminal may consider markings or potential markings that can affect the navigation path when determining the navigation path for the robot. Therefore, the marker recognition system can be used to recognize markers with marker coordinates estimated to be along the potential sailing path during the course of determining the navigation path. For example, some potential navigation paths may be used to reach the desired target and the selection of the navigation path to be used may depend on the markings related to each potential navigation path. The robot chooses to identify relevant markers among multiple potential navigation paths to determine which navigation path will provide the best wireless connectivity. Other factors such as ramps, elevators, distance, congestion, objects,

[0322] As shown in the exemplary user interface 605, the dashboard window 640 provides battery charge status, wireless signal strength indicators, and a robot outline, which will light up when repairs are required. The options window 690 allows the user to disconnect or park the robot from the docking station and set software and/or robot options.

[0323] Referring to FIG. 14, in some specific embodiments, when performing the following human behavior 512b, the robot 100 can detect, track, and follow the human 1330. Since the robot 100 can use the neck 150 to translate and tilt the head 160, the robot 100 can use the second three-dimensional image sensor 450b as a reference to maintain the corresponding field of view area 452 on the person 1330. Besides, since the head

160 can move relatively faster than the base 120 (for example, using the drive system 200), the head 160 (and the related second three-dimensional image

CN 104898652 Β

The image sensor 450b) can track the person 1330 faster than by rotating the robot 100 appropriately. The robot 100 can drive to the person 1330 to keep the person 1330 within the threshold following distance range Df (for example, corresponding to the sensor field of view). In some examples, the robot 100 turns to face the person/user 1330 when tracking the person 1330. The robot 100 can use speed commands and/or signpost commands to follow people 1330.

[0324] Additional and features regarding person recognition and person following are found in the PCT application No. PCT/US11/35488 filed on May 6, 2011, which is incorporated herein by reference in its entirety.

[0325] FIGS. 15A and 15B show alternative three-dimensional diagrams 622c and two-dimensional diagrams 622a that can be displayed on a plan view window 620. The plan view window 620 includes hypermarkers 1310 related to various information and can be used for Cause the robot to automatically navigate to a specific target. The super mark 1310 may include various information related to the location or information of the patient. The user can add tags 1502 or marks 1504, such as personal notes, shared notes, sketches, drawings, and so on. The robot position 1510 is also recognizable. The user can specify the robot target 1512, such as a nurse's station. The robot 100 can automatically navigate to a specific robot target 1512.

[0326] The remote software application 601 may display information on the remote video view 612 and/or graph 622 indicating the physical area of interest. For example, a small arrow that looks like "Pharma" on the additional light bulb can indicate the location of a pharmacy. The bulb may include indicia. For example, the tag may include tag coordinates indicating where the word "Pharma" should be displayed; tag information, such as related information related to a pharmacy; and tag annotations, such as a two-dimensional and/or three-dimensional illustration of the word "Pharma". In some examples, the user can determine which information is useful to nearby rooms by placing the mouse on it or making gestures in the area, resulting in any corresponding available information being displayed. With this information, the user can quickly select a destination (for example, a pharmacy) by selecting the robot target 618 on the long-range navigation view 612b (Figure 12).

[0327] For example, according to an example, the robot or the remote terminal can retrieve the marker coordinates corresponding to the marker associated with the robot map. Use robot positioning to identify marks that are very close to the robot. Marks in the robot's field of view can be identified using the orientation of the robot's head (upper part). The robot and/or the remote terminal can then calculate a series of coordinates for all the pixels on the video screen and propose marker annotations related to each marker in the line of sight based on the position of the robot and the current head orientation by the robot ( Translation and/or tilt). According to some specific embodiments, Denavit-Hartenberg parameters (DH parameters) can be used as a standard coordinate system for the spatial connection between the video input and the plan view.

[0328] Referring again to FIG. 8E, the marker view 600 of the user interface 605 allows the user to place a marker 662 on the plan view 810 to indicate the location of interest and/or to mark the plan view 810 with information, such as obstacles, suggested robot travel routes, etc. . Also referring to FIG. 13, the user can place a hypermarker 1310 on the long-range navigation view 612b to mark the location with environmentally sensitive information. The map data source 1620 may store markers and hyper-marker information (eg, location, marker identifier, marker content) along with the layout plan and/or robot map information. As used herein, a hyperlabel may appear as a label and uses a similar data structure as the label as described herein.

[0329] In addition to or alternatively allowing the user to place the mark 662 and the hyper mark 1310 on the user interface 605, the user can enter the user-specific hyper mark 1310 during the operation of the robot 100. The user can refer to a command that allows the hyper mark 1310 to be inserted into the current robot position . Another mission may allow the removal of the super mark 1310. Further, other users (for example, nurses) may allow the addition of a super mark 1310, which may be shown to one user of the robot 100. The "nurse map application" may display a top-down map or marked view 660 that allows temporary over-marking 1310 placement, for example, to identify rooms of interest for a doctor about to log in. In addition, some hyper-tags 1310 may be user-specific and/or time-specific. For example, a stroke patient in a room may be showing signs of deterioration. The nurse can search the "Nurse Map Application" to find the room on the map and key in the hyperlabel 1310. The nurse can fill in the hyper-label as follows: hyper-label-name = worsening stroke patient, user specific = Dr. Reynolds, continuous

CN 104898652 Β

Time = 1 hour. Therefore, if Dr. Reynolds logs in within the next hour, he will see the hypermark 1310 associated with the patients room on the map, which also indicates "deterioration of stroke patients." When approaching the flank, he can also see the super mark 1310 suddenly appearing in the video stream and pointing to the room, marked as "stroke patient worsening". No other doctor will see the labels, and Dr. Reynolds can only see them during the first hour.

[0330] The doctor can also directly set a temporary mark and propose a hyper mark 1310 on the local or remote station interface 606, 608 to assist his/her work plan. In some examples, the doctor can assign numbers to several patient rooms at the beginning of the meeting. Later, during the meeting, he/she can see the number in the displayed picture 622 and the suddenly appearing super mark 1310± to remind him/her of the order of patrolling the patient 614. The doctor can add notes that can be seen during the rest of the meeting or the next time they come back, for example, "come back after the meeting" on a patient, or "write a prescription" or "check again at 4 pm" on a patient.

[0331] In addition, the "smart" super mark 1310 can be automatically displayed. For example, the nurse may register photos of the patient about to enter to the database (eg, stored on the local and/or cloud storage 722) to compare with their electronic medical records. The remote software application 601 can execute a facial recognition calculation program on the video stream captured by the robot camera 320 to identify the patients 614 and 1330, which can be compared with the database. When recognizing the patient's face, the remote software application 601 can automatically stop and display the patient's electronic medical record.

[0332] Referring again to FIG. 14, in some embodiments, each patient 614, 1330 receives a radio frequency identification (RFID) chip 497, for example on the cuff. The robot 100 may have an RFID reader in contact with the controller 500 as part of its sensor system 400 to identify nearby patients through the RFID chip. The remote software application 601 may display a corresponding super mark when the patient enters the RFID range of the robot 100 (for example, 6 feet). Hypermarker 1310 may float in the air because RFID is not direction-specific. An alternative hybrid method may use computer vision technology to recognize the presence of patients 614, 1330 in the field of view area 322 of the robot 100 by recognizing human faces, and then assume that the RFID match belongs to the patient and is located on the patient 614, 1330 Super mark 1310.

[0333] Referring to FIGS. 16A-16D, in some examples, the robot system 1600 includes one or more remote robots 100 in contact with a bridge 602, which is connected to a local robot terminal server 604a and a remote terminal server 604b (for example, like a cloud Computing Services 720 (Figure 7)) contact. The local robot terminal server 604a contacts the local technician computing device 606 and the remote terminal server 604b contacts the remote operator computing device 608. The robot system 1600 also includes one or more data sources 1610 for storing sensor data received from the robot sensor system 400 and/or user interaction data, such as information obtained by the user through the networking board 310 and/or the user interface 605. As shown in the example, the robot system 1600 includes at least one robot sensor data source 1610a for storing sensor data and at least one head data source 1610b for storing user interaction data._oThe data source 1610 may be located on the robot 100, the cloud storage 722 (FIG. 7), the local robot terminal server 604a and/or the remote terminal server 604b.

[0334] The map data source 1620 can be the plan view 810, the robot map 820, the mark 662 information, and/or the hyper mark 1310 information storage information, such as stored in the robot 100, the cloud storage 722, the local robot terminal server 604a, and/or The database on the remote terminal server 604b. The graph data source 1620 can be a single database or a combination of data sources 1610, such as a robot sensor data source 1610a and a head data source 1610b_oThe remote software application 601 and/or the robot 100 (for example, the controller 500) can enter the graph data source 1620 to perform real-time or offline consistency processing, provide user interface feedback, execute navigation procedures, and propose graphs 622, etc.

[0335] In some embodiments, the control system 510 executes on the controller 500 of the robot 100 to approach one or more data sources 1610 to issue events that can be recognized by the behavior system 510a, such as the robot sensor data source 1610a, head The data source 1610b, and/or the graph data source 1620. In response to the proposed event, the behavior system 510a can perform a

CN 104898652 Β

One or more actions 512 that affect the selection of a command, which is executed by the resource control arbiter 560 of the robot resource 530 (FIG. 5). In the example shown in FIG. 16C, the robot control system 510 contacts the graph data source 1620 to enter the consistency matrix/database, which can store consistency processing information, such as real-time sensor/sign data 1622a, operator commands 1622b, and local perception Spatial data 1622c (e.g., point-shaped data of the measured volume received from the three-dimensional image sensor 450), occupancy bitmap data 1622d (e.g., robot map 820), ground plane data 1622e (e.g., plan view 810), and end user tags Table 1622f (for example, storing x, y, z coordinates and marked area), and/or robot behavior marking table 1622g (for example, storing x, y, z coordinates and marked area). Also referring to FIG. 5, the behavior 512 of the behavior system 510a can evaluate the possible results of the robot actions based on the proposed events, such as the sensor events of the sensor system 400 and the tag events proposed by placing the tags 662 and 1310 stored in the tag tables 1622f and 1622g ( For example, it can simulate sensor events). Therefore, the action selection engine 580 can select the feasible robot action with the best result based on the behavior evaluation. As a result, the robot 100 can be Automatic operation, this method considers the marks 662 and 1310 received through the remote software application 601.

[0336] Referring again to the ramp example shown in FIG. 11C, when the robot 100 approaches the ramp 1122, the robot control system 510 may perceive the ramp 1122 as an obstacle based on the sensor signal received from the sensor system 400. In order to distinguish and perceive the obstacle. 1120a and the actual obstacle 1120b, the control system 510 may need to enter a public database storing robot data and user data, such as a graph data source 1620. Using the graph data source 1620, the control system 510 can determine that the detected ramp 1122 is a perceived obstacle 1120a Instead of actual obstacles 1120b_oIn addition, the control system 510 can contact the remote software application 601 to receive user input about whether the user perceives the ramp 1122 as an actual obstacle 1120b, and/or receive the alternative robot path 652 and/or the alternative robot target 619 The remote software application 601 can use the image data source 1620 to determine the transformation between the two-dimensional image 622a, 810 and the three-dimensional image 622c, the real-time video input of the remote view 612 and the two-dimensional and/or three-dimensional image 622a, 622c. Provide a hybrid map 622b (Figure 9B). In addition, the remote software application 601 can use the graph data source 1620 to bring up the leading view 612a in the plan view window 620 (FIG. 10C).

[0337] Referring again to FIG. 12, in another specific embodiment, when the user selects the robot path 652 on one of the two-dimensional image 622a, the hybrid image 622b, the three-dimensional image 622c, and the remote view 610, the remote software application The program 601 can use the graph data source 1620 to determine the deformation between any graph 622 and the remote view 612, and when the drive command to the target 618 is executed, the robot controller 500 applies the deformation to the selected robot path to determine the robot path for the user. A series of corresponding robot path graph coordinates on graph 820. In addition, the remote software application 601 can apply the determined deformation to determine the corresponding series of robot path coordinates to display the robot path 652 on any map 622 and remote view 612. The map data source 1620 can store the determined deformation and/or a series of robots. The path coordinates are to display the robot path 652 on any map 622 and remote view 612.

[0338] Therefore, it should be widely understood that the term "deformation" as used herein relates to the determination of spatial coordinate errors, the difference between the conversion from one coordinate system to another coordinate system, and the conversion between coordinate systems of different dimensions. For example, the robot and/or the remote terminal can determine the deformation between at least a part of the two-dimensional plan view and the two-dimensional map generated by the robot, such as those generated using various robot sensors or laser scanners. Therefore, the robot and/or the terminal can determine the deformation between the 3D image or video input and the 2D plan view. In addition, determining the deformation may involve the first-person view, the third-person view, the plan view, the hybrid view, and/or the coordinate conversion between any two different coordinate systems or projections within the same coordinate system.

[0339] Referring again to FIG. 13, in some specific embodiments, when the user places the mark 662 or the hyper mark 1310 on the plan view 810 displayed like the image 622 of the image window, the remote software application 601 displays the image 622. The user selects the position on the electronic display and overlays the annotations related to the marks 662, 1310 on the map 622. Remote software application

CN 104898652 Β

The sequence 601 can also determine the deformation between the plan view 810 and the remote view 610 (ie, the first-person video captured by the robot camera 320) and apply the deformation to the coordinates of the marks 662, 1310 on the plan view 810 to determine the remote view 610 The corresponding video coordinates. The mark annotations related to the marks 662, 1310 and stored by the image data source 1620 can be displayed on the remote view 610 through the remote software application 601 using the determined video coordinates.

[0340] Referring to FIG. 17, in some embodiments, the user interface 605 provides enhanced coverage 1710 on, for example, the remote video input window 610 and/or the image window 620, which allows the user to imagine the robot relative to the robot base 120 The position of the head 160. The enhanced coverage 1710 may allow the user to recognize the robot sensor system 400 relative to the full 360. The current view areas 322, 442, and 452 of the view area, which are indicated by arc 1720 in the example shown. This allows the user to make choices for rotation outside the current field of view areas 322, 442, and 452 (for example, the head 160 and/or the base 120). [0341] The user can be in the area defined by the first and second rings 1722 and 1724 Click to rotate the virtual head 1728 to the point. When the user rotates the virtual head 1728, the robot head 160 can move in real time, and the remote software application 601 also updates the real-time video input from the robot camera 320 and displayed on the remote view 612 of the remote video input window 610 in real time. As shown in the example, the enhanced coverage 1710 has a virtual base 1726 corresponding to the robot base 120 and a virtual head 1728 corresponding to the robot head 160, which is positioned relative to the virtual base 1726 corresponding to the current pose of the robot. The rows are angled/oriented. In some examples, one of the virtual base 1726 and the virtual head 1728 appears stationary, while the other is free to move relative to the stationary one.

[0342] If the user clicks in the area defined by the first and second rings 1722 and 1724 to rotate the head 1728 outside the current field of view area 1720, the robot 100 can rotate the head 160 and/or the base 120 to complete the user's command. In some examples, after rotating the head 160 according to a user command, the base 120 may be rotated and the head 160 may be moved to a center position thereafter. If the user later attempts to change the position of the robot 100 based on the previous rotation, the change in the position may become a problem. In order to alleviate the problem, certain embodiments may utilize a system to reduce the requirement for base rotation to accommodate head rotation. For example, when the virtual head 1728 rotates to a certain angle, it may start to move in the reverse direction. If the robot head 160 maintains the angle at specified intervals, the system can slowly rotate the base 120 to focus the head 160 relative to the base 120, while at the same time rotating the head 160 in the opposite direction at the same speed. This keeps the current object in the field of view, and at the same time confirms that the head 160 and the base 120 are now in a straight line and the previous frame of reference is dominated by where the user is looking. Further, if the user wants to continue to see further in the direction, the full translational range of the head 160 is available.

[0343] FIG. 8 shows an exemplary series of robot events, such as in the remote software application 601, used to respond to user commands. In the initial state, the robot 100 can move from one position to another after receiving a driving command. The command can be initiated by the operator, by behavior (for example, the behavior performed on the control system of the controller), or by the planner (for example, a pre-planned task or program). In this example, the command includes a new heading to move in the opposite direction to the new target. In response to the command, the robot 100 can rotate its head (left or right) to the translation limit. After reaching the translation limit, the robot 100 may rotate the base 120 (for example, holonomically in place) to allow the head 160 to rotate in a new direction. The term "translation limit" may refer to a point when the upper part of the robot cannot physically rotate relative to the lower part of the robot, at which point the upper part deviates from the lower part by a predetermined number of rotation angles, and/or the term "translation limit" It can be a function of the number of angles the upper part deviates from the lower part and the length of time the upper part deviates from the lower part.

[0344] In some examples, the robot 100 continues to rotate the base 120 so that the forward driving direction F is the same as the new heading, thus providing the head 160 with relatively equivalent left/right translation capabilities. When the robot 100 rotates the base, it can rotate the head 160 at the same time to face the new heading and arbitrarily so that the field of view area 322, 452 of the sensors 320, 450, 450b on the head 160

CN 104898652 Β

Can point to a new course. In some embodiments, the robot 100 rotates the base 120 and the head 160 together to allow the head 160 to face the new heading relatively faster. If the base 120 rotates excessively, the head 160 can reverse translation to restore alignment.

[0345] FIG. 19 shows an exemplary remote view 612 on which the remote software application 601 overlays the screen indicator 1910 on the remote video input received from the robot 100. The screen indicator 1910 can be displayed near the mouse cursor 1912 and can be Represents the current range of head movement. When the user moves the mouse cursor 1912 to the left or right of the remote video view 612 (the possible intention of clicking is to move the head to point there), the on-screen indicator 1910 can be displayed above the cursor 1912 to indicate to maintain the direction How much of the head is active (for example, how much of the active holding range of the head 160 is available) ο [0346] The highlight frame 1920 may highlight the area of interest in the remote video view 612. The user can create a highlight frame 1920 around the area of interest on a part of the remote video view 612, for example, by dragging and dropping the frame on the screen and/or opening the frame by clicking and dragging around the area of interest. In response, the remote software application 601 may cause the robot head 160 to move to focus in the highlight frame 1920. In addition, the camera 320 can be enlarged to match the size of the highlight frame 1920.

[0347] Referring to 20A-20B, in some specific embodiments, if the robot 100 accidentally loses a communication connection (for example, loses a wireless signal), the robot 100 can stop or continue to drive to its target. When the remote robot moves through the environment, communication can be disrupted, for example, weak signal strength can be caused when the robot 100 switches between various wireless access points and/or encounters disruption in data transmission. By continuing to navigate automatically, communication can be restored when the robot reaches the desired goal.

[0348] When the robot 100 experiences a loss of communication connection, the robot 100 can refer to the last trusted position/posture (for example, stored locally by the controller 500) and/or the currently determined position/posture (for example, based on the robot sensor system 400) to continue sailing to the target. If the robot path is a planned path, the robot 100 can return to the planned path to reach the target. On the other hand, if the user remotely monitors that the robot 100 reaches the target, the robot 100 can follow the planned path to reach the nearest/last trusted location with a communication connection (for example, radio frequency and/or wireless). As an option, the robot 100 can be driven along the shortest path (ie, a new path) to the nearest/last position with a communication connection.

[0349] After reaching the nearest/last trusted location, the robot 100 can confirm whether the communication connection has been reestablished and, if so, whether it has reached the target. If the communication connection has not been reestablished, the robot 100 can synchronize its video recording (and any other sensor data of the sensor system 400) to move to the next trusted location stored by the controller 500. In addition, if the target has not been reached but the communication connection has been reestablished, the controller 500 can perform a safe haven countermeasure, which can cause continuous recording of sensor data and display of a new robot-side user interface (for example, on the network board 310), which indicates The conference has been terminated because of loss of communication connection. The robot 100 can improve its connection recovery probability by re-evaluating and/or executing path planning activities to the last trusted location (using ODOA). The robot 100 can also move its antennas 490a, 490b (FIGS. 2 and 4C) to possibly obtain better communication reception. The robot 100 may use a mobile ad-hoc network (MANET), which is a self-configuration of the network infrastructure of mobile devices connected by wireless links.

[0350] In some examples, the robot can improve the completeness and accuracy of the robot map 820 by marking the location of the event communication and any reconstructed communication location. The robot 100 can use station navigation to move to an area known to have connectivity (WLAN hot area or strong signal area). A site is a series of coordinates for identifying points in physical space. The robot 100 may use the station established on the robot map 820 to operate to reach the target.

[0351] Another safe haven strategy may include planning a path or activity to the nearest charging/docking station based on the robot map 820 to reach the nearest minimum traffic area. In some examples, the robot 100 may move toward another nearest robot 100, which may use multiple antennas 490a, 490b as multiple input and multiple output (ΜΙΜΟ) to act as a Wi-Fi bridge 602.

[0352] The specific implementations of the various systems and technologies described herein can be used in digital electronic circuits, integrated circuits, and particularly

CN 104898652 Β

Designed to be implemented in application-specific integrated circuits (ASICs), computer hardware, firmware, software, and/or a combination of them. These different embodiments can include specific embodiments on one or more computer programs that can be executed and/or described on a programmable system that includes at least one programmable process that can be dedicated or general purpose A device, which is coupled to receive data and instructions, and transmit data instructions; a storage system; at least one input device; and at least one output device.

[0353] These computer programs (also called programs, software, software applications or codes) include machine instructions for programmable processors, and can be implemented in high-level programs and/or object-oriented programming languages, and/or assembly/ Machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, instrument and/or device (for example, magnetic disks, optical disks, memories, programmable logic devices (PLDs)), A machine-readable medium used to provide machine commands and/or data to a programmable processor, including receiving machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

[0354] The specific embodiments and functional operations of the subject described in this document can be implemented in digital electronic circuits, or computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or their A combination of one or more. The specific implementation of the subject described in this document can be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a computer-readable medium, which are executed by a data processing instrument or controlled operating. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of problems affecting a machine-readable propagated signal, or a combination of one or more of them. The term "data processing instrument" includes all instruments, devices and machines used to process data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the instrument can include codes that create an execution environment for the computer program in question, for example, codes that constitute processor firmware, protocol stack, database management system, operating system, or a combination of one or more of them. The propagated signal is an artificially generated signal, for example, a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiving instrument.

[0355] A computer program (also called a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted language, and it can be deployed in any form, including as a stand-alone A program or as a module, component, subroutine or other unit suitable for use in a computing environment. Computer programs do not necessarily correspond to files in the file system. The program can be stored in part of a file that holds other programs or data (for example, one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files ( For example, storing one or more modules, subroutines or coded files). A computer program can be deployed and executed on one computer, or executed on multiple computers located at one location or distributed across multiple sites and interconnected by communication networks.

[0356] The processes and logic flows described in this document can be executed by one or more programmable processors that execute one or more computer programs, and perform functions by manipulating input data and generating output. The process and logic flow can also be executed by a dedicated logic circuit, and the instrument can also be implemented as a dedicated logic circuit, such as a field programmable gate array (FPGA) or ASICo

[0357] Processors suitable for the execution of computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, or operably coupled to receive data from it or transmit data to it, or both, such devices as magnetic disks, magnetic Compact disc, or compact disc. However, the computer need not have such a device. In addition, the computer can be embedded in another device,

CN 104898652 Β

For example, mobile phones, personal digital assistants (PDA), mobile audio players, global positioning system (GPS) receivers, to name a few. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or Removable disks; magneto-optical disks; CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by dedicated logic circuits, or composed.

[0358] The specific implementation of the subject described in this document can be implemented in a computing system. The computing system can be a computing system that includes back-end components, such as a data server; and can be a computing system that includes intermediate components, such as an application server. Computing system; or it can be a computing system that includes front-end components, for example, a client computer with a graphical user interface or a web browser. Through the graphical user interface or web browser, the user can interact with the specific implementation of the subject described in this document. Ways to influence each other, or include any of the back-end, intermediate, or front-end components of one or more combinations of computing systems. The components of the system can be connected to each other through any form or medium of digital data communication, for example, a communication network. Examples of communication networks include local area networks (LAN) and wide area networks (WAN), such as the Internet.

[0359] The computing system can include a client and a server. The client and the server are usually far away from each other and usually influence each other through the communication network. The relationship between the client and the server is generated by computer programs running on their respective computers, and they have a client-server relationship with each other.

[0360] Although this document contains many details, these should not be construed as limitations on the scope of the invention or what may be required, but as descriptions of specific functions of specific embodiments unique to the invention. Some of the features described in the content of individual specific embodiments can also be implemented in a combination of a single specific embodiment. Conversely, various characteristics described in the content of a single specific embodiment can also be implemented in separate multiple specific embodiments or in any suitable sub-combination. In addition, although the characteristics may act in certain combinations as described above, and even as required initially, one or more characteristics from the combination are omitted from the combination in some instances, and the combination may be designated as Subgroups or variants of subgroups.

[0361] Similarly, although the operations are depicted in the drawings in a specific order, this should not be construed as requiring the operations to be performed in the specific order shown or in order, or that all the operations shown must be performed to Achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-mentioned specific embodiments should not be understood as requiring the separation in all specific embodiments, and it should be understood that the described program components and systems can usually be integrated together in a single Software products or packaged into multiple software products.

[0362] Many specific implementations have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Therefore, other specific embodiments are also within the scope of the following claims. For example, the actions described in the claims can be performed in a different order and still achieve the desired result.

CN 104898652 Β