Systems and methods for configurable operation of a robot based on area classification

CLAIM OF PRIORITY

0001This application is a continuation application of and claims priority from U.S. patent application Ser. No. 15/704,814, filed Sep. 14, 2017, which claims priority from U.S. Provisional Patent Application No. 62/394,638, entitled “SYSTEMS AND METHODS FOR CONFIGURABLE OPERATION OF A ROBOT USING A REMOTE USER INTERFACE” and filed Sep. 14, 2016, in the United States Patent and Trademark Office, the disclosures of each are incorporated by reference herein.

FIELD

0002The present invention relates generally to robotic systems and, more specifically, to configuring the operation of autonomous robots.

BACKGROUND

0003Automated robots and robotic devices are used to perform tasks traditionally considered mundane, time-consuming, or dangerous. For example, substantially autonomous robots may be used to provide coverage of a surface to perform operations such as cleaning, performing surface treatments and/or painting. However, from a user's perspective, autonomous robot missions may be unpredictable leaving the user with a lack of understanding regarding which area the robot will move to next. Additionally, the user may lack any information and/or input regarding a mission.

SUMMARY

0004Some embodiments of the present invention include a computing device including a processor and a memory coupled to the processor, where the memory includes a non-transitory computer-readable storage medium storing computer-readable program code therein. The computer-readable program code is executable by the processor to perform operations including generating a segmentation map defining respective regions of a surface based on occupancy data that is collected by a mobile robot responsive to navigation of the surface, classifying or otherwise identifying sub-regions of at least one of the respective regions as first and second areas (such as non-clutter and clutter areas) based on differences therein, and computing a coverage pattern based on identification of the sub-regions. The coverage pattern indicates a sequence for navigation of the first and second areas (such as the non-clutter and clutter areas), and is provided to the mobile robot. Responsive to the coverage pattern, the mobile robot sequentially navigates the first and second areas (for example, the non-clutter and clutter areas) of the at least one of the respective regions of the surface in the sequence indicated by the coverage pattern.

0005Some embodiments of the present invention include a method of operating a mobile robot. The method includes executing, by at least one processor, computer readable instructions stored in a non-transitory computer readable storage medium to perform operations including generating a segmentation map defining respective regions of a surface based on occupancy data that is collected by the mobile robot responsive to navigation of the surface, classifying or otherwise identifying sub-regions of at least one of the respective regions as non-clutter and clutter areas, and computing a coverage pattern based on identification of the sub-regions. The coverage pattern indicates a sequence for navigation of the non-clutter and clutter areas, and is provided to the mobile robot. Responsive to the coverage pattern, the mobile robot sequentially navigates the non-clutter and clutter areas of the at least one of the respective regions of the surface in the sequence indicated by the coverage pattern.

0006Some embodiments of the present invention include a computer program product for operating a mobile robot. The computer program product includes a non-transitory computer readable medium having computer readable instructions stored therein, that, when executed by a processor, causes a processor to perform operations including generating a segmentation map defining respective regions of a surface based on occupancy data that is collected by the mobile robot responsive to navigation of the surface, classifying or otherwise identifying sub-regions of at least one of the respective regions as non-clutter and clutter areas, and computing a coverage pattern based on identification of the sub-regions. The coverage pattern indicates a sequence for navigation of the non-clutter and clutter areas, and is provided to the mobile robot. Responsive to the coverage pattern, the mobile robot sequentially navigates the non-clutter and clutter areas of the at least one of the respective regions of the surface in the sequence indicated by the coverage pattern.

0007In some embodiments, the computing device may be a component of the mobile robot, and the occupancy data may be detected by at least one sensor of the mobile robot. The operations may further include operating a drive of the mobile robot to sequentially navigate the non-clutter and clutter areas of the at least one of the respective regions of the surface in the sequence indicated by the coverage pattern.

0008In some embodiments, the computing device may be a component of a server communicatively coupled to the mobile robot. The operations may further include receiving the occupancy data from the mobile robot responsive to the navigation of the surface, and storing the segmentation map in a data store.

0009In some embodiments, the sequence may further include navigation of a perimeter of the surface, and the mobile robot may sequentially navigate the non-clutter area, the clutter area, and the perimeter of the surface in the sequence indicated by the coverage pattern.

0010In some embodiments, the coverage pattern may further indicate a rank direction for the non-clutter area. The rank direction may correspond to a dimension of the at least one of the respective regions that is longer than another dimension thereof, and the mobile robot may traverse the non-clutter area in the rank direction indicated by the coverage pattern.

0011In some embodiments, the respective regions defined by the segmentation map may correspond to respective rooms. The coverage pattern may further indicate an order of navigation of the respective rooms, and the mobile robot may sequentially navigate the non-clutter and clutter areas of one of the respective rooms in the sequence indicated by the coverage pattern before navigation of a next one of the respective rooms in the order indicated by the coverage pattern.

0012In some embodiments, computing the coverage pattern may further include determining a location on the surface for ending the navigation of the non-clutter area based on a proximity to the clutter area. Responsive to traversing the non-clutter area in the rank direction indicated by the coverage pattern, the mobile robot may end the navigation of the non-clutter area at the location having the proximity to the clutter area and may traverse the clutter area in a random pattern. Responsive to traversing the clutter area, the mobile robot may not traverse the non-clutter area in the rank direction before traversing at least one other of the sub-regions.

0013In some embodiments, the operations may further include providing the segmentation map to a user device. The user device may include a display or other user interface that is configured to display the segmentation map for viewing by a user. The operations may also include receiving, from the user device, a selection input responsive to providing the segmentation map thereto, and computing the coverage pattern based on the selection input.

0014In some embodiments, the selection input may include respective levels of cleaning for the at least one of the respective regions and/or the sub-regions thereof. The coverage pattern may be generated to indicate the respective levels of cleaning, and the mobile robot may execute a number of cleaning passes in navigating the at least one of the respective regions and/or the sub-regions thereof according to the respective levels of cleaning indicated by the coverage pattern.

0015In some embodiments, the selection input may include a user-defined boundary or label for one or more of the respective regions and/or sub-regions thereof. The coverage pattern may be generated to indicate the user-defined boundary or label, and the mobile robot may avoid navigation of the one or more of the respective regions and/or sub-regions thereof according to the user-defined boundary or label indicated by the coverage pattern.

0016In some embodiments, the operations may further include detecting respective flooring types of the sub-regions, and computing the coverage pattern may further include determining the sequence for navigation of the non-clutter and clutter areas based on the respective flooring types of the sub-regions corresponding thereto.

0017In some embodiments, the segmentation map may include data previously received from a user device and/or collected by the mobile robot responsive to at least one previous navigation of the surface.

0018In some embodiments, generating the segmentation map may include comparing the occupancy data collected by the mobile robot responsive to the navigation of the surface with the data collected by the mobile robot responsive to the at least one previous navigation of the surface, and updating the segmentation map to include commonalities indicated by the comparing and to exclude outliers indicated by the comparing.

0019In some embodiments, generating the segmentation map may include computing a binary image based on the occupancy data by applying a thresholding function, and performing a watershed transformation on the binary image to partition the binary image into the respective regions by applying a distance transform.

0020In some embodiments, the segmentation map may include simplified boundaries relative to actual boundaries indicated by the occupancy data collected by the mobile robot responsive to navigation of the surface.

0021In some embodiments, the operations may further include modifying the segmentation map having the simplified boundaries to indicate the clutter areas responsive to identification thereof prior to providing the segmentation map to a user device.

0022In some embodiments, identifying the sub-regions as non-clutter and clutter areas may further include accessing a data store comprising a plurality of patterns and identifications thereof, and classifying pixel regions within the sub-regions based on similarities to the plurality of patterns stored in the data store.

0023In some embodiments, one or more of the identifications of the plurality of patterns may be based on respective labeling inputs received from one or more user devices.

0024In some embodiments, identifying the sub-regions as non-clutter and clutter areas may further include classifying the pixel regions as static structures corresponding to walls or dynamic structures corresponding to clutter responsive to accessing the data store.

0025In some embodiments, identifying the sub-regions as non-clutter and clutter areas may further include identifying at least one boundary based on the occupancy data, and prior to accessing the data store, distinguishing the pixel regions as attached pixel regions that are adjacent the at least one boundary and detached pixel regions that are spaced apart from the at least one boundary.

0026Some embodiments of the present invention include methods for operating a mobile floor cleaning robot. Methods may include generating a segmentation map of a surface of an enclosed space using the mobile floor cleaning robot to identify a plurality of regions of the surface based on an occupancy grid that is generated by the mobile floor cleaning robot, sending the segmentation map of the surface to a user device that includes a display that is configured to display the segmentation map of the surface, and receiving a modified segmentation map from the user device based on inputs received from the user into the user device.

0027Some embodiments include performing a cleaning operation on the surface by sequentially performing the cleaning operation in individual ones of the plurality of regions, wherein the cleaning operation in a given one of the plurality of regions is completed before starting the cleaning operation in another of the plurality of regions.

0028In some embodiments, sending the segmentation map comprises wirelessly transmitting segmentation map data using at least one wireless communication link. Some embodiments provide that the at least one wireless communication link includes a local area network that provides communication between the user device and the mobile floor cleaning robot. In some embodiments, the at least one wireless communication link comprises a server that is configured to provide a persistent storage of the occupancy grid, the segmentation map and/or a modified segmentation map.

0029Some embodiments provide that the modified segmentation map identifies a subset of the plurality of regions to be cleaned, the subset including a portion of one of the plurality of regions, and the modified segmentation map identifies an order in which the plurality of regions are to be cleaned.

0030Some embodiments include performing a cleaning operation on the surface by performing the cleaning operation based on the modified segmentation map. In some embodiments, performing the cleaning operation based on the modified segmentation map comprises performing the cleaning operation on a subset of the plurality of regions. In some embodiments, performing the cleaning operation based on the modified segmentation map comprises performing the cleaning operation of ones of the plurality of regions each at a respective one of a plurality of levels that are selected by the user and in a rank direction that corresponds to a largest dimension of area in the respective region without obstacles, and the plurality of levels are associated with a number of cleaning passes the mobile floor cleaning robot makes in the ones of the plurality of regions.

0031In some embodiments, generating the occupancy grid by exploring the surface with the mobile floor cleaning robot comprises generating visual maps for localization of the mobile floor cleaning robot within the occupancy. Some embodiments provide that generating the visual maps for localization comprises detecting a feature using a plurality of images captured by the mobile floor cleaning robot, determining a feature description of the feature based on the image captured by the mobile floor cleaning robot, classifying the feature by performing a feature lookup in a feature database, determining a pose and position of the mobile floor cleaning robot that is associated with the plurality of images and generating a landmark in the occupancy grid that is associated with the pose and the position of the mobile floor cleaning robot. Some embodiments provide that the method further comprises augmenting the occupancy grid with updated landmark data that is generated over multiple missions of the mobile floor cleaning robot.

0032In some embodiments, the occupancy grid comprises a plurality of pixels each having a grayscale value that corresponds to whether the pixel location is occupied, traversable or unexplored. Some embodiments provide that generating the segmentation map of the surface using the occupancy grid to identify the plurality of regions of the surface comprises applying a median filter to the plurality of pixels that replaces each pixel value with a median value corresponding to pixels that surround the corresponding one of the plurality of pixels to generate a filtered occupancy grid, computing a binary image of the filtered occupancy grid by applying a thresholding function to the filtered occupancy grid, and performing a watershed transformation on the binary image to partition the binary image into the plurality of regions and vertices of the plurality of regions by applying a distance transform.

0033Some embodiments include performing a clutter removal operation on the plurality of regions in the binary image. In some embodiments, performing the clutter removal operation comprises identifying a boundary of the binary image, identifying and removing pixels that correspond to obstacles that are detached from the boundary of the binary image, and identifying and removing pixels that correspond to obstacles that are attached to the boundary region of the binary image. Some embodiments include merging the vertices of the plurality of regions to join the plurality of regions and to define corridors that are adjacent the plurality of regions.

0034Some embodiments include generating an automated floor coverage pattern that is based on the segmentation map that includes the plurality of regions, the automated floor coverage pattern including a selection of a rank direction corresponding to each of the plurality of regions. In some embodiments, the rank direction defines a dimension of the region that is longer than other dimensions of the region. Some embodiments provide that the rank direction defines a dimension of a portion of the respective region that is without obstacles and that is different from a dimension of the region that is longer than other dimensions of the region. In some embodiments, the automated floor coverage pattern includes, for each region of the plurality of regions, sequentially selecting an uncluttered portion of the region for coverage, a perimeter of the region for coverage and a cluttered portion of the region for coverage.

0035Some embodiments include sending the segmentation map of the surface to a remote computing device that is configured to communicate with the mobile floor cleaning robot and with a user mobile terminal that includes a display that is configured to display the segmentation map of the surface. In some embodiments, sending the segmentation map of the surface to the remote computing device is performed responsive to the generation of the segmentation map to maintain historical data corresponding to the mobile floor cleaning robot. Some embodiments include receiving a modified segmentation map from the remote computing device based on data sent from the mobile floor cleaning robot and/or inputs received from a user into the user device. Some embodiments include performing a cleaning operation on the surface by performing the cleaning operation based on the modified segmentation map. In some embodiments, the segmentation map includes operational data corresponding to the mobile floor cleaning robot. Some embodiments provide that the operational data comprises a cleaning function map that identifies debris collection that corresponds to a quantity of debris that is collected by the mobile floor cleaning robot as a function of location on the segmentation map.

0036Some embodiments of the present invention include a method of operating a user terminal, including executing, by at least one processor of the user terminal, computer readable instructions stored in a memory comprising a non-transitory computer readable storage medium to perform operations including receiving, via a transceiver of the user terminal, a segmentation map defining respective regions of a surface based on occupancy data that is collected by a mobile robot responsive to navigation of the surface. The segmentation map identifies sub-regions of at least one of the respective regions as clutter and non-clutter areas. The operations further include displaying, via a user interface of the user terminal, a graphical representation of the segmentation map, receiving, via the user interface, a selection input corresponding to one or more of the respective regions of the segmentation map and/or the sub-regions thereof, and transmitting, via the transceiver, a modified segmentation map including the selection input to the mobile robot.

0037Some embodiments of the present invention include computer program products that include a computer readable storage medium having computer readable program code embodied in the medium that when executed by a processor of a computer causes the computer to perform operations comprising receiving a segmentation map of a surface of an enclosed space, the segmentation map including a plurality of regions of the surface and generated by a mobile floor cleaning robot, displaying an image of the segmentation map on a display of the computer, receiving, via a user input device of the computer, a user input corresponding to the segmentation map of the surface, and sending, to the mobile floor cleaning robot, a modified segmentation map that is modified corresponding to the user input.

0038In some embodiments, receiving the user input comprises receiving the user input to modify an appearance of the image of the segmentation map. Some embodiments provide that the user input to modify the appearance of the image includes a user input to change an angle of perspective of the user relative to the image, a dimensional selection, a zoom selection, and a map feature selection. In some embodiments, the map feature selection comprises a heat map selection that causes the image of the segmentation map displayed to illustrate a level of historical debris collection as a function of location on the segmentation map. Some embodiments provide that the map feature selection comprises a clutter map selection that causes the image of the segmentation map displayed to selectively illustrate clutter on the segmentation map.

0039In some embodiments, receiving the user input comprises receiving a divide region input that is configured to cause one of the plurality of regions to divide into more than one region.

0040Some embodiments provide that receiving the user input comprises receiving a combine region input that is configured to cause more than one of the plurality of regions to combine into a single region.

0041In some embodiments, operations further comprise receiving, by the computer, a status communication that is generated by the mobile floor cleaning robot and responsive to receiving the status communication, displaying a message on the display of the computer. In some embodiments, the status communication comprises a progress update corresponding to a cleaning mission. Some embodiments provide that the progress update corresponding to the cleaning mission is displayed as an animated graphic of a cleaning operation on the image of the segmentation map. In some embodiments, the progress update is a substantially real-time progress update that includes a visual indication on the image of the segmentation map that is displayed.

0042In some embodiments, the status communication comprises an alert message that identifies a condition that is detected by the mobile floor cleaning robot. Some embodiments provide that the condition comprises an incomplete mission condition and the mobile floor cleaning robot is unable to complete a cleaning mission.

0043In some embodiments, the condition comprises an unexpected obstacle condition, the mobile floor cleaning robot detects an obstacle that is not included in the segmentation map, and a location of the unexpected obstacle is displayed on the image of the segmentation map.

0044Some embodiments provide that receiving the user input comprises receiving the user input that includes a cleaning mission instruction. In some embodiments, the cleaning mission instruction comprises a cleaning order instruction that includes an identification of which ones of the plurality of regions are to be cleaned in a given mission and a cleaning order corresponding to identified ones of the plurality of regions. In some embodiments, the cleaning mission instruction includes a first cleaning mission instruction that identifies a first selection of the plurality of regions to be cleaned and a second cleaning mission that identifies a second selection of the plurality of regions to be cleaned that is different from the first selection of the plurality of regions. Some embodiments provide that the first cleaning mission instruction includes a first scheduled time and the second cleaning mission includes a second scheduled time that is different from the first scheduled time.

0045In some embodiments, the cleaning mission instruction comprises a first cleaning level for a first region of the plurality of regions and a second cleaning level that is different from the first cleaning level for a second region of the plurality of regions, the first cleaning level identifies a first quantity of times that the mobile floor cleaning robot cleans the first region, and the second cleaning level identifies a second quantity of times that the mobile floor cleaning robot cleans the second region.

0046In some embodiments, the cleaning mission instruction comprises an identification of a portion of the occupancy floor space that is not to be cleaned by the mobile floor cleaning robot. Some embodiments provide that the identification of the portion of the occupancy floor space that is not to be cleaned by the mobile floor cleaning robot is persistent. In some embodiments, the identification of the portion of the occupancy floor space that is not to be cleaned by the mobile floor cleaning robot applies to a single cleaning mission.

0047Some embodiments provide that the cleaning mission instruction comprises a primary cleaning direction for each of the plurality of regions that identifies a cleaning direction that the mobile floor cleaning robot cleans in obstacle free portions of a cleaning operation.

0048In some embodiments, receiving the user input comprises receiving a user preference data input that includes a request for user preference data and operations further comprise displaying user preference data. Some embodiments provide that displaying user preference data comprises displaying historical user preference data and/or user preference data that corresponds to a single cleaning mission.

0049Some embodiments provide that receiving the user input comprises receiving a robot team configuration input that includes a first cleaning mission instruction corresponding to a first mobile floor cleaning robot and a second cleaning mission corresponding to a second mobile floor cleaning robot. In some embodiments, the first and second cleaning missions correspond to the segmentation map of the surface of the occupancy. Some embodiments provide that the first and second cleaning missions correspond to different segmentation maps of different portions of the occupancy. In some embodiments, the first mobile floor cleaning robot performs a first type of cleaning operation and the second mobile floor cleaning robot performs a second type of cleaning operation.

0050Some embodiments provide that the computer is a hand-held mobile terminal and the display is a touch-screen display. Some embodiments provide that the operations further comprise generating a graphical user interface via the touch-screen display and receiving the user input via the user input device comprises receiving the user input via the graphical user interface using the touch-screen.

0051In some embodiments, sending the modified segmentation map to the mobile floor cleaning robot comprises sending the modified segmentation map to a remote server that is configured to provide a persistent storage the modified segmentation map.

0052Some embodiments of the present invention include a robot management node comprising a processor and a memory coupled to the processor and comprising computer readable program code that when executed by the processor causes the processor to perform operations. The operations include receiving a segmentation map of a surface of an enclosed space from a robot, the segmentation map identifying a plurality of regions of the surface based occupancy data that is collected by the robot and storing the segmentation map of the surface of the occupancy in a data store that is associated with the robot.

0053In some embodiments, operations further comprise receiving user input data from a user device that receives a user input corresponding to the segmentation map. In some embodiments, the segmentation map is modified based on the user input data to generate a modified segmentation map. In some embodiments, operations further include sending the modified segmentation map based on the user input data and the segmentation map to the robot that is operable to perform a cleaning mission responsive to the modified segmentation map.

0054In some embodiments, receiving the segmentation map from the robot comprises receiving multiple segmentation maps that are generated by the robot during multiple cleaning missions, the segmentation map is updated and/or augmented by later received segmentation maps, and storing the segmentation map comprises storing the multiple segmentation maps to generate historical segmentation map data including the modified segmentation map.

0055Some embodiments provide that the modified segmentation map identifies a subset of the plurality of regions to be cleaned, the subset including a portion of one of the plurality of regions, and the modified segmentation map identifies an order in which the plurality of regions are to be cleaned.

0056Some embodiments provide that the cleaning mission includes performing a cleaning operation of ones of the plurality of regions each at a respective one of a plurality of levels that are selected by the user and in a rank direction that corresponds to a largest dimension of area in the respective region without obstacles and the plurality of levels are associated with a number of cleaning passes the robot makes in the ones of the plurality of regions.

0057In some embodiments, the rank direction defines a dimension of the region that is longer than other dimensions of the region.

0058Some embodiments provide that the rank direction defines a dimension of a portion of the respective region that is without obstacles and that is different from a dimension of the region that is longer than other dimensions of the region.

0059In some embodiments, the segmentation map includes operational data corresponding to the robot. Some embodiments provide that the operational data comprises a cleaning function map that identifies debris collection that corresponds to a quantity of debris that is collected by the robot as a function of location on the segmentation map. In some embodiments, the operational data comprises a cleaning function map that identifies a quantity of passes that are performed by the robot as a function of location on the segmentation map.

0060In some embodiments, receiving user input data comprises receiving a robot team configuration input that includes a first cleaning mission instruction corresponding to a robot and a second cleaning mission corresponding to a second robot.

0061Some embodiments provide that the processer and the memory are included in a cloud-based server that is remote from the robot and the occupancy.

0062Some embodiments are directed to an electronic device that is operable to performed operations described herein.

0063It is noted that aspects of the invention described with respect to one embodiment, may be incorporated in a different embodiment although not specifically described relative thereto. That is, all embodiments and/or features of any embodiment can be combined in any way and/or combination. These and other objects and/or aspects of the present invention are explained in detail in the specification set forth below.

0064Further features, advantages and details of the present invention will be appreciated by those of ordinary skill in the art from a reading of the figures and the detailed description of the embodiments that follow, such description being merely illustrative of the present invention. Other methods, devices, and computer program products according to embodiments will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such additional methods, devices, and computer program products be included within this description, be within the scope of the present inventive subject matter, and be protected by the accompanying claims. Moreover, it is intended that all embodiments disclosed herein can be implemented separately or combined in any way and/or combination.

BRIEF DESCRIPTION OF THE DRAWINGS

0065Aspects of the present invention are illustrated by way of example and are not limited by the accompanying drawings. In the drawings

0066<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram illustrating a system of devices that may be used to provide robot management according to some embodiments of the present invention.

0067<figref idref="DRAWINGS">FIG. 2A</figref> is a schematic diagram representing a mobile robot according to some embodiments of the present invention.

0068<figref idref="DRAWINGS">FIGS. 2B and 2C</figref> are top and bottom perspective views, respectively, of a mobile robot according to some embodiments of the present invention.

0069<figref idref="DRAWINGS">FIG. 3</figref> is a block diagram of an electronic device that may be an example of the robot management node according to some embodiments of the present invention.

0070<figref idref="DRAWINGS">FIG. 4</figref> is a block diagram of a wireless communication device and methods of operation according to some embodiments of the present invention

0071<figref idref="DRAWINGS">FIG. 5</figref> is a block diagram illustrating a method for operating a mobile floor cleaning robot according to some embodiments of the present invention.

0072<figref idref="DRAWINGS">FIG. 6</figref> is a block diagram illustrating operations of computer readable program product on a user device according to some embodiments of the present invention.

0073<figref idref="DRAWINGS">FIG. 7</figref> is a block diagram illustrating operations of computer program product on a robot management node according to some embodiments of the present invention.

0074<figref idref="DRAWINGS">FIGS. 8A through 8C</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating a raw data map, a cleaned map and a segmentation map in an automated map cleaning and segmentation operation according to some embodiments of the present invention.

0075<figref idref="DRAWINGS">FIGS. 9A through 9C</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating user display preference options of a raw data map, a segmentation map with clutter pixels and segmentation map with clutter regions in operations according to some embodiments of the present invention.

0076<figref idref="DRAWINGS">FIGS. 10A through 10F</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating operations for performing user region merging on a segmentation map according to some embodiments of the present invention.

0077<figref idref="DRAWINGS">FIGS. 11A through 11C</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating operations for performing user region splitting on a segmentation map according to some embodiments of the present invention.

0078<figref idref="DRAWINGS">FIGS. 12A through 12D</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating operations for performing user region labeling on a segmentation map according to some embodiments of the present invention.

0079<figref idref="DRAWINGS">FIG. 13</figref> is a graphical representation that may be displayed via a user interface of a communication device illustrating operations for displaying progress of a robot on a segmentation map according to some embodiments of the present invention.

0080<figref idref="DRAWINGS">FIG. 14</figref> is a graphical representation that may be displayed via a user interface of a communication device illustrating operations for providing an alert to a user of a failure condition on a segmentation map according to some embodiments of the present invention.

0081<figref idref="DRAWINGS">FIG. 15</figref> is a graphical representation that may be displayed via a user interface of a communication device illustrating operations for receiving user input to provide a cleaning sequence of a robot on a segmentation map according to some embodiments of the present invention.

0082<figref idref="DRAWINGS">FIGS. 16A through 16G</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating operations for map aggregation using multiple runs in the same occupancy according to some embodiments of the present invention.

0083<figref idref="DRAWINGS">FIGS. 17A and 17B</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating a segmentation map from the aggregating operations based on the raw data maps illustrated in <figref idref="DRAWINGS">FIGS. 16A-16G</figref> that are without clutter and that include clutter, respectively, according to some embodiments of the present invention.

0084<figref idref="DRAWINGS">FIGS. 18A and 18B</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating raw data of an occupancy map and a three-dimensional representation of the occupancy map, respectively, based on the raw data maps illustrated in <figref idref="DRAWINGS">FIGS. 16A-16G</figref> according to some embodiments of the present invention.

0085<figref idref="DRAWINGS">FIG. 19</figref> is a graphical representation that may be displayed via a user interface of a communication device illustrating an occupancy map illustrating operations for generating persistent information corresponding to an enclosed space according to some embodiments of the present invention.

0086<figref idref="DRAWINGS">FIGS. 20A through 20K</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating views of occupancy maps and segmentation maps corresponding to data gathered during robot exploration according to some embodiments of the present invention.

0087<figref idref="DRAWINGS">FIG. 21</figref> is a diagram illustrating a system of devices that may be used to provide robot management according to some embodiments of the present invention.

0088<figref idref="DRAWINGS">FIG. 22</figref> is a graphical representation that may be displayed via a user interface of a communication device illustrating a history screen including a mission coverage map corresponding to a cleaning mission according to some embodiments of the present invention.

0089<figref idref="DRAWINGS">FIG. 23</figref> is a graphical representation that may be displayed via a user interface of a communication device illustrating an occupancy map illustrating real-time reporting corresponding to a cleaning mission according to some embodiments of the present invention.

0090<figref idref="DRAWINGS">FIG. 24</figref> is a graphical representation that may be displayed via a user interface of a communication device illustrating an occupancy map illustrating a multi-mission coverage display according to some embodiments of the present invention.

0091<figref idref="DRAWINGS">FIG. 25</figref> is a graphical representation that may be displayed via a user interface of a communication device illustrating room-based data that may be displayed according to some embodiments of the present invention.

0092<figref idref="DRAWINGS">FIG. 26</figref> is a graphical representation demonstrating operations in which user input may be provided via a communication device using a voice command according to some embodiments of the present invention.

0093<figref idref="DRAWINGS">FIG. 27</figref> is a graphical representation of receiving user input to define a cleaning mission boundary responsive to a gesture on a touch screen of a communication device according to some embodiments of the present invention.

0094<figref idref="DRAWINGS">FIGS. 28A through 28C</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating a raw data map, a cleaned map and a cleaned map with detected clutter regions according to some embodiments of the present invention.

0095<figref idref="DRAWINGS">FIGS. 29A through 29D</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating a segmentation map that includes respective area-specific cleaning operations according to some embodiments of the present invention.

0096<figref idref="DRAWINGS">FIGS. 30A through 30G</figref> are graphical representations that may be displayed via a user interface of a communication device for user selection of respective area-specific cleaning operations and/or a sequence of performing the area cleaning operations according to some embodiments of the present invention.

0097<figref idref="DRAWINGS">FIGS. 31A through 31C</figref> are graphical representations that may be displayed via a user interface of a communication device for user selection of cleaning patterns for open area cleaning operations according to some embodiments of the present invention.

0098<figref idref="DRAWINGS">FIG. 32</figref> is a graphical representation that may be displayed via a user interface of a communication device illustrating detection and/or user selection of floor types for area-specific cleaning operations according to some embodiments of the present invention.

0099<figref idref="DRAWINGS">FIGS. 33A through 33D</figref> illustrate operations for generating a simplified map for display via a user interface of a communication device according to some embodiments of the present invention.

0100<figref idref="DRAWINGS">FIGS. 34A through 34F</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating object detection operations according to some embodiments of the present invention.

0101<figref idref="DRAWINGS">FIGS. 35A through 35D</figref> are graphical representations illustrating operations for clutter classification and feature extraction according to some embodiments of the present invention.

0102<figref idref="DRAWINGS">FIGS. 36A through 36C</figref> are graphical representations illustrating operations for updating persistent map data by feature matching and segmentation transfer between multiple maps according to some embodiments of the present invention.

0103<figref idref="DRAWINGS">FIG. 37</figref> is a graphical representation that may be displayed via a user interface of a communication device illustrating operation of a mobile robot responsive to user selection of temporal status according to some embodiments of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

0104The present invention now will be described more fully hereinafter with reference to the accompanying drawings, in which illustrative embodiments of the invention are shown. In the drawings, the relative sizes of regions or features may be exaggerated for clarity. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

0105It will be understood that when an element is referred to as being “coupled” or “connected” to another element, it can be directly coupled or connected to the other element or intervening elements may also be present. In contrast, when an element is referred to as being “directly coupled” or “directly connected” to another element, there are no intervening elements present. Like numbers refer to like elements throughout.

0106In addition, spatially relative terms, such as “under,” “below,” “lower,” “over,” “above,” “upper,” and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as “under” or “beneath” other elements or features would then be oriented “over” the other elements or features. Thus, the example term “under” can encompass both an orientation of over and under. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.

0107The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. As used herein the expression “and/or” includes any and all combinations of one or more of the associated listed items.

0108A “mobile robot” may refer to any device including a processor, memory, and drive system for navigating variable environment conditions and making autonomous decisions based on a plurality of sensor inputs. Mobile robots as described herein, may include robot cleaners (such as iRobot® ROOMBA®, BRAAVA®, and/or BRAAVA Jet™ cleaners). A “communication signal” may refer to any signal transmitted by a network-enabled electronic device. Such electronic devices may include a processor, memory, and a transmitter and/or receiver for communication via a wireless personal, local, and/or wide area network, and may include, but are not limited to, Internet-of-Things (IoT) devices. Wireless communication signals may include radio frequency signals, including but not limited to Wi-Fi signals, Bluetooth signals, and/or optical signals.

0109As used herein, a “wireless communication device” includes, but is not limited to, a device that is configured to receive/transmit communication signals via a wireless interface with, for example, a cellular network, a wireless local area network (WLAN), a digital television network such as a DVB-H network, a satellite network, an AM/FM broadcast transmitter, and/or another communication terminal. A wireless communication device may be referred to as a “wireless communication terminal,” a “wireless terminal” and/or a “mobile terminal.” Examples of wireless communication devices include, but are not limited to, a satellite or cellular radiotelephone; a Personal Communications System (PCS) terminal that may combine a cellular radiotelephone with data processing, facsimile and data communications capabilities; a PDA that can include a radiotelephone, pager, Internet/intranet access, Web browser, organizer, calendar and/or a global positioning system (GPS) receiver; and a conventional laptop and/or palmtop receiver or other appliance that includes a radio transceiver, including WLAN routers and the like.

0110Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

0111<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram illustrating a system of devices that may be used to provide robot management according to some embodiments of the present invention. Some embodiments provide that systems and/or operations disclosed herein may include a robot management node <b>200</b> that may be communicatively coupled to one or more robots <b>100</b> and/or one or more user devices <b>202</b> via one or more wired and/or wireless communication and/or data networks <b>204</b>. In some embodiments, the robot <b>100</b> and the user device <b>202</b> may be directly communicatively coupled via a near field communication protocol. The user device <b>202</b> may be a wired or wireless communication terminal, such as a desktop or laptop computer, tablet, or smartphone. The user device <b>202</b> may include an application stored in a memory thereof for communication with the robot(s) <b>100</b> and/or the robot management node <b>200</b>.

0112In some embodiments, the robot management node <b>200</b> may include a dedicated computer that includes a running instance of an application that is capable of receiving and transmitting communications to the one or more robots <b>100</b> and/or the one or more user devices <b>202</b>. Some embodiments provide that the robot management node <b>200</b> may further include instances and/or operations corresponding to a file server, a database server, an application server and/or a web server, including cloud-based servers. In some embodiments, the robot management node <b>200</b> includes a stationary or fixed location processing device that receives power via a wired power connection to a facility and/or building power source. Some embodiments provide that the robot management node <b>200</b> transmits and/or receives data via a wired data connection. In some embodiments, the robot management node <b>200</b> may be provided on a mobile processing device that includes an on-board power source and that transmits and/or receives data using a wireless and/or wired data connection. As disclosed herein, the robot management node <b>200</b> may provide a persistent store of segmentation maps, user inputs, robot operational data, and the like. For example, the persistent store may include maps generated from occupancy data and/or wireless signal strength/coverage data collected by the robot(s) <b>100</b> during navigation of one or more operating environments, as described for example in U.S. patent application Ser. No. 15/588,117 entitled “METHODS, SYSTEMS, AND DEVICES FOR MAPPING WIRELESS COMMUNICATION SIGNALS FOR MOBILE ROBOT GUIDANCE,” the disclosure of which is incorporated by reference.

0113<figref idref="DRAWINGS">FIGS. 2A-2C</figref> illustrate an autonomous coverage robot system that includes a mobile robot <b>100</b> (illustrated as a vacuum cleaning robot). The mobile robot <b>100</b> is adapted to mate with a base station or charging dock and/or an evacuation dock. The system also includes a charging or energy management system and an auto-docking control system each including cooperatively operating components of the mobile robot <b>100</b> and the dock.

0114A communications/guidance signal detector <b>152</b> is mounted on the top front of the housing <b>106</b> of the mobile robot <b>100</b>. The detector <b>152</b> is operable to receive signals projected from an emitter (e.g., the avoidance signal emitter and/or homing and alignment emitters of the dock) and (optionally) an emitter of a navigation or virtual wall beacon.

0115A navigational control system may be used advantageously in combination with the mobile robot <b>100</b> to enhance the cleaning efficiency thereof, by adding a deterministic component (in the form of a control signal that controls the movement of the mobile robot <b>100</b>) to the motion algorithms, including random motion, autonomously implemented by the mobile robot <b>100</b>. The navigational control system operates under the direction of a navigation control algorithm. The navigation control algorithm includes a definition of a predetermined triggering event. Broadly described, the navigational control system, under the direction of the navigation control algorithm, monitors the movement activity of the mobile robot <b>100</b>. The predetermined triggering event is a specific occurrence or condition in the movement activity of the robot <b>100</b>. Upon the realization of the predetermined triggering event, the navigational control system operates to generate and communicate a control signal to the robot <b>100</b>. In response to the control signal, the mobile robot <b>100</b> operates to implement or execute a conduct prescribed by the control signal, i.e., the prescribed conduct. This prescribed conduct represents a deterministic component of the movement activity of the mobile robot <b>100</b>.

0116While the mobile robot <b>100</b> is vacuuming, it will periodically approach the stationary dock. Contact with the dock could damage or move the dock into an area that would make docking impossible. Therefore, avoidance functionality is desirable. To avoid inadvertent contact, the dock may generate an avoidance signal that may be transmitted from the emitter on the top of the dock. The radial range of the avoidance signal from the dock may vary, depending on predefined factory settings, user settings, or other considerations. At a minimum, the avoidance signal need only project a distance sufficient to protect the dock from unintentional contact with the mobile robot <b>100</b>. The avoidance signal range can extend from beyond the periphery of the dock, to up to and beyond several feet from the dock, depending on the application.

0117The mobile robot <b>100</b> may be any suitable robot and associated computing device(s), and it will be appreciated that not all of the components, features and functionality described herein are required in mobile robots according to embodiments of the present disclosure. With reference to <figref idref="DRAWINGS">FIGS. 2A-2C</figref>, the example mobile robot <b>100</b> may include a chassis <b>210</b>, a controller <b>220</b>, memory <b>222</b>, a battery <b>224</b>, a battery charger <b>226</b>, a human-machine interface (HMI) <b>228</b>, a drive system <b>230</b>, a mapping/navigation system <b>240</b>, a service operation system <b>242</b> (also referred to herein as “cleaning system” and “cleaning head”), a wireless communication system <b>250</b>, an IR emitter <b>260</b>, and environmental sensors <b>270</b>A-H, a debris bin <b>242</b>A (to store debris collected by a cleaning operation), a bin level sensor <b>242</b>B, a dirt extraction sensor <b>242</b>C (to detect the density of characteristics of the debris collected by the cleaning operation), an indicator light <b>274</b>A, an audio transducer <b>274</b>B, and a cleaning mode selection switch or button <b>274</b>C.

0118The environmental sensors <b>270</b>A-<b>270</b>H may include a camera <b>270</b>B mounted on a top surface of the mobile robot <b>100</b>, as shown in the top perspective view of <figref idref="DRAWINGS">FIG. 2A</figref>. The camera <b>270</b>B can be used to navigate the robot <b>100</b> and acquire images for other operational use. In some embodiments, the camera <b>270</b>B is a visual simultaneous location and mapping (VSLAM) camera and is used to detect features and landmarks in the operating environment and build an occupancy map based thereon.

0119As shown in the bottom perspective view of <figref idref="DRAWINGS">FIG. 2B</figref>, the mobile robot <b>100</b> may further include a bumper <b>104</b>, cliff sensors <b>195</b>A-<b>195</b>D, an edge brush <b>111</b> mounted or otherwise positioned at a periphery of the mobile robot housing <b>106</b>. The housing <b>106</b> is illustrated in <figref idref="DRAWINGS">FIGS. 2A-2B</figref> as having a squared front section on which the bumper <b>104</b> is mounted; however, the housing may have a rounded or circular shape in other embodiments. A caster wheel <b>196</b> may be provided on the underside of the mobile robot <b>100</b>. In some embodiments, the caster wheel <b>196</b> may be positioned at an opposite end of the mobile robot <b>100</b> than the cleaning head <b>242</b>, with the drive rollers/tracks <b>232</b>A, <b>232</b>B therebetween, such that the cleaning head <b>242</b> is a cantilevered arrangement. The mobile robot <b>100</b> may also include downward- or floor-facing camera <b>197</b>.

0120The cameras <b>270</b>B and <b>197</b> and/or other imaging sensors may collectively operate as an image sensing device that can be used to acquire information for guidance and operation of the robot during various operations of the mobile robot <b>100</b>. In some embodiments, the image sensing device is used to detect obstacles and hazards about the mobile robot <b>100</b> so that those obstacles and hazards can be avoided or otherwise addressed. Within the operational range of the image sensor device, a downwardly directed beam can be used to detect obstacles at or near the floor level as well as cliffs or depressions in the floor. An upwardly directed beam can be used to detect obstacles at or above the top of the robot <b>100</b> in order to detect and avoid obstacles under which the robot may become wedged. In some embodiments, the image sensing device is operative to effectively detect objects and voids up to at least 10 inches forward of the mobile robot <b>100</b> and, in some embodiments, up to at least 12 inches.

0121The controller <b>220</b> may include any suitably configured processor or processors. The processor(s) may include one or more data processing circuits, such as a general purpose and/or special purpose processor (such as a microprocessor and/or digital signal processor) that may be collocated or distributed across one or more networks. The processor is configured to execute program code stored in the memory <b>222</b>, described below as a computer readable storage medium, to perform some or all of the operations and methods that are described above for one or more of the embodiments. The memory <b>222</b> is representative of the one or more memory devices containing the software and data used for facilitating operations of the robot in accordance with some embodiments of the present disclosure. The memory <b>222</b> may include, but is not limited to, the following types of devices: cache, ROM, PROM, EPROM, EEPROM, flash, SRAM, and DRAM. The processor is thus in communication with the controller <b>200</b>, memory <b>222</b>, the cleaning system <b>242</b> and drive system <b>230</b>.

0122The drive system <b>230</b> may include any suitable mechanism or system for actively and controllably transiting the robot <b>100</b> through the living space <b>20</b>. According to some embodiments, the drive system <b>230</b> includes a roller, rollers, track or tracks <b>232</b>A, <b>232</b>B and one or more onboard (i.e., carried by the mobile robot <b>100</b>) electric motors <b>234</b> (collectively referred to herein as a “drive” or “drive system”) operable by the controller <b>220</b> to convey the robot <b>100</b> across the floor of the operating environment <b>10</b>.

0123The service operation system <b>242</b> may be optional in some embodiments, and is operable to execute a service operation in the living space <b>20</b>. According to some embodiments, the service operation system <b>242</b> includes a floor cleaning system that cleans a floor surface of the living space <b>20</b> as the robot <b>100</b> transits through the space <b>20</b>. In some embodiments, the service operation system <b>242</b> includes a suction head and an onboard vacuum generator to vacuum clean the floor. In some embodiments, the service operation system <b>242</b> includes an end effector such as (but not limited to) a sweeping or mopping mechanism, one or more rotating brushes, rollers, wet or dry stationary or oscillating and/or vibrating cloths, or multilayer pad assemblies.

0124The wireless communication system <b>250</b> includes a wireless communication transceiver or module <b>252</b> and an associated antenna <b>254</b> that are configured to send/receive data to/from one or more computing devices to enable wireless communication between the robot <b>100</b> and the various other connected devices in the operating environment <b>10</b>, as well as network segments serviced by WAPs, gateways and hubs which make up the private network <b>160</b>, of which the mobile robot <b>100</b> constitutes a node. For example, the wireless communication transceiver or module <b>252</b> may be a Wi-Fi module. In some embodiments, the robot <b>100</b> may communicate wirelessly directly with the dock using narrowband or broadband RF communication. For example, if the robot <b>100</b> is not equipped with a transmitter compatible with a wireless access point (WAP), the robot <b>100</b> may communicate with the dock, which may in turn relay data from the robot <b>100</b> onto the private network and onward to the intended network entity (such as the robot management server <b>200</b>).

0125In some embodiments, the mobile robot <b>100</b> may be generally configured in the manner of or include features from floor cleaning robots and/or robots as described in U.S. Pat. No. 7,024,278 and U.S. Published Application No. 2007/0250212, the disclosures of which are incorporated herein by reference, with suitable modifications. Other suitable configurations for the vacuum cleaning system are disclosed in U.S. Pat. No. 9,215,957 to Cohen et al., U.S. Patent Publication No. 2016/0166126 to Morin et al., U.S. Pat. No. 8,881,339 to Gilbert, Jr. et al., U.S. Patent Publication No. 2014/0222279 to Stout et al., and U.S. patent application Ser. No. 15/647,677 to O'Brien et al. filed Jul. 12, 2017, the disclosures of which are incorporated herein by reference. Suitable structured light image sensing devices for use as the image sensing devices may include the Global Shutter Image Sensor available from PixArt Imaging, Inc. of Taiwan. The robot <b>100</b> may further include a bin detection system for sensing an amount of debris present in a cleaning bin (e.g., as described in U.S. Patent Publication No. 2012/0291809, the entirety of which is hereby incorporated by reference).

0126Execution of a room segmentation aims to reach optimality by avoiding unnecessary changes in direction when traversing an open environment. Also, based on predefined boundaries, frequent traveling across rooms while cleaning can be reduced or eliminated. This makes the behavior more understandable to the user, who may expect or desire the robot to systematically finish one room after another.

0127<figref idref="DRAWINGS">FIG. 3</figref> is a block diagram of an electronic computing device that may be an example of the robot management node <b>200</b> according to some embodiments of the present invention. The electronic device <b>300</b> includes one or more network interfaces <b>330</b>, processor circuitry (“processor”) <b>310</b>, and memory <b>320</b> containing program code <b>322</b>. The processor <b>310</b> may include one or more data processing circuits, such as a general purpose and/or special purpose processor (e.g., microprocessor and/or digital signal processor) that may be collocated or distributed across one or more networks. The processor <b>310</b> is configured to execute program code <b>322</b> in the memory <b>320</b>, described below as a computer readable storage medium, to perform some or all of the operations and methods that are described above for one or more of the embodiments. The memory <b>320</b> is representative of the one or more memory devices containing the software and data used for facilitating operations for managing the robot in accordance with some embodiments of the present invention. The memory <b>320</b> may include, but is not limited to, the following types of devices: cache, ROM, PROM, EPROM, EEPROM, flash, SRAM, and DRAM.

0128The electronic device <b>300</b> may also include a display device <b>340</b> (which may display a drag-and-drop user interface) and/or an input device <b>350</b>, such as a keyboard, touch sensitive display device, mouse, pointer, etc. The network interface <b>330</b> can be configured to communicate through one or more networks with any associated available resource server(s) and/or data repositories.

0129Although <figref idref="DRAWINGS">FIG. 3</figref> illustrates hardware/software architectures that may be used in managing a robot according to some embodiments described herein, it will be understood that the present invention is not limited to such a configuration but is intended to encompass any configuration capable of carrying out operations described herein.

0130<figref idref="DRAWINGS">FIG. 4</figref> is a block diagram of a wireless communication device and methods of operation according to some embodiments of the present invention. The wireless communication device <b>400</b> is configured to transmit and/or receive wireless signals over multiple wireless communication interfaces, and may be an example of the user device <b>202</b> described above. For example, a wireless communication device <b>400</b> according to some embodiments can include a cellular communication module, a Bluetooth module, an infrared communication module, a global positioning system (GPS) module, a WLAN module, and/or other types of communication modules.

0131In particular, the wireless communication device <b>400</b> includes a display <b>408</b>, such as a liquid crystal display (LCD) and/or an organic light emitting diode (OLED) display. The wireless communication device <b>400</b> may optionally include a keypad <b>102</b> or other user input mechanism as a user interface <b>410</b> of the device <b>400</b>. In some embodiments, the display <b>408</b> may be provided with touch screen capability to replace and/or supplement the keypad <b>402</b>.

0132The wireless communication device <b>400</b> may include a microphone <b>406</b> and an earphone/speaker <b>404</b>. The front housing may be designed to form an acoustic seal to the user's ear when the earphone/speaker <b>404</b> is placed against the user's head.

0133The keypad <b>402</b>, display <b>408</b>, microphone <b>406</b>, speaker <b>404</b> and camera <b>424</b> may be coupled to a processor <b>427</b>, such as a microprocessor or microcontroller, which may be configured to control operations of the wireless communication device <b>400</b>. The wireless communication device <b>400</b> may further include a transceiver <b>440</b> and a memory <b>428</b> coupled to the processor <b>427</b>. Other electronic circuitry, such as a WLAN communication interface, a Bluetooth interface, a GPS interface, a digital signal processor, etc., may also be included in the electronic circuitry of the device <b>400</b>.

0134The memory <b>428</b> may be a general purpose memory that is used to store both program instructions for the processor <b>427</b> as well as data, such as audio data, video data, configuration data, and/or other data that may be accessed and/or used by the processor <b>427</b>. The memory <b>428</b> may include a nonvolatile read/write memory, a read-only memory and/or a volatile read/write memory. In particular, the memory <b>428</b> may include a read-only memory in which basic operating system instructions are stored, a non-volatile read/write memory in which re-usable data, such as configuration information, directory information, and other information may be stored, as well as a volatile read/write memory, in which short-term instructions and/or temporary data may be stored.

0135The transceiver <b>440</b> typically includes a transmitter circuit <b>442</b>, a receiver circuit <b>444</b>, and a modem <b>446</b>, which cooperate to transmit and receive radio frequency signals to remote transceivers via an antenna array <b>450</b>. The radio frequency signals transmitted between the device <b>400</b> and the remote transceivers may comprise both traffic and control signals (e.g., paging signals/messages for incoming calls), which are used to establish and maintain communication with another party or destination. More particularly, the transceiver <b>440</b>, in operational cooperation with the processor <b>427</b>, may be configured for communication according to multiple radio access technologies. The radio access technologies may include, but are not limited to, WLAN (e.g., 802.11), WiMAX (Worldwide Interoperability for Microwave Access), 3GPP LTE (3rd Generation Partnership Project Long Term Evolution), Universal Mobile Telecommunications System (UMTS), Global Standard for Mobile (GSM) communication, General Packet Radio Service (GPRS), enhanced data rates for GSM evolution (EDGE), DCS, PDC, PCS, code division multiple access (CDMA), wideband-CDMA, and/or CDMA2000. Other radio access technologies and/or frequency bands can also be used in embodiments according to the invention.

0136<figref idref="DRAWINGS">FIG. 5</figref> is a block diagram illustrating a method for operating a mobile floor cleaning robot according to some embodiments of the present invention. The method includes generating a segmentation map of a surface of an enclosed space or other operating environment using the mobile floor cleaning robot <b>100</b> to identify multiple regions of the surface (block <b>504</b>). In some embodiments, an enclosed space may include an occupancy, whether occupied or not, including, for example, a residential building, a commercial building, a storage building, a manufacturing building and/or a portion thereof.

0137In some embodiments, the segmentation map is generated from an occupancy grid that is generated by the mobile floor cleaning robot during one or more exploration and/or cleaning missions, more generally referred to herein as navigation. For example, the occupancy grid may be generated by exploring the surface of the enclosed space with the mobile floor cleaning robot by generating one or more visual maps for localization of the mobile floor cleaning robot within the enclosed space. In some embodiments, the visual maps may be generated by detecting features using multiple images that may be captured by one or more cameras in the mobile floor cleaning robot. For example, visual features (VSLAM) generated from the video images may be used in order build the “visual map” used to localize the robot. For each feature, a feature description may be determined and the feature may be classified by performing a feature lookup in a feature database. The occupancy grid may be integrated with the visual map, and may further include information generated from bumper hits or collisions (“bumping” events) and/or IR/PixArt data (i.e., time of flight and/or ranging sensor data for obstacle detection).

0138Some embodiments provide that extracted regions are then merged into corridors and rooms according to heuristic decision rules. Then for each region, starting locations and directions are calculated for systematic room cleaning. Finally, regions are systematically cleaned by the robot according to the shortest sequence computed by route optimization and/or by a sequence selected by a user. Final segmentation is then used on the robot for systematic room-by-room cleaning. In some embodiments, both automatic and user-guided segmentation results are stored in the Cloud in order to incrementally evaluate and improve the segmentation procedure. In other embodiments, the automatic and/or user-guided segmentation results may be stored in a memory of the robot or at a local hub, for example, for sharing with one or more other robots that are local to the operating environment.

0139A pose and/or position of the mobile floor cleaning robot may be determined and associated with the corresponding images. A landmark in the occupancy grid may be generated based on the images. The landmark may be associated with the pose and/or the position of the mobile floor cleaning robot. Some embodiments provide that the occupancy grid may be augmented and/or updated with updated landmark data that is generated over multiple missions of the mobile floor cleaning robot. Features may also be generated from the grid map, for example, to transfer an existing occupancy grid to a recently generated one. This may be used to transfer existing room segmentation to a newly generated map. That is, embodiments of the present invention may utilize multiple mechanisms in combination, such as, but not limited to, robot mapping and localization based on visual features, and transfer of segmentation based on features generated from the occupancy grid.

0140In some embodiments, the occupancy grid includes multiple pixels each having a grayscale value that corresponds to whether the pixel location corresponds to a surface location that is occupied, traversable or unexplored. As used herein, the term “traversable” may refer to any surface location in the enclosed space that is traversable by a robot as disclosed herein. Some embodiments provide that the segmentation map may be generated by applying a median filter to the pixels. In some embodiments, the median filter may replace each pixel value with a median value corresponding to pixels that surround the corresponding pixel. For example, some non-limiting embodiments provide that the median value is determined based on the values of the pixels that are within five pixels of each pixel. In this manner, a filtered occupancy grid may be generated.

0141A binary image of the filtered occupancy grid may be computed by applying a thresholding function to the filtered occupancy grid. A watershed transformation may be performed on the binary image to partition the binary image into multiple regions and to identify the vertices of each of the regions by applying a distance transform.

0142Some embodiments further include performing a clutter removal operation on the regions in the binary image. For example, removing clutter may be performed by identifying a boundary of the binary image, identifying and removing pixels that correspond to obstacles that are detached from the boundary of the binary image and identifying and removing pixels that correspond to obstacles that are attached to the boundary region of the binary image, for example, by accessing a data store including ground truth information generated at least in part from labeling inputs received from multiple users as described in greater detail below with reference to <figref idref="DRAWINGS">FIGS. 35A-35D</figref>. Some embodiments further include merging the vertices of the regions to join the regions and to define corridors that are adjacent the regions.

0143Some embodiments provide further processing operations that identify clutter in a map represented by the image M. First, a boundary of an image Xis identified by taking the distance transform to its inverse. The boundary may refer to the connected set of all obstacle pixels starting at pixel (1, 1). Note that for a map M it can be assumed that all pixels on the edges of the image have value 1 after thresholding, i.e., the map is bounded by either obstacles or unexplored regions. Detached obstacles R_det, i.e., obstacles that are not connected to the boundary, are then determined. The attached obstacles R_attare computed by segmenting the boundary using watershed segmentation ws(.) and removing regions that contains pixels on the edges of the original image. Algorithm 1, which follows, describes the computation of the obstacle regions from a binary image in detail:

0000<tables id="TABLE-US-00001" num="00001"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="1"><colspec colname="1" colwidth="217pt" align="left" /><thead><row><entry namest="1" nameend="1" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry>Algorithm obstExt(X)</entry></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="2"><colspec colname="1" colwidth="28pt" align="left" /><colspec colname="2" colwidth="189pt" align="left" /><tbody valign="top"><row><entry>|</entry><entry>DT ← dt (<o ostyle="single">X</o>)</entry></row><row><entry>|</entry><entry>C ← ∅, Q ← ∅// Closed list and queue</entry></row><row><entry>|</entry><entry>Q.PUSH((1,1))//Initial pixel//</entry></row><row><entry>|</entry><entry>Initialize Boundary</entry></row><row><entry>|</entry><entry>for u ← 1 . . . n do</entry></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="3"><colspec colname="1" colwidth="28pt" align="left" /><colspec colname="2" colwidth="35pt" align="left" /><colspec colname="3" colwidth="154pt" align="left" /><tbody valign="top"><row><entry>|</entry><entry>|</entry><entry>for v ← 1 . . . m do</entry></row><row><entry>|</entry><entry>|</entry><entry>└B(u,v) ← 0</entry></row><row><entry>|</entry><entry>└</entry><entry /></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="2"><colspec colname="1" colwidth="28pt" align="left" /><colspec colname="2" colwidth="189pt" align="left" /><tbody valign="top"><row><entry>|</entry><entry>while Q ≠ ∅ do</entry></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="3"><colspec colname="1" colwidth="28pt" align="left" /><colspec colname="2" colwidth="35pt" align="left" /><colspec colname="3" colwidth="154pt" align="left" /><tbody valign="top"><row><entry>|</entry><entry>|</entry><entry>p ← Q.POP( )</entry></row><row><entry>|</entry><entry>|</entry><entry>C ← C ∪ {p}</entry></row><row><entry>|</entry><entry>|</entry><entry>B(p) ← 1</entry></row><row><entry>|</entry><entry>|</entry><entry>if DT(p) ≥ 1 then</entry></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="4"><colspec colname="1" colwidth="28pt" align="left" /><colspec colname="2" colwidth="35pt" align="left" /><colspec colname="3" colwidth="28pt" align="left" /><colspec colname="4" colwidth="126pt" align="left" /><tbody valign="top"><row><entry>|</entry><entry>|</entry><entry>|</entry><entry>for q ∈ Neighbors(p) do</entry></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="5"><colspec colname="1" colwidth="28pt" align="left" /><colspec colname="2" colwidth="35pt" align="left" /><colspec colname="3" colwidth="28pt" align="left" /><colspec colname="4" colwidth="28pt" align="left" /><colspec colname="5" colwidth="98pt" align="left" /><tbody valign="top"><row><entry>|</entry><entry>|</entry><entry>|</entry><entry>|</entry><entry>if q ∉ C then</entry></row><row><entry>|</entry><entry>|</entry><entry>|</entry><entry>|</entry><entry>└Q.PUSH(q)</entry></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="4"><colspec colname="1" colwidth="28pt" align="left" /><colspec colname="2" colwidth="35pt" align="left" /><colspec colname="3" colwidth="28pt" align="left" /><colspec colname="4" colwidth="126pt" align="left" /><tbody valign="top"><row><entry>|</entry><entry>|</entry><entry>|</entry><entry>└</entry></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="3"><colspec colname="1" colwidth="28pt" align="left" /><colspec colname="2" colwidth="35pt" align="left" /><colspec colname="3" colwidth="154pt" align="left" /><tbody valign="top"><row><entry>|</entry><entry>|</entry><entry>└</entry></row><row><entry>|</entry><entry>└</entry><entry /></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="2"><colspec colname="1" colwidth="28pt" align="left" /><colspec colname="2" colwidth="189pt" align="left" /><tbody valign="top"><row><entry>|</entry><entry><img file="US2018284792A1_D0001.tif" />_det← cc_R(X − B)</entry></row><row><entry>|</entry><entry><img file="US2018284792A1_D0002.tif" />_att← cc_R(ws(B))</entry></row><row><entry>|</entry><entry><img file="US2018284792A1_D0003.tif" />_bou← {r|r ∈ <img file="US2018284792A1_D0004.tif" />_att, (x,y) ∈ r, x = 1 or y = 1} </entry></row><row><entry>|</entry><entry><img file="US2018284792A1_D0005.tif" />_att← <img file="US2018284792A1_D0006.tif" />_att\ <img file="US2018284792A1_D0007.tif" />_bou</entry></row><row><entry>└</entry><entry><img file="US2018284792A1_D0008.tif" />_obs← <img file="US2018284792A1_D0009.tif" />_att∪ <img file="US2018284792A1_D0010.tif" />_det</entry></row><row><entry namest="1" nameend="2" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>

0144The region set R_obscontains the obstacles which can be discriminated between clutter and walls. The following filter rules distinguish between walls and clutter, i.e., R_obsR₃is the clutter and R₃contains no clutter.

0145R₁=R_obs\{r_i∈R_obs|A(r_i)&lt;n₁ΛE(r_i)&lt;n₂}

0146R₂=R₂\{r_i∈R₁|maxline(r_i)&lt;n₃},

0147R₃=R₂\{E(r_i)&lt;n₄ζS(r_i)&lt;n₅}.

0148The function l=maxline(r_i) returns the length of the longest line segment found on the boundary of region r_i. A Split- and-Merge algorithm can be utilized for detecting lines in region boundaries. In some embodiments, n₁=0.0, n₂=<b>2</b>.<b>0</b>, n₃=0.7, n₄=0.8, n₅=0.38 were used as examples. Additionally or alternatively, clutter can be distinguished responsive to predetermined movements of the robot over a given time period or surface area. For example, obstacles detected responsive to a predetermined number of robot collision events or non-contact obstacle detection with an optical or acoustic sensor and/or changes in direction in a relatively small area and/or distance of travel may be used to distinguish clutter. In a specific example, numerous detections of obstacles to the left or right of the robot when traveling in a substantially straight path over longer distances may be indicative of walls or corridors, while numerous detections of obstacles to the left or right of the robot responsive to non-linear travel over shorter distances (and accompanying changes in the direction of travel) may be indicative of clutter. Also, a “shape” of an obstacle defined by the collision events and/or changes in direction of the robot may be detected and matched with shapes known to correspond to clutter, for instance, as stored in a data store based on labeling inputs received from multiple users.

0149For example, a robot encountering an obstacle or collection of obstacles constituting clutter may detect and avoid the clutter by turning from a forward heading at least three times in a two foot distance. For example, a robot encountering an obstacle or collection of obstacles constituting clutter may detect and avoid the clutter by turning left and/or right at least three times in a distance of less than 10 feet. For example, a robot attempting to wall follow along an edge of an obstacle or collection of obstacles constituting clutter may detect and avoid the clutter by turning left and/or right at least three times in a distance of less than 5 feet. For example, a robot attempting to wall follow along an edge of an obstacle or collection of obstacles constituting clutter may detect and avoid the clutter by turning left and/or right at least three times in a distance spanning the length of the clutter over a length of 10 feet or less. For example, a robot encountering an obstacle or collection of obstacles constituting clutter may detect and avoid the clutter by turning from a forward heading at least three times in a two foot distance. For example, a robot encountering an obstacle or collection of obstacles constituting clutter may detect and avoid the clutter by turning left and/or right at least three times in a distance of less than 10 feet along one primary direction and at least 10 feet along a direction orthogonal to the primary direction so as to identify an outer boundary of the clutter region.

0150Once the segmentation map is generated, the segmentation map may be sent to a user device that includes a display that is configured to display the segmentation map of the surface (block <b>504</b>). Some embodiments provide that the user device includes a wireless communication device <b>400</b>. In some embodiments, the segmentation map may be wirelessly transmitted using at least one wireless communication link.

0151Some embodiments provide that the wireless communication link may include a local area network that provides communication between the user device and the mobile floor cleaning robot. In some embodiments, the user device and the mobile floor cleaning robot may communicate with one another directly via one or more near field communications protocols, such as Bluetooth or Wi-Fi, among others. In some embodiments, the user device and mobile floor cleaning robot may communicate with one another through one or more intervening wireless communication devices. In some embodiments, the wireless communication link includes a server that is configured to provide a persistent storage of the occupancy grid, the segmentation map and/or a modified segmentation map, among others.

0152In some embodiments, the segmentation map includes operational data corresponding to the mobile floor cleaning robot, such as the segmentation maps shown in <figref idref="DRAWINGS">FIGS. 20G-20K</figref>. Operational data may include a cleaning function map that identifies dirt and/or debris collection that corresponds to the quantity of dirt and/or debris that is collected by the mobile floor cleaning robot as a function of location on the segmentation map. In some embodiments, the cleaning function map may be a heat map that indicates which portions of the enclosed space were the dirtier than other portions. Some embodiments provide that the cleaning function map may be provided for each of the regions individually. In some embodiments, the cleaning function map may correspond to either a single cleaning mission or multiple cleaning missions. The multiple cleaning missions may include a defined subset of all cleaning missions completed in the enclosed space and/or region or may corresponding to all of the cleaning missions for which data is stored.

0153Some embodiments include sending the segmentation map of the surface to a remote computing device (block <b>506</b>). Some embodiments provide that the remote computing device may include electronic device <b>300</b> described above. In some embodiments, the remote computing device <b>300</b> may communicate with the mobile floor cleaning robot and with a user mobile terminal, such as wireless communication device <b>400</b> described above. Some embodiments provide that the segmentation map is sent to the remote computing device <b>300</b> in response to the generation of the segmentation map. In this manner, historical data corresponding to the mobile floor cleaning robot may be preserved, maintained, and/or analyzed.

0154Some embodiments provide that a modified segmentation map may be received from the user device based on inputs received from the user into the user device (block <b>508</b>). The modified segmentation map may identify a subset of the regions to be cleaned. For example, a user device may receive an input instructing that specific regions be cleaned without cleaning other ones of the regions, that is, an input specifying “keep-out” regions, as shown for example, in <figref idref="DRAWINGS">FIG. 27</figref>. Additionally, some embodiments provide that the modified segmentation map identifies a specific order in which the selected regions are to be cleaned, as shown for example, in <figref idref="DRAWINGS">FIG. 15</figref>. In some embodiments, the modified segmentation map may include different cleaning schedules for different times and/or intervals. For example, a first cleaning schedule may choose a first subset of regions for cleaning on certain days of the week while a second cleaning schedule may choose a second subset of regions for cleaning on other days of the week. Some embodiments provide that some regions may be included in both the first and the second subset.

0155Operations also include generating an automated floor coverage pattern (block <b>510</b>). Some embodiments provide that the automated floor coverage pattern is based on the segmentation map that includes different regions, as discussed below with reference to <figref idref="DRAWINGS">FIGS. 29A-29D</figref>. In some embodiments, the automated floor coverage pattern includes identification of sub-regions as clutter areas and non-clutter areas, as shown for example in <figref idref="DRAWINGS">FIG. 29A</figref>, and selection of cleaning sequence and rank direction for each of the regions and/or sub-regions, as shown for example in <figref idref="DRAWINGS">FIGS. 30A-30G</figref>. Rank direction may be the primary direction that the mobile floor cleaning robot travels in when cleaning a particular region. For example, the rank direction may define a dimension of a given region that is longer than other dimensions of the region, as shown for example in <figref idref="DRAWINGS">FIG. 31A</figref>. By selecting the rank direction, each region may be cleaned with fewer turns and/or less time.

0156In some embodiments, the rank direction may define a dimension of a sub-region or portion of a region that is without obstacles, that is, a non-clutter area or region, such as the regions <b>914</b> that do not include detected clutter <b>916</b> shown in <figref idref="DRAWINGS">FIG. 9B</figref>. For example, in a region that includes significant clutter areas, the rank direction may be selected based on the dimensionality of the non-cluttered region, regardless of the overall dimensions of the region. Some embodiments provide that the automated floor coverage pattern includes, for each region, a cleaning pattern that includes sequentially selecting an uncluttered portion of the region for coverage (<figref idref="DRAWINGS">FIG. 29B</figref>), a perimeter of the region for coverage (<figref idref="DRAWINGS">FIG. 29D</figref>), and a cluttered portion of the region for coverage (<figref idref="DRAWINGS">FIG. 29C</figref>).

0157Some embodiments include performing a cleaning operation on the surface by sequentially performing the cleaning operation in individual ones of the regions and/or sub-regions based on the automated floor coverage pattern (block <b>512</b>). As such, the cleaning operation in a given region or sub-region (such as a non-clutter area or areas) is completed before starting the cleaning operation in another region (such as a clutter area or areas), according to the sequence specified by the automated floor coverage pattern. Some embodiments provide that the cleaning operation is further performed based on the modified segmentation map. For example, the cleaning operation may only be performed on a subset of the regions or sub-regions responsive to a selection input received from the user device.

0158In some embodiments, the cleaning operation may be performed on specific regions at different respective cleaning levels that may be selected by the user. Different cleaning levels may correspond to a different number of cleaning passes that the mobile floor cleaning robot makes in a given region. For example, the user may specify that a first region gets a higher level cleaning operation that corresponds to the mobile floor cleaning robot making three cleaning passes in the region. In contrast, a second region may be specified to have a lower level of cleaning that corresponds to a single cleaning pass in the region.

0159<figref idref="DRAWINGS">FIG. 6</figref>, which is a block diagram illustrating operations of computer readable program product including computer readable program code <b>600</b> on a user device according to some embodiments of the present invention. Some embodiments provide that the user device includes a user mobile terminal, such as wireless communication device <b>400</b> described above. Operations include receiving occupancy data and/or a segmentation map of a surface of an enclosed space from a robot (block <b>602</b>). As used herein, a segmentation map may refer to a segmented map showing demarcated regions on a mapped area. The segmentation map may include multiple regions of the surface and the robot may be a mobile floor cleaning robot.

0160In some embodiments, the segmentation map is generated by the robot based on data gathered during exploration and/or cleaning missions on the surface. Some embodiments provide that the segmentation map is generated by a remote computing device that may include electronic device <b>300</b> as described above. In such cases, the remote computing device may receive raw data corresponding to surface of the enclosed space from the robot and the raw data may be used to generate the segmentation map.

0161An image of the segmentation map may be displayed on the display of the user device (block <b>604</b>). Some embodiments provide that user device is a hand-held mobile terminal and the display is a touch-screen display. In such embodiments, a graphical user interface may be generated and displayed via the touch-screen display and the user input may be received via the graphical user interface using the touch-screen. The display may include a monitor, a tablet computer, a networked device including a screen and a mouse or touch interface, and/or a combination of remote and local computing devices that are communicatively coupled via a network.

0162A user input corresponding to the segmentation map of the surface may be received via a user input device of the user device (block <b>606</b>), as shown for example in <figref idref="DRAWINGS">FIGS. 10A-10F</figref>. Some embodiments provide that the user input received includes a user input to modify an appearance of the image of the segmentation map. For example, the user input may be to change an angle of perspective of the user relative to the image, such as changing the pan or tilt of the image. The user input may correspond to a dimensional selection such as two-dimensional or three-dimensional. Additionally, a user input may change the zoom to view some or all of the image at different sizes.

0163In some embodiments, the user input may include a map feature selection such as a heat map selection that causes the image of the segmentation map displayed to illustrate a level of dirt and/or debris collected as a function of location on the segmentation map, as shown for example in <figref idref="DRAWINGS">FIG. 24</figref>. In this manner, the image may illustrate which portions and/or regions have required more cleaning <b>2420</b> relative to other portions and/or regions <b>2410</b>. Some embodiments provide that a map feature may include a clutter map selection that causes the image of the segmentation map displayed to selectively illustrate clutter on the segmentation map, as shown for example in <figref idref="DRAWINGS">FIGS. 9B and 9C</figref>.

0164Some embodiments provide that the user input to the segmentation map may be used to modify the segmentation map and/or regions thereof. For example, a user input may be a divide region input that is configured to cause one of the regions to be divided into more than one region, as shown for example in <figref idref="DRAWINGS">FIGS. 11A-11C</figref>. Similarly, a user input may be a combine region input that is configured to cause multiple ones of the regions to be combined into a single region, as shown for example in <figref idref="DRAWINGS">FIGS. 10A-10F</figref>.

0165Some embodiments provide that the user input to the segmentation map may include a cleaning mission instruction. For example, a cleaning mission instruction may include a cleaning order instruction that includes an identification of the order in which the regions are to be cleaned in a given mission, as shown for example, in <figref idref="DRAWINGS">FIG. 15</figref>. The cleaning order instruction may include an identification regarding a subset of the regions that are to be cleaned in the given mission. Embodiments directed to cleaning mission instructions and associated user inputs are described in greater detail below with reference to <figref idref="DRAWINGS">FIGS. 29A-32</figref>.

0166In some embodiments, the user input may include one or more cleaning mission schedules for the enclosed space. For example, a cleaning mission instruction may include a first cleaning mission instruction that identifies a first selection of the regions to be cleaned and a second cleaning mission that identifies a second selection of the regions to be cleaned. In this example, the first cleaning mission instruction may be scheduled to be performed on certain days of the week and/or times of day, while the second cleaning mission instruction may be scheduled to be performed on different days and/or at different times.

0167In some embodiments, the cleaning mission instruction includes different levels of cleaning for different regions. For example, a first region or set of regions may be designated to receive a first cleaning level and a second region or set of regions may be designated to receive a second cleaning level that is different from the first cleaning level. Different cleaning levels may be accomplished by specifying the number of times the robot has to traverse the corresponding region, also referred to herein as the number of cleaning passes.

0168In some embodiments, the cleaning mission instruction includes an identification of a portion of the enclosed space floor space that is not to be cleaned by the mobile floor cleaning robot, as shown for example in <figref idref="DRAWINGS">FIG. 27</figref>. Such designations may be selected by region and/or by portion of the region. In some embodiments, the cleaning mission instruction may specify that the portion of the enclosed space floor space that is not to be cleaned is a global instruction and thus is persistent from mission to mission. However, some embodiments provide that the identification of the portion of the enclosed space floor space that is not to be cleaned is a one mission and/or a limited number of missions instruction.

0169Some embodiments provide that the cleaning mission instruction includes a primary cleaning direction for each of the regions, as shown for example in <figref idref="DRAWINGS">FIG. 29A</figref>. The primary cleaning direction, which may be referred to as rank, identifies a primary cleaning direction that the mobile floor cleaning robot may clean in obstacle free portions of a cleaning operation. In some embodiments, rank may be determined for an entire region and/or a portion thereof.

0170Some embodiments provide that the user input includes a request for user preference data, which may be displayed responsive thereto. User preferences that may be displayed include historical user preference data and/or user preference data that corresponds to a single cleaning mission.

0171In some embodiments, the user input includes a robot team configuration input that corresponds to multiple different robots. The robot team configuration may include a first cleaning mission instruction corresponding to a first mobile floor cleaning robot and a second cleaning mission corresponding to a second mobile floor cleaning robot. In some embodiments, the first and second cleaning missions correspond to the entire segmentation map of the surface of the enclosed space. For example, a first cleaning mission may clean a first portion of the regions and a second cleaning mission may clean a second portion of the regions.

0172However, some embodiments provide that the first and second cleaning missions correspond to different segmentation maps of different portions of the enclosed space. For example, in the context of a multilevel occupancy, a first segmentation map may correspond to a first level and a second segmentation map may correspond to a second level.

0173Some embodiments provide that the first mobile floor cleaning robot performs a first type of cleaning operation and the second mobile floor cleaning robot performs a second type of cleaning operation. For example, the first mobile floor cleaning robot may be a vacuuming robot including sophisticated mapping sensors and the second mobile floor cleaning robot may be a mopping robot having less sophisticated sensors and relying on the mapping created by the vacuuming robot.

0174Some embodiments include sending a modified segmentation map that is modified corresponding to the user input to the mobile floor cleaning robot (block <b>608</b>). Some embodiments provide that the modified segmentation map is sent to a remote server, such as the remote computing device <b>300</b>, that is configured to provide a persistent storage of the modified segmentation map.

0175A status communication that is generated by the mobile floor cleaning robot may be received (block <b>610</b>). Some embodiments provide that a message is displayed in the on the display responsive thereto.

0176In some embodiments, the status communication includes a progress update corresponding to a cleaning mission. The progress update may be displayed as an animated graphic of a cleaning operation on the image of the segmentation map. Some embodiments provide that the progress update is updated in real time and includes a visual indication on the image of the segmentation map that is displayed, as shown for example in <figref idref="DRAWINGS">FIG. 13</figref>.

0177Some embodiments provide that the status communication includes an alert message that identifies a condition that is detected by the mobile floor cleaning robot. For example, the alert message may be reporting an incomplete mission condition that renders the robot unable to complete a cleaning mission. For example, the mobile floor cleaning robot may indicate that it is stuck and unable to proceed, as shown for example in <figref idref="DRAWINGS">FIG. 14</figref>. The condition may correspond to an unexpected obstacle condition. In some embodiments, the mobile floor cleaning robot may detect an obstacle that is not included in the segmentation map and report that this is a new obstacle.

0178<figref idref="DRAWINGS">FIG. 7</figref> is a block diagram illustrating operations of computer program product <b>700</b> on a robot management node according to some embodiments of the present invention. The robot management node may be implemented on and/or include electronic device <b>300</b> as described above. Some embodiments provide that the processer and the memory are included in a cloud-based server that is remote from the robot and the enclosed space.

0179In some embodiments, a cloud-based server may aggregate data corresponding to multiple different users at multiple different enclosed spaces. Examples of the aggregated data may include data corresponding to standard features, fixtures, floorplans, footprints and/or arrangements. For example, multiple users may share a floor plan that is the same as other users and that may be identified by name, number and/or specific dimensional features. Having the data collected corresponding to the same or similar features, fixtures, floorplans, footprints and/or arrangements may provide a data repository that accessible for other users to improve performance and/or accuracy regarding operations such as classifying obstacles.

0180Operations may include receiving occupancy data and/or a segmentation map of a surface of an enclosed space from a robot (block <b>702</b>). The segmentation map may identify multiple regions of the surface based occupancy data that is collected by the robot. Some embodiments provide that multiple segmentation maps that are generated by the robot during multiple cleaning missions are received. In such embodiments, the segmentation map may be updated and/or augmented by later received segmentation maps to provide persistent maps. Some embodiments provide that the segmentation map includes operational data corresponding to the robot.

0181The segmentation map of the surface of the enclosed space may be stored in a data store that is associated with the robot (block <b>704</b>). Some embodiments include storing multiple segmentation maps that are received, and transferring common features between the multiple segmentation maps using feature matching and outlier removal operations described herein to generate persistent maps. In this manner, historical segmentation map data including the modified segmentation map may be stored.

0182User input data is received from a user device (block <b>706</b>). The user device is configured to receive a user input corresponding to the segmentation map. Responsive to receiving the user input, the segmentation map is modified based on the user input data to generate a modified segmentation map. The modified segmentation map may identify a subset of the regions that are to be cleaned, portions or sub-regions of one or more of the regions to be cleaned, and/or an order in which the regions and/or sub-regions are to be cleaned. The modified segmentation map may also be stored and re-used in subsequent navigation and/or cleaning operations performed by the mobile robot.

0183In some embodiments, the segmentation map includes operational data corresponding to the robot. Operational data may include a cleaning function map that identifies dirt and/or debris that was collected in a single cleaning mission and/or in multiple cleaning missions as a function of location on the segmentation map. In this manner, a heat map that identifies regions and/or portions thereof that require more cleaning activity relative to other regions and/or portions thereof may be communicated. For example, some embodiments provide that the operational data includes a cleaning function map that identifies a quantity of passes that are performed by the robot as a function of location on the segmentation map.

0184In some embodiments, the user input data includes a robot team configuration input that includes a first cleaning mission instruction corresponding to a first robot and a second cleaning mission corresponding to a second robot.

0185Operations include sending the modified segmentation map based on the user input data and the segmentation map to the robot (block <b>708</b>). The robot may be operable to perform a cleaning mission responsive to the modified segmentation map.

0186In some embodiments, the cleaning mission includes performing a cleaning operation of regions at different cleaning levels that are selected by the user. The cleaning mission may further provide a rank direction that corresponds to a largest dimension of area in the respective region without obstacles. The different levels may be associated with a number of cleaning passes the robot makes in the corresponding regions. In some embodiments, the rank direction defines a dimension of the region that is longer than other dimensions of the region. Some embodiments provide that the rank direction defines a dimension of a portion of the respective region that is without obstacles and that is different from a dimension of the region that is longer than other dimensions of the region.

0187<figref idref="DRAWINGS">FIGS. 8A through 8C</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating a raw data map, a cleaned map and a segmentation map in an automated map cleaning and segmentation operation according to some embodiments of the present invention. Some embodiments provide that a robot explores a surface of an enclosed space during a first pass. An occupancy grid is generated that includes raw data that is generated during the first pass. The occupancy grid is presented to a user on a user device as a raw data map <b>802</b> as illustrated in <figref idref="DRAWINGS">FIG. 8A</figref>. The raw data map <b>802</b> may include multiple pixels that may be shaded white for traversable space and black for obstacles, including occupancy walls.

0188Referring to the embodiment of <figref idref="DRAWINGS">FIG. 8B</figref>, the robot generates a cleaned map <b>812</b> that is displayed to the user on the user device. The cleaned map <b>812</b> may illustrates the enclosed space with the boundary data <b>816</b> cleaned up or simplified to be substantially more linear than the raw data. The cleaned map <b>812</b> also highlights the traversable space <b>814</b> of the enclosed space surface, for example, by removing visual indications of the detected clutter.

0189<figref idref="DRAWINGS">FIG. 8C</figref> illustrates a segmentation map <b>822</b> that may be displayed to the user on the user device. Similar to the cleaned map <b>812</b>, the segmentation map <b>822</b> illustrates the enclosed space with the boundary data <b>826</b>. Additionally, the traversable space is segmented into regions <b>824</b> that may correspond to different rooms in the enclosed space according to some embodiments. Based on the segmentation map <b>822</b>, the robot may perform room-by-room cleaning in which the cleaning operation in a given region <b>824</b> is completed before the robot begins to perform a cleaning operation in another region <b>824</b>.

0190<figref idref="DRAWINGS">FIGS. 33A through 33D</figref> illustrate operations for generating simplified map boundaries for display via a user interface from raw data maps, such as the map <b>802</b> in <figref idref="DRAWINGS">FIG. 8A</figref>, in greater detail. Referring now to <figref idref="DRAWINGS">FIG. 33A</figref>, a raw data map <b>3302</b> is generated from data collected during navigation of a mobile robot an operating environment, such as occupancy data and VSLAM feature detection data. The raw data map <b>3302</b> provides a pixelated representation of the operating environment, which may not be aesthetically pleasing and/or intuitive to a user. As such, the raw data map <b>3302</b> can be visually simplified for presentation to a user by removing noise to generate the de-noised map <b>3312</b> shown in <figref idref="DRAWINGS">FIG. 33B</figref>, and removing areas identified as clutter to generate a de-cluttered map <b>3322</b> as shown in <figref idref="DRAWINGS">FIG. 33C</figref>. The de-noised map <b>3312</b> may have reduced distortion relative to the raw data map <b>3302</b>, for example, by sharpening or reducing blurred edges. Pixelated edges of the decluttered map <b>3322</b> are replaced by linear segments and/or polygonal shapes to generate the simplified map <b>3332</b> shown in <figref idref="DRAWINGS">FIG. 33D</figref>. The simplified map <b>3332</b> thus replaces data indicating actual boundaries detected by navigation of the mobile robot with simplified boundaries, which may be more aesthetically pleasing and/or intuitive for a user viewing the map <b>3332</b> via a user interface of a user device, such as the user device <b>202</b>. Generating the simplified map <b>3332</b> may also involve merging rooms or regions as described herein. In some embodiments, clutter regions <b>3306</b> representing the areas identified as clutter in <figref idref="DRAWINGS">FIG. 33B</figref> may be added back into to the simplified map <b>3332</b> to provide a more accurate representation of the operating environment to the user. Also, because clutter obstacles can represent recognizable structures, such as furniture, the inclusion of the clutter regions <b>3306</b> in the simplified map <b>3332</b> may make it easier for the user to recognize rooms and corridors. The operations to generate the simplified map <b>3332</b> from the raw data map <b>3302</b> may be performed by the robot <b>100</b>, the robot management server <b>200</b>, and/or the user device <b>202</b>.

0191Some embodiments provide that a user may have a display preference relative to how clutter regions may be displayed. <figref idref="DRAWINGS">FIGS. 9A through 9C</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating user display preference options of raw data, a segmentation map with clutter pixels and segmentation map with clutter regions in operations according to some embodiments of the present invention. Referring to <figref idref="DRAWINGS">FIG. 9A</figref>, a raw data map <b>902</b> may include the identification of multiple pixels that correspond to clutter <b>906</b> or other obstacles in the enclosed space. As shown in <figref idref="DRAWINGS">FIG. 9B</figref>, a segmentation map <b>912</b> may include multiple regions <b>914</b> and the clutter <b>916</b> or other obstacles may be illustrated in each of the regions <b>914</b>. In some embodiments, as illustrated in <figref idref="DRAWINGS">FIG. 9C</figref>, a segmentation map <b>922</b> may include clutter regions <b>926</b> within the regions <b>924</b> that illustrate specific areas in which multiple obstacles are detected. As described herein, clutter may refer to a concentration of obstacles occupying a region of space and preventing systematic, deterministic coverage of the region, and regions including clutter may be referred to herein as clutter or cluttered regions. As described in greater detail below, once the cluttered regions are identified, cleaning operations based on rank optimization may be performed on non-cluttered regions.

0192<figref idref="DRAWINGS">FIGS. 10A through 10F</figref> are graphical representations <b>1002</b> that may be displayed via a user interface of a communication device illustrating operations for performing user region merging on a segmentation map according to some embodiments of the present invention. Referring to <figref idref="DRAWINGS">FIG. 10A</figref>, a user may select a particular region <b>1005</b><i>a </i>of the displayed image of the segmentation map. The selection may be performed using a pointing input such as a mouse, trackball and/or stylus and/or via a touch screen user input. <figref idref="DRAWINGS">FIG. 10B</figref> illustrates the displayed image of the segmentation map with the selected region <b>1005</b><i>a </i>highlighted.

0193Referring to <figref idref="DRAWINGS">FIG. 10C</figref>, the user may select another region <b>1005</b><i>c </i>of the displayed image of the segmentation map that is adjacent the first selected region. <figref idref="DRAWINGS">FIG. 10D</figref> illustrates the displayed image of the segmentation map with both of the selected regions <b>1005</b><i>a, </i><b>1005</b><i>c </i>highlighted. Referring to <figref idref="DRAWINGS">FIG. 10E</figref>, the user may provide a user input to merge the selected regions <b>1005</b><i>a, </i><b>1005</b><i>c </i>into a single region <b>1005</b><i>e. </i>The user input may be received via a menu that identifies actions and/or by a voice command. <figref idref="DRAWINGS">FIG. 10F</figref> illustrates the displayed image of the segmentation map with multiple of the original regions merged into a single region<b>1005</b><i>f. </i>In some embodiments, the merge operation may be performed multiple times on two regions to generate a single region that includes more than two of the originally defined regions. Some embodiments provide that more than two of the regions may be selected before the user input to perform the merge operation is received.

0194<figref idref="DRAWINGS">FIGS. 11A through 11C</figref> are graphical representations <b>1102</b> that may be displayed via a user interface of a communication device illustrating operations for performing user region splitting on a segmentation map according to some embodiments of the present invention. In some embodiments, for various reasons, a user may determine that a region as provided in the segmentation map should be divided into more than one region. For example, a single space may have a defined kitchen area and adjoining living space separated by various design artifacts, such as flooring type changes, and may be treated differently based on cleaning need. Referring to <figref idref="DRAWINGS">FIG. 11A</figref>, an endpoint <b>1105</b><i>a </i>of a division line in the region to be divided may be selected by the user. Then, referring to <figref idref="DRAWINGS">FIG. 11B</figref>, the other endpoint <b>1105</b><i>b </i>of the division line in the region to be divided may be selected by the user. The user may thus provide a user input to divide the selected region into the two regions <b>1105</b><i>c</i><b>1</b>, <b>1105</b><i>c</i><b>2</b> shown in <figref idref="DRAWINGS">FIG. 11C</figref>. In some embodiments, the division operation is selected before the endpoints <b>1105</b><i>a, </i><b>1105</b><i>b </i>of the division line are identified. In such embodiments, the division operation may automatically be performed once the second endpoint <b>1105</b><i>b </i>of the division line is selected.

0195<figref idref="DRAWINGS">FIGS. 12A through 12D</figref> are graphical representations <b>1202</b> that may be displayed via a user interface of a communication device illustrating operations for performing user region labeling on a segmentation map according to some embodiments of the present invention. Each of the regions of the segmentation map may be labeled according to a user preference. Referring to <figref idref="DRAWINGS">FIG. 12A</figref>, a labeling operation may begin with the user selecting a region <b>1205</b><i>a </i>of the segmentation map. <figref idref="DRAWINGS">FIG. 12B</figref> illustrates text box <b>1205</b><i>b </i>that may be displayed that includes the label provided by the user. For example, in the current example, the user selects a region in <figref idref="DRAWINGS">FIG. 12A</figref> and then inputs the label “DEN” <b>1205</b><i>b </i>for the selected region <b>1205</b><i>a </i>in <figref idref="DRAWINGS">FIG. 12B</figref>. <figref idref="DRAWINGS">FIGS. 12C and 12D</figref> illustrate similar operations corresponding to the selection <b>1205</b><i>c </i>and labeling <b>1205</b><i>d </i>of the “LIVING ROOM” region.

0196<figref idref="DRAWINGS">FIG. 13</figref> is a graphical representation <b>1300</b> that may be displayed via a user interface of a communication device illustrating operations for displaying progress of a robot on a segmentation map according to some embodiments of the present invention. As illustrated, the display of the segmentation map may include an indication of the cleaning progress corresponding to a cleaning mission being performed. For example, cleaned regions and/or portions thereof <b>1302</b> may be represented by a first color, pattern, and/or shading that is different from the color, pattern and/or shading representing uncleaned regions and/or portions thereof <b>1304</b>. In this manner, a user may be informed as to which regions and/or portions thereof may expect or not expect to be occupied by the robot.

0197<figref idref="DRAWINGS">FIG. 14</figref> is a graphical representation <b>1402</b> that may be displayed via a user interface of a communication device illustrating operations for providing an alert <b>1405</b> to a user of a failure condition on a segmentation map according to some embodiments of the present invention. In some embodiments, the display may include a message and/or other symbol that is generated based on a message from the robot. For example, the communication device may receive a message that is generated by the robot that indicates a condition that has or is occurring during a cleaning mission. For example, if the robot gets stuck or trapped, an alert message may be received and displayed to the user. Other examples may include alerts generated by the robot that indicate that the robot may not have completed a cleaning mission.

0198<figref idref="DRAWINGS">FIG. 15</figref> is a graphical representation <b>1502</b> that may be displayed via a user interface of a communication device illustrating operations for receiving user input to provide a cleaning sequence (1, 2, 3) of a robot on a segmentation map according to some embodiments of the present invention. In some embodiments, a user may provide a user input corresponding to a cleaning order. For example, each of the regions of the segmentation map may be labeled with a cleaning order number 1, 2, 3. In some embodiments, the cleaning order may be entered via a table <b>1505</b> of the regions that is displayed to the user, while other embodiments provide that the user simply selects the region and enters the value corresponding to the order of cleaning. Some embodiments provide that some of the regions may be deselected for cleaning during a given mission. For example, as illustrated, the kitchen and living room may have an order number of “X” that indicates that these regions are not to be cleaned.

0199<figref idref="DRAWINGS">FIGS. 16A through 16G</figref> are graphical representations <b>1602</b><i>a</i>-<b>1602</b><i>g </i>that may be displayed via a user interface of a communication device illustrating operations for map data aggregation using multiple runs in the same occupancy according to some embodiments of the present invention. <figref idref="DRAWINGS">FIGS. 16A-16D</figref> illustrate displays of a raw data maps that include additional raw data that is gathered by the robot during subsequent runs in the same occupancy. For example, for <figref idref="DRAWINGS">FIG. 16A</figref> to <figref idref="DRAWINGS">FIG. 16B</figref> the raw data map undergoes autorotation (180 degrees) and alignment. As illustrated, the raw data from <figref idref="DRAWINGS">FIG. 16B</figref> reflects an alignment of different portions of the raw data relative to <figref idref="DRAWINGS">FIG. 16A</figref> and a 180 degree rotation relative to the initial raw data illustrated in <figref idref="DRAWINGS">FIG. 16A</figref>. <figref idref="DRAWINGS">FIGS. 16C and 16D</figref> illustrate that additional raw data is used to substantially complete the raw data map. For example, <figref idref="DRAWINGS">FIG. 16C</figref> illustrates additional raw data being added relative to <figref idref="DRAWINGS">FIG. 16B</figref>, thus advancing the overall completion of the raw data map. Similarly, <figref idref="DRAWINGS">FIG. 16D</figref> illustrates even further map completion of the raw data. <figref idref="DRAWINGS">FIGS. 16E through 16G</figref> illustrate the removal of non-static obstacles detected intermittently or only on one or some of the total missions. In this manner, the integrity, accuracy and/or content of the raw data map may be increased during each additional run. For example, <figref idref="DRAWINGS">FIG. 16E</figref> illustrates clutter detection in a portion of a room. <figref idref="DRAWINGS">FIG. 16F</figref> illustrates identification of non-static objects that are part of the detected clutter and <figref idref="DRAWINGS">FIG. 16G</figref> illustrates the removal of non-static objects from the raw data map. It will be understood that persistency of the map may be required in order to accurately overlay maps from each run shown in <figref idref="DRAWINGS">FIGS. 16A-16D</figref> with each other. Also, clutter detection can be significantly improved because, when observing obstacles over time, the location of static obstacles may stay the same whereas the locations of clutter/non-static obstacles may change over time.

0200<figref idref="DRAWINGS">FIGS. 17A and 17B</figref> are graphical representations <b>1702</b><i>a </i>and <b>1702</b><i>b </i>that may be displayed via a user interface of a communication device illustrating a segmentation map from the aggregating operations based on the raw data maps illustrated in <figref idref="DRAWINGS">FIGS. 16A-16G</figref> that are without clutter and that include clutter, respectively, according to some embodiments of the present invention. As illustrated, <figref idref="DRAWINGS">FIG. 17A</figref> is an image of a segmentation map <b>1702</b><i>a </i>corresponding to the raw data maps illustrated in <figref idref="DRAWINGS">FIGS. 16A-16G</figref> without clutter. In contrast, <figref idref="DRAWINGS">FIG. 17B</figref> is an image of a segmentation map <b>1702</b><i>b </i>corresponding to the raw data maps illustrated in <figref idref="DRAWINGS">FIGS. 16A-16G</figref> with clutter regions identified in the corresponding segmentation map regions.

0201<figref idref="DRAWINGS">FIGS. 18A and 18B</figref> are graphical representations <b>1802</b><i>a </i>and <b>1802</b><i>b </i>that may be displayed via a user interface of a communication device illustrating raw data of an occupancy map and a three-dimensional representation of the occupancy map, respectively, based on the raw data maps illustrated in <figref idref="DRAWINGS">FIGS. 16A-16G</figref> according to some embodiments of the present invention. As illustrated, <figref idref="DRAWINGS">FIG. 18A</figref> is the raw data map <b>1802</b><i>a </i>illustrated above as <figref idref="DRAWINGS">FIG. 16E</figref> and <figref idref="DRAWINGS">FIG. 18B</figref> is an image of a three-dimensional representation <b>1802</b><i>b </i>of the occupancy map based thereon.

0202<figref idref="DRAWINGS">FIG. 19</figref> is a graphical representation <b>1900</b> that may be displayed via a user interface of a communication device illustrating an occupancy map illustrating operations for generating persistent information corresponding to an enclosed space according to some embodiments of the present invention. The occupancy map <b>1900</b> illustrates the floor plan of the enclosed space as mapped by the robot. The robot cleaning path <b>1910</b> is identified over all of the available floor surface of the enclosed space. Additionally, landmarks <b>1904</b>, which may be determined, generated and/or assigned by the robot, are identified throughout the occupancy map <b>1900</b>. Specific regions corresponding to multiple ones of the landmarks <b>1904</b> may be identified as landmark regions <b>1902</b> and may include trajectories and/or primitives defined by the landmarks <b>1904</b>. In some embodiments, the term “landmark” may include locations that are specifically identified in one or more maps and may also be referred to as waypoints.

0203<figref idref="DRAWINGS">FIGS. 20A through 20K</figref> are graphical representations <b>2002</b><i>a</i>-<b>2002</b><i>k </i>that may be displayed via a user interface of a communication device illustrating views of occupancy maps and segmentation maps corresponding to data gathered during robot exploration of an enclosed space according to some embodiments of the present invention. As illustrated, <figref idref="DRAWINGS">FIG. 20A</figref> includes an image of a raw data occupancy map <b>2002</b><i>a </i>that includes clutter and other non-traversable regions within the traversable floor space and the path that the robot traversed during the exploration and/or cleaning pass of a cleaning mission. A raw data map <b>2002</b><i>b </i>of the enclosed space is provided in <figref idref="DRAWINGS">FIG. 20B</figref> and a clean occupancy map <b>2002</b><i>c </i>is illustrated in <figref idref="DRAWINGS">FIG. 20C</figref>. <figref idref="DRAWINGS">FIG. 20D</figref> illustrates a display of a clean occupancy map <b>2002</b><i>d </i>with clutter pixels <b>2008</b><i>d. </i>A clean map <b>2002</b><i>e </i>including the path <b>2005</b><i>e </i>that the robot traversed without clutter displayed is illustrated in <figref idref="DRAWINGS">FIG. 20E</figref> and a clean map including the path <b>2005</b><i>e </i>that the robot traversed with clutter displayed as solidly filled patches <b>2008</b><i>f </i>is illustrated in <figref idref="DRAWINGS">FIG. 20F</figref>.

0204A display of a segmentation map <b>2002</b><i>g </i>of the enclosed space without clutter is illustrated in <figref idref="DRAWINGS">FIG. 20G</figref> and a display of the segmentation map <b>2002</b><i>h </i>of the enclosed space with clutter displayed as solidly filled patches <b>2008</b><i>h </i>is illustrated in <figref idref="DRAWINGS">FIG. 20H</figref>. <figref idref="DRAWINGS">FIG. 20I</figref> illustrates a display of a segmentation map <b>2002</b><i>i </i>including the path <b>2005</b><i>e </i>that the robot traversed without the clutter displayed and <figref idref="DRAWINGS">FIG. 20J</figref> illustrates a display of a segmentation map <b>2002</b><i>j </i>including the path <b>2005</b><i>e </i>that the robot traversed with the clutter <b>2008</b><i>j </i>displayed. A display of the segmentation map <b>2002</b><i>k </i>that includes boundaries between regions therein and that includes region numbers <b>1</b>-<b>9</b> is illustrated in <figref idref="DRAWINGS">FIG. 20K</figref>.

0205<figref idref="DRAWINGS">FIG. 21</figref> is a diagram illustrating a system of devices that may be used to provide robot management according to some embodiments of the present invention. The system may include a robot <b>2102</b> that may generate occupancy data and/or a segmentation map <b>2110</b> of a floor surface of an enclosed space. The robot <b>2102</b> may be the same as the robot <b>100</b> described herein. As such, additional description thereof will be omitted. In some embodiments, the occupancy data and/or segmentation map <b>2110</b> may be transmitted to a cloud device and/or system <b>2104</b>. In some embodiments, the cloud device and/or system <b>2104</b> may include the robot management node <b>200</b> described herein. The cloud device and/or system <b>2104</b> may include a processor <b>2105</b> and user data store <b>2106</b> that may store user input history (including identifications of clutter patterns from a plurality of users), occupancy map history, segmentation map history, and/or robot performance and/or communication history, which may be used in accordance with embodiments described herein to maintain and update persistent maps. While some embodiments provide that the robot <b>2102</b> performs the computation of the segmentation map <b>2110</b>, in some embodiments the computation may be performed by the cloud device and/or system <b>2104</b> based on the occupancy data collected by the robot <b>2102</b>.

0206In some embodiments, the cloud device and/or system <b>2104</b> may provide a communication channel between the robot <b>2102</b> and a user device <b>2120</b>. The user device <b>2120</b> may be the same as the user device <b>202</b> described herein. The user device <b>2120</b> may provide user inputs that allow a user to edit and/or label an occupancy and/or segmentation map. Further, the user device <b>2120</b> may be used to modify the segmentation map to include specific instructions for one or more cleaning missions. Such instructions may include cleaning order, rank direction, and/or cleaning level, among others. The user device <b>2102</b> may also provide inputs for selecting display characteristics such as clutter inclusion/exclusion, view dimensionality, zoom, viewing angle, cleaning mission playback options, cleaning progress, expected time of cleaning mission completion, historical, per use and/or aggregate data, among others.

0207Additionally, the user device <b>2120</b> may be operative to receive messages and/or data from the robot <b>2102</b> and/or the cloud device and/or system <b>2104</b>. For example, the user device <b>2120</b> may receive performance data corresponding to the status and/or performance of the robot <b>2102</b> and/or alert messages that are generated by the robot <b>2102</b>, among others. For example, a heat map that displays the level of cleaning that was performed and/or the amount of dirt and/or debris that was collected as a function of location may be provided.

0208<figref idref="DRAWINGS">FIG. 22</figref> is a graphical representation that may be displayed via a user interface of a communication device illustrating history screen <b>2200</b> including a mission coverage map <b>2202</b> corresponding to a cleaning mission according to some embodiments of the present invention. As illustrated, responsive to receiving a user input to provide cleaning history, the user device may display a history screen <b>2200</b> that includes time information such as the date <b>2210</b>, a mission coverage map <b>2202</b>, and cleaning mission performance data <b>2203</b>. Cleaning mission performance data <b>2203</b> may include the duration of the cleaning mission, the start and/or end time, a mission status, the amount of area cleaned during the cleaning mission and/or a dirt detection count.

0209<figref idref="DRAWINGS">FIG. 23</figref> is a graphical representation that may be displayed via a user interface of a communication device illustrating a map <b>2302</b> including real-time reporting corresponding to a cleaning mission of an enclosed space according to some embodiments of the present invention. Responsive to receiving a user input requesting cleaning mission progress data, the user device may display the occupancy map and/or segmentation map <b>2302</b> including an identification of which regions and/or portions thereof are cleaned <b>2305</b> and which regions and/or portions thereof that are not cleaned <b>2307</b> in the current cleaning mission. Some embodiments provide that additional text <b>2303</b> may be provided that identifies which regions have been cleaned and which region is in the cleaning process.

0210<figref idref="DRAWINGS">FIG. 24</figref> is a graphical representation that may be displayed via a user interface of a communication device illustrating a map including a multi-mission coverage display according to some embodiments of the present invention. As illustrated, the occupancy and/or segmentation map may be displayed to include historical coverage data corresponding to multiple cleaning missions. For example, a heat map <b>2402</b> may be generated that identifies areas of the enclosed space that have been historically dirtier relative to other areas. The heat map <b>2402</b> may include different colors, shading, and/or brightness/intensity to illustrate a level of dirt and/or debris collected at respective locations. For example, the shading in the lower right area <b>2420</b> indicates a greater amount of detected and/or collected debris and/or dirt relative to the upper left area <b>2410</b>, which is not shaded.

0211<figref idref="DRAWINGS">FIG. 25</figref> is a graphical representation that may be displayed via a user interface of a communication device illustrating room-based data <b>2503</b> that may be displayed according to some embodiments of the present invention. In some embodiments, the user device may include a user input for display of room based statistics <b>2503</b>. For example, some embodiments provide that a user may select a region <b>2501</b> on a segmentation map <b>2502</b> and that data <b>2503</b> corresponding to that region <b>2501</b> may be displayed responsive to receiving the input. For example, in response to a user input selecting the living room <b>2501</b> on the segmentation map <b>2502</b>, the user device may display summary cleaning mission data <b>2503</b> corresponding to the living room including how many times the living room was cleaned, the date of the last cleaning, the total time spent cleaning the living room, and/or a date range corresponding to the displayed data.

0212<figref idref="DRAWINGS">FIG. 26</figref> is a graphical representation demonstrating operations in which a user input may be provided via a communication device using a voice command <b>2601</b> according to some embodiments of the present invention. In some embodiments, the user device <b>2600</b> may include user inputs to clean a specific room. For example, the user input may be to clean a specific room immediately. In some embodiments, the user input may be received via voice command <b>2601</b> using voice recognition technology. Some embodiments provide that the voice command <b>2601</b> may be received from a device that is external to the user device <b>2600</b>. For example, household management devices that may receive voice commands to perform other functions such as HVAC control, entertainment system control, etc. may receive the voice command <b>2601</b> and translate the message to the user device <b>2600</b>, the robot <b>100</b>, and/or the robot management node <b>200</b>.

0213<figref idref="DRAWINGS">FIG. 27</figref> is a graphical representation of receiving a user input to define a cleaning mission boundary <b>2705</b> responsive to a gesture on a touch screen of a communication device according to some embodiments of the present invention. In some embodiments, a condition in a particular room of the enclosed space <b>2702</b> may be utilized in a manner that renders the cleaning operation undesirable. In such circumstances, a boundary <b>2705</b> may be added to a region of the segmentation map via a user input of the user device. For example, a simple gesture on a touch screen blocking a region <b>2707</b> may cause the modified segmentation map to exclude that region <b>2707</b> from subsequent cleaning missions.

0214<figref idref="DRAWINGS">FIGS. 28A through 28C</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating a raw data map <b>2802</b>, a cleaned map <b>2812</b>, and a cleaned map <b>2822</b> with detected clutter regions <b>2824</b> according to some embodiments of the present invention. As discussed above regarding <figref idref="DRAWINGS">FIG. 8A</figref>, some embodiments provide that a robot may explore a surface of an enclosed space during a first pass. An occupancy grid may be generated that includes raw data that is generated during the first pass. The occupancy grid may be presented to a user on a user device as a raw data map <b>2802</b> as illustrated in <figref idref="DRAWINGS">FIG. 28A</figref>. The raw data map <b>2802</b> may include multiple pixels that may be shaded white for traversable space and black for obstacles, including occupancy walls.

0215Referring to <figref idref="DRAWINGS">FIG. 28B</figref>, the robot may generate a cleaned map <b>2812</b> that may be displayed to the user on the user device. The cleaned map <b>2812</b> may illustrate the enclosed space with the boundary data <b>2816</b> cleaned up to be substantially more linear than the raw data and may highlight the traversable space <b>2814</b> of the enclosed space surface.

0216<figref idref="DRAWINGS">FIG. 28C</figref> illustrates cleaned map with detected clutter regions <b>2822</b> that may be displayed to the user on the user device. Relative to the cleaned map <b>2812</b>, the cleaned map with detected clutter regions <b>2822</b> illustrates the enclosed space with the clutter regions <b>2824</b>. In some embodiments, the operations may further determine whether the clutter <b>2824</b> is dynamic clutter or static clutter. For example, the robot may determine that clutter <b>2824</b> is static clutter by determining a probability that the clutter will be encountered more frequently, and/or by accessing a data store including patterns of clutter identified by one or more other robots and/or users, as described below with reference to <figref idref="DRAWINGS">FIGS. 35A through 35D</figref>. Some embodiments provide that clutter that is determined to be static may be marked on the map using darker cells relative to clutter that has a lower probability of being encountered repeatedly. The darker cells may indicate known areas that are not traversable. By identifying such features, the robot may be operative to identify clutter regions that may be cleaned after traversable space is cleaned deterministically in ranks. Cleaning non-clutter regions first can increase mission efficiency.

0217<figref idref="DRAWINGS">FIGS. 29A through 29D</figref> are graphical representations that may be displayed via a user interface of a communication device illustrating a segmentation map that includes respective area-specific cleaning operations according to some embodiments of the present invention. Referring to <figref idref="DRAWINGS">FIG. 29A</figref>, a segmentation map <b>2902</b> includes regions and portions thereof identified as open area <b>2904</b> and clutter area <b>2906</b>. As provided herein, an open area may refer to an identified area of the enclosed space, regions and/or portion thereof that is identified as having few or no identified clutter or clutter areas, and is also referred to herein as a non-clutter area. As mentioned, clutter may refer to obstacles occupying a surface and preventing systematic and deterministic coverage of the surface in ranks (rows). In some embodiments, an entire region of a segmented map may be determined to be an open area or a clutter area. Some embodiments provide that an open area and/or a clutter area may be defined as a portion of a region, also referred to herein as a sub-region. For example, the clutter area <b>2906</b> is a portion of a region that also includes defined open areas <b>2904</b>. Also, an open area <b>2904</b> may include small areas of clutter therein. As shown in <figref idref="DRAWINGS">FIG. 29A</figref>, the displayed segmentation map <b>2902</b> may visually highlight these differently-classified sub-regions <b>2904</b> and <b>2906</b>, for example, by displaying visually-distinct boundaries around the clutter area <b>2906</b> and the non-clutter areas <b>2904</b>.

0218In some embodiments, the clutter area may be determined based on a clutter density determination. For example, some embodiments provide that clutter density may be determined by measuring dimensions between specific clutter points and nearby obstacles or clutter points. This dimension and/or function thereof may be compared to some dimension of the robot and/or portion thereof. A result of the comparison may determine whether or not to classify the obstacle as clutter. For example, if the measured distance is less than a threshold that is based on a dimension of the robot, then the obstacle may be determined to be clutter. Relative dimensions of the robot may include total robot length, width, and/or cleaning head work width, among others. Non-limiting examples include a range of about 70% to about 300% of the work width of the cleaning head. In other words, obstacles that are spaced apart by a distance of about 70% to about 300% of the width of the cleaning head or roller(s) of the robot may constitute (or be interpreted as) “clutter” because the robot cannot traverse floor space occupied by such obstacles in unimpeded, deterministic ranks. In the example of <figref idref="DRAWINGS">FIG. 29A</figref>, the lowermost area shown is classified as an open or non-clutter area <b>2904</b> based on the relatively low density of clutter <b>2908</b> therein.

0219Once the open areas <b>2904</b> and/or the clutter areas <b>2906</b> are identified, a route and strategy for cleaning the different areas of the surface of the enclosed space, also referred to herein as a coverage pattern, may be determined by the mobile robot <b>100</b>, the robot management node <b>200</b>, and/or the user device <b>202</b>. The coverage pattern may define an improved or optimized cleaning strategy that treats areas differently with respect to their recognized context or classification, for instance, by indicating sequential cleaning of regions or sub-regions of the surface based on their respective classifications. For example, the coverage pattern may indicate first executing cleaning of the open areas <b>2904</b> as shown in <figref idref="DRAWINGS">FIG. 29B</figref> so as to temporarily ignore and avoid the clutter areas <b>2906</b>, then executing cleaning of the clutter areas <b>2906</b> as shown in <figref idref="DRAWINGS">FIG. 29C</figref>, and then executing cleaning of the perimeter <b>2930</b> as shown in <figref idref="DRAWINGS">FIG. 29D</figref>.

0220Referring to <figref idref="DRAWINGS">FIG. 29B</figref>, in performing the open area cleaning, a rank direction <b>2910</b> may be determined for each of the open areas <b>2904</b>. In some embodiments, the rank direction <b>2910</b> may correspond to the longest dimension in the given region. However, some embodiments provide that the rank direction is based on the dimensionality of the open area that may be different from that of the region, for example, as specified by a user input. Once the cleaning operation of the open areas is determined, then an open area cleaning operation may be performed. This improves efficiency in some embodiments because the robot is not slowed by navigation of clutter (and associated battery depletion) in areas <b>2904</b>. As the open areas <b>2904</b> can represent a majority of the floor space to be cleaned, the robot can complete cleaning of a majority (for example, up to 80% to 90% or more) of the floor space first, and then focus on the more laborious and potentially mission-disrupting clutter areas <b>2906</b>. For example, the robot could be trapped between chairs or other furniture in cleaning the clutter areas <b>2906</b> in some instances, thereby disrupting the cleaning mission with respect to open areas <b>2904</b>.

0221The open area cleaning may include determining advantageous locations for ending the cleaning operation in a particular region or portion thereof. For example, a cleaning operation for a first space that ends proximate a planned starting point for a next region to be cleaned may reduce the unnecessary traversal across a region or portion thereof that has already been cleaned. In some embodiments, the next region to be cleaned may be determined based on its proximity to the ending point of the previous cleaning operation. For example, the room adjacent the closest doorway at the completion of the current cleaning operation may be selected to be the next region and/or portion thereof to be cleaned.

0222In the examples of <figref idref="DRAWINGS">FIGS. 29A through 29D</figref>, the coverage pattern indicates sequential navigation of differently-classified areas, that is, once the open area cleaning operation is performed, the clutter area cleaning operation is performed. For example, <figref idref="DRAWINGS">FIG. 29C</figref> illustrates the remaining uncleaned areas are the clutter areas <b>2906</b> and the cleaning path for cleaning those areas (shown by arrows indicating a random pattern of movement within the clutter areas <b>2906</b>). As illustrated in <figref idref="DRAWINGS">FIG. 29D</figref>, once the open areas and clutter areas are cleaned, a perimeter cleaning operation may be performed along the perimeter <b>2930</b> of the enclosed space.

0223It will be understood that the coverage pattern may not be determined and/or modified upon initial detection of the clutter areas <b>2906</b>, as learning and generation of the map <b>2902</b> may require that at least one navigation or cleaning operation has been performed. That is, in computing the coverage patterns as described herein, clutter detection may be based on several navigations of the surface using persistent map data. For example, detected obstacles may be stored in a data store, such as the data store <b>2106</b> in the cloud <b>2104</b>, after each navigation. By comparing obstacles and/or structures from several maps generated over time, probabilities may be computed for each region of the surface, denoting the degree or density of static versus dynamic obstacles (such as walls versus clutter) therein, which may be used to classify the regions as clutter areas <b>2906</b> or open areas <b>2904</b>. This information can also be displayed via a user interface of the user device <b>202</b>, so that the classification of the regions as clutter areas <b>2906</b> or open areas <b>2904</b> is observable by a user.

0224More particularly, after multiple navigations of the surface, the robot <b>100</b> may be better informed about regions in which the locations of obstacles may change over time, and such regions can therefore be classified as clutter areas <b>2906</b>. Other regions may be determined as being static, such as either static free or static occupied, and may be classified as open areas <b>2904</b>. Based on this information, a coverage pattern or cleaning strategy can be computed. For example, the coverage pattern may specify cleaning of the open areas <b>2904</b> more efficiently in a ranking pattern (as shown in <figref idref="DRAWINGS">FIG. 29B</figref>), and cleaning the clutter areas <b>2906</b> in an edge cleaning pattern (as shown in <figref idref="DRAWINGS">FIG. 29C</figref>). These example cleaning behaviors shown in <figref idref="DRAWINGS">FIGS. 29B and 29C</figref> in response to the determined coverage pattern can be observed and distinguished. In a particular example, the robot <b>100</b> may first clean efficiently all static free areas <b>2904</b> by ranking without entering clutter areas <b>2906</b>. After completing cleaning of the open areas <b>2904</b>, the robot <b>100</b> may perform the edge cleaning behavior by traversing all cluttered areas <b>2906</b> sequentially. Also, in some embodiments, the robot <b>100</b> may repeatedly execute the same pattern during subsequent navigations of the surface. However, when the environment is changed (for example, when clutter is removed from one or more of the clutter areas <b>2906</b>), the computed coverage pattern may be modified by re-classifying one or more of the clutter areas <b>2906</b> as an open area <b>2904</b>, resulting in a change in the behavior of the robot <b>100</b> in navigation of the same surface after the re-classification.

0225For example, in <figref idref="DRAWINGS">FIG. 29A</figref>, elongated open area <b>2904</b> may represent the entrance corridor of a house. The majority of such a corridor may be free, so that people can walk when entering or leaving the house. The lowermost clutter area <b>2906</b> to the left of the elongated open area <b>2904</b> may be a place where people leave their shoes when entering the house. In this area <b>2906</b>, the robot <b>100</b> may therefore frequently bump into these obstacles (shoes) or otherwise detect the obstacles via a non-contact sensor (for example, a PixArt sensor). In addition, the arrangement of the shoes will typically differ each time the robot <b>100</b> cleans the area <b>2906</b>, because the placement of the shoes may frequently change over time. Consequently, the robot <b>100</b> can learn and classify this particular area <b>2906</b> as dynamically changing (and thus, cluttered), whereas portions of the elongated corridor <b>2904</b> (such as the middle and lower areas of the corridor) can be learned and classified as static free (such as traversable, open floor space).

0226Accordingly, a coverage pattern may be determined and provided to the robot <b>100</b> such that the robot <b>100</b> first cleans the elongated corridor <b>2904</b> by efficient ranking without entering the shoe-cluttered area <b>2906</b>. After executing the cleaning of the elongated corridor <b>2904</b>, the robot <b>100</b> will sequentially clean the clutter area <b>2906</b> (to the left of the corridor) by edge cleaning, according to the coverage pattern. However, if people subsequently begin placing shoes near the right wall of the elongated corridor <b>2904</b> (and no longer in the area <b>2906</b> to the left), the area classifications and coverage pattern may be modified responsive to detection of this change after subsequent navigations, and the robot <b>100</b> may observably alter its cleaning behavior after a few cleaning runs. More particularly, the coverage pattern may be modified such that, in response to the coverage pattern, the robot <b>100</b> may first clean the now-open area (formerly classified as clutter area <b>2906</b>) on the left side of the elongated corridor <b>2904</b>, and then execute edge cleaning on the shoe-cluttered area along the right wall of the elongated corridor <b>2904</b>. More generally, there may be multiple indications of clutter detection during navigation of the robot <b>100</b>. These may include, but are not limited to, display and marking of the clutter areas <b>2906</b> and open areas <b>2904</b> via the user interface of a user device <b>202</b>, different behavior of the robot <b>100</b> in navigating the open areas <b>2904</b> and clutter areas <b>2906</b>, and repeated execution of the different behaviors in the same order or sequence (as well as changes to the repeated execution responsive to changes in the environment).

0227In some embodiments, the computing of the coverage pattern and the cleaning of the identified areas of the surface by the mobile robot <b>100</b> may be responsive to receiving a selection of regions/sub-regions to be cleaned, a cleaning sequence/priority, cleaning patterns, and/or a level of cleaning from a user device, for example, via a user interface <b>410</b> of the wireless communication device <b>400</b> or other user device <b>202</b>. <figref idref="DRAWINGS">FIGS. 30A through 30G</figref> are graphical representations of segmentation map views <b>3002</b> that may be displayed via a user interface of a user device illustrating operations for user selection of respective area-specific cleaning operations and/or preferences to define the coverage pattern according to some embodiments of the present invention. For example, the segmentation maps <b>3002</b> of <figref idref="DRAWINGS">FIGS. 30A-30G</figref> may visually highlight the differently-classified regions to allow user selection of initial cleaning of the open areas <b>2904</b> and the sequence in which open areas <b>2904</b><i>a, </i><b>2904</b><i>b, </i><b>2904</b><i>c, </i>and <b>2904</b><i>d </i>are to be cleaned (illustrated in <figref idref="DRAWINGS">FIGS. 30A-30D</figref> by numerals <b>1</b>-<b>4</b>), subsequent cleaning of the clutter areas <b>2906</b> and the sequence in which clutter areas <b>2906</b><i>a </i>and <b>2906</b><i>b </i>are to be cleaned (illustrated in <figref idref="DRAWINGS">FIGS. 30E-30F</figref> by numerals <b>5</b>-<b>6</b>), and completion of cleaning by performing perimeter coverage of the perimeter <b>2930</b> of <figref idref="DRAWINGS">FIG. 29D</figref> (illustrated in <figref idref="DRAWINGS">FIG. 30G</figref> by numeral <b>7</b>). In performing user-defined area cleaning operations based on the coverage pattern, the mobile robot <b>100</b> may treat areas selected for subsequent cleaning as a keep-out zone, until cleaning of the current area is completed. In some embodiments, the mobile robot <b>100</b> may traverse the differently-classified sub-regions <b>2904</b><i>a</i>-<i>d </i>in substantially linear rank directions corresponding to a longer dimension of the regions or sub-regions, and may further traverse the sub-regions <b>2906</b> in arcs or other non-linear patterns so as to navigate systematically around corners.

0228Such user-defined area cleaning operations may also allow for user selection of cleaning patterns (including the direction of travel) and/or cleaning level (including the number of passes) performed by the mobile robot <b>100</b> in executing the cleaning operations, thereby allowing for user-assigned cleaning levels. <figref idref="DRAWINGS">FIGS. 31A through 31C</figref> are graphical representations of segmentation map <b>3102</b> that may be displayed via a user interface of a user device for user selection of cleaning patterns for cleaning of the open area <b>2904</b><i>b </i>shown in <figref idref="DRAWINGS">FIG. 30B</figref>. The segmentation maps <b>3102</b> displayed via the user interface may thus allow for selection of a north-south rank direction <b>2910</b><i>a, </i>a west-east rank direction <b>2910</b><i>b, </i>or a criss-cross rank direction <b>2910</b><i>c </i>in the non-clutter region <b>2904</b><i>b. </i>A level of thoroughness of the cleaning (for example, a single cleaning pass or multiple cleaning passes in each direction of travel) may also be selected for each of the directions of travel <b>2910</b><i>a</i>-<b>2910</b><i>c. </i>That is, in accordance with embodiments described herein, a coverage pattern may be determined not only based on the differently-classified sub-regions of a surface, but also based on user-specified parameters, including direction of travel, levels of cleaning, and/or order or priority of cleaning. User inputs labeling one or more regions of the segmentation map may further facilitate and/or optimize cleaning operations performed by a mobile robot as described herein.

0229Determining coverage patterns based on classification of clutter and non-clutter areas and/or user-defined area cleaning operations described herein with reference to <figref idref="DRAWINGS">FIGS. 29A-29D</figref> may also be based on flooring type (for example, wet cleaning of hard floors, dry cleaning of carpeted floors). <figref idref="DRAWINGS">FIG. 32</figref> is a graphical representation of a segmentation map view <b>3202</b> that may be displayed via a user interface of a user device illustrating detection and/or user selection of floor types for area-specific cleaning operations according to some embodiments of the present invention. As shown in <figref idref="DRAWINGS">FIG. 32</figref>, the graphical representation <b>3202</b> may indicate flooring types including hard floor <b>3204</b> and soft floor or carpet <b>3206</b>, in combination with the detected clutter areas <b>2906</b> and open areas <b>2904</b>. That is, the clutter areas <b>2906</b> and open areas <b>2904</b> may be further identified as hard floor open areas <b>3205</b>, hard floor clutter areas <b>3207</b>, soft floor open areas <b>3209</b>, and soft floor clutter areas <b>3211</b>, based on detection or user indication of the corresponding flooring type. Also, areas identified as clutter <b>2906</b> or open <b>2904</b> may include more than one floor type <b>3206</b> or <b>3204</b>, or vice versa. For example, the lowermost open area <b>2904</b> shown in <figref idref="DRAWINGS">FIG. 32</figref> includes both a soft floor/carpet area <b>3206</b> and a hard floor area <b>3204</b>.

0230The flooring types <b>3204</b>, <b>3206</b> may be detected by the robot <b>100</b> during navigation of the surface based on inputs from one or more sensors thereof, and/or based on receiving identification of the floor types corresponding to one or more regions of the segmentation map <b>3202</b> from a user device via a user interface. For example, detection of the flooring types <b>3204</b>, <b>3206</b> may be performed by the robot <b>100</b> as described in U.S. Patent Application Publication No. 2016/0235270 to Santini, filed Feb. 13, 2015, the disclosure of which is incorporated by reference herein. Also, although only hard floor <b>3204</b> and carpeted floor <b>3206</b> types are shown, it will be understood that additional flooring types, such as types of hard floor (for example, wood or concrete) and/or types of carpet (for example, shag or Berber) may also be detected by the robot <b>100</b> and/or specified via the user interface. Also, the type of cleaning operations (for example, wet or dry cleaning) to be performed by the robot <b>100</b> may be determined responsive to detection or identification of the flooring types <b>3204</b>, <b>3206</b>.

0231User-indicated flooring types (as received, for example, via the user interface) may also be aligned or otherwise determined to correspond with the flooring types <b>3204</b>, <b>3206</b> detected by the robot <b>100</b> based on sensor inputs and/or machine learning. For example, via the user interface, a user may draw or otherwise define a boundary <b>3205</b> around a rug <b>3209</b> in lowermost open area <b>3204</b> shown in the map <b>3202</b> to indicate that it corresponds to a carpet area and/or a keep-out zone. The robot <b>100</b> may also detect (for example, using an acoustic sensor) edges of the rug <b>3209</b> on a hard floor area <b>3204</b>; however, it may be determined that the user-defined boundary <b>3205</b> (as displayed on the map <b>3202</b>) does not exactly correspond to the actual boundaries of the rug <b>3209</b> as detected by the robot sensor(s). For example, as shown in <figref idref="DRAWINGS">FIG. 32</figref>, the user defined boundary <b>3205</b> is shifted relative to the boundaries of the rug <b>3209</b>. In response to detecting this discrepancy, the robot <b>100</b> may recognize a margin of error associated with the user-defined boundary <b>3205</b> based on the displayed map <b>3202</b> relative to the detected boundaries of the rug <b>3209</b> (particularly as the map <b>3202</b> may be simplified for display via the user interface), and may verify its detection of the transition between the hard floor area <b>3204</b> and the soft floor area <b>3206</b> responsive to the user-defined boundary <b>3205</b>. In some embodiments, the user-defined boundary <b>3205</b> may be adjusted on the displayed map <b>3202</b> to correspond to the detected boundaries of the rug <b>3209</b>, for example by shifting, re-sizing, or otherwise re-aligning the user defined boundary <b>3205</b> on the map <b>3202</b>.

0232Coverage patterns based on user-defined area cleaning operations as described herein may be further improved by employing room-by-room cleaning, for example, based on room segmentation operations described herein. In some instances, room-by-room cleaning operations may allow for cleaning time reduction of up to <b>50</b>%. Also, cleaning performance may be further improved when based on persistent mapping (for example, by updating static versus dynamic areas of an operating environment over time) in combination with user labeling of areas and/or context-sensitive behaviors. For example, in based on the determined coverage pattern, the robot <b>100</b> may perform edge cleaning responsive to encountering a perimeter or boundary after completing open area cleaning. As another example, after completing clutter area cleaning, the robot <b>100</b> may not traverse a previously-cleaned open area in the same rank direction that was used to clean the open area. The operations to compute the coverage patterns described herein may be performed by the robot <b>100</b>, the robot management server <b>200</b>, and/or the user device <b>202</b>.

0233Clutter detection and classification operations described herein may utilize the occupancy data collected by a robot <b>100</b> in combination with feedback received from a user device in some embodiments. For example, objects (including static objects such as walls and dynamic objects such as clutter) can be detected by the robot <b>100</b> using a classifier including hard-coded or programmed rules. The programmed rules may be based on, for example, object area and eccentricity, object solidity, and/or object similarity to linear structures. However, as noted above, objects defined by pixels that appear to be attached to a detected boundary can be interpreted as being merged into walls. Such inaccurate interpretation of the collected occupancy data may result in generation of inaccurate maps for presentation to the user.

0234As such, in some embodiments, the classifier may be trained based on the occupancy data collected by the robot <b>100</b>, and further fine-tuned based on feedback input received from a plurality of users. In particular, patterns of clutter identified by one user can be stored in a data store (for example, the data store <b>2106</b> of <figref idref="DRAWINGS">FIG. 21</figref>) and utilized by other users and/or robots, for example using convolutional neural net (CNN)-based object detection. <figref idref="DRAWINGS">FIGS. 34A through 34F</figref> illustrate example operations for distinguishing and removing clutter from map data. In particular, an occupancy or grid map <b>3402</b> generated based on data collected by navigation of the mobile robot (as shown in <figref idref="DRAWINGS">FIG. 34A</figref>) may be used to detect and distinguish pixel areas <b>3406</b> representing objects from pixel areas <b>3404</b> representing background areas (as shown in <figref idref="DRAWINGS">FIG. 34B</figref>). More particularly, from the map <b>3402</b> shown in <figref idref="DRAWINGS">FIG. 34A</figref>, a boundary <b>3404</b> shown in <figref idref="DRAWINGS">FIG. 34B</figref> is extracted, and the pixel areas <b>3406</b> representing objects are classified as boundary-detached obstacle pixels <b>3406</b><i>a </i>and boundary-attached obstacle pixels <b>3406</b><i>b </i>in <figref idref="DRAWINGS">FIG. 34C</figref>. The boundary-detached obstacle pixels <b>3406</b><i>a </i>are removed as shown in <figref idref="DRAWINGS">FIG. 34D</figref>, such that boundary-attached obstacle pixels <b>3406</b><i>b </i>remain as shown in <figref idref="DRAWINGS">FIG. 34E</figref>. In some embodiments, the boundary-attached obstacle pixels <b>3406</b><i>b </i>may be identified and classified as clutter (rather than as a wall) by accessing a data store including ground truth information generated at least in part from labeling inputs received from multiple users. The boundary-attached obstacle pixels <b>3406</b><i>b </i>are thereby removed such that only the boundaries <b>3404</b> representing walls remains in the grid map <b>3402</b> as shown in <figref idref="DRAWINGS">FIG. 34F</figref>.

0235A data store, such as the data store <b>2106</b> described above, that may be used in identifying and classifying pixels as clutter may be generated from user inputs labeling particular structures as walls (static structures) or clutter (dynamic structures). CNN-based object detection may be performed remote from the mobile robot <b>100</b>, for example, by the robot management server <b>200</b> and/or other cloud server in some embodiments. The CNN classifier may be pre-trained using stored classification data collected by many robots (for example, based on static maps collected from robot navigation), and fine-tuned using stored data generated from user feedback received from one or more user devices.

0236<figref idref="DRAWINGS">FIGS. 35A through 35D</figref> illustrate examples of patterns <b>3504</b>, <b>3506</b> that may be stored in the data store and used for clutter classification and feature extraction as described herein. More particularly, <figref idref="DRAWINGS">FIGS. 35A-35B</figref> illustrate patterns <b>3506</b> indicating clutter, while <figref idref="DRAWINGS">FIGS. 35C-35D</figref> illustrate patterns <b>3504</b> indicating walls, as received and/or identified based on feedback received from a plurality of user labeling inputs. The user labeling inputs may be received via a user interface of the user device, such as the user device <b>202</b> and/or the wireless electronic device <b>400</b> described herein. Features represented by or corresponding to the patterns shown in <figref idref="DRAWINGS">FIGS. 35A through 35D</figref> may thus be learned and extracted from subsequent occupancy data collected by a robot <b>100</b> using, for example, an autoencoder. The patterns and/or shapes shown in <figref idref="DRAWINGS">FIGS. 35A through 35D</figref> may be further classified based on additional user labeling inputs. For example, particular ones of the patterns <b>3506</b> indicating dynamic structures in <figref idref="DRAWINGS">FIG. 35B</figref> may be distinguished as kitchen appliances rather than furniture, or vice versa, based on user labeling of the corresponding patterns. Likewise, types of furniture (for example, chairs vs. tables) may be distinguished responsive to accessing a data store that maintains an archive of user labeling inputs classifying types and sub-types of clutter, which may be subsequently used by one or more mobile robots in recognizing such patterns in one or more different operating environments. In further embodiments, pattern-based object detection in accordance with <figref idref="DRAWINGS">FIGS. 35A through 35D</figref> may used or combined with image-based data for image classification.

0237<figref idref="DRAWINGS">FIGS. 36A through 36C</figref> are graphical representations illustrating operations for updating persistent map data by segmentation transfer and feature matching between multiple maps according to some embodiments of the present invention. In particular, grid maps <b>3602</b><i>a, </i><b>3602</b><i>b </i>of an operating environment generated from occupancy data collected by a mobile robot <b>100</b> may differ from day-to-day navigation of the operating environment. These differences may arise, for example, from sensor noise and/or differing placement or movement of clutter (including people, animals, and/or furniture) in the operating environment.

0238In order to maintain consistent segmentation in such a dynamic operating environment, embodiments herein may transfer segmentation between maps collected at different times and/or under different conditions. Operations for such segmentation transfer may include feature matching (as shown in <figref idref="DRAWINGS">FIG. 36A</figref>) and outlier removal (as shown in <figref idref="DRAWINGS">FIG. 36B</figref>). Feature matching may include updating a subsequently obtained map <b>3602</b><i>b </i>of the operating environment with one or more features identified from a previous map <b>3602</b><i>a, </i>for example, by matching boundaries and/or correlating shapes that are common to the multiple maps <b>3602</b><i>a, </i><b>3602</b><i>b </i>that were generated from occupancy data detected at different times and/or under different conditions in the same operating environment. The features <b>3606</b> that are present or persistent in multiple maps <b>3602</b><i>a, </i><b>3602</b><i>b </i>may be recognized as corresponding to static features, such as walls or other non-movable structures. Outlier removal may include deleting areas <b>3604</b> (for example, doorways and/or hallways) appearing in some of the maps <b>3602</b><i>a, </i><b>3602</b><i>b </i>but not in others. Feature matching operations can be based both on imaging and non-imaging data collected by the robot, for example, based on bumper hits captured by non-imaging sensors as well as images captured by the camera of the mobile robot.

0239Also, boundaries <b>3609</b> indicating room segmentation can be transferred between maps <b>3602</b><i>a, </i><b>3602</b><i>b </i>that are generated based on occupancy data collected at different times and/or operating conditions (as shown in <figref idref="DRAWINGS">FIG. 36C</figref>). User inputs and/or modifications to a segmentation map displayed via a user interface of a user device may also be transferred in updating maps as described herein. Such maintenance of consistent segmentation <b>3609</b> may allow users to observe the same/similar cleaning behavior or sequence at each operation of the mobile robot, increasing predictability of robot operation. Also, transferring segmentation <b>3609</b> in updating maps as described herein may allow users to maintain previously-defined editing of the segmentation and/or a previously-defined operating sequence for room-by-room or region-based cleaning. As such, a user may observe relatively stable behavior by the mobile robot despite some changes in the operating environment itself.

0240<figref idref="DRAWINGS">FIG. 37</figref> is a graphical representation that may be displayed via a user interface <b>3710</b> of a communication device <b>3700</b> illustrating operation of a mobile robot responsive to user selection of temporal status according to some embodiments of the present invention. As shown in <figref idref="DRAWINGS">FIG. 37</figref>, a map indicating navigation of the operating environment by the mobile robot <b>100</b> may be generated and stored along with temporal data indicating the times at which the robot was at each location shown in the map. Based on this data, a user input element <b>3750</b> (illustrated as a time slider user interface) may be presented via the user interface <b>3710</b> of a user device <b>3700</b>. The user input element <b>3750</b> is configured to receive a user specification of a time for which operating state information for the robot <b>100</b> is desired. In some embodiments, an animation <b>3720</b> illustrating the location and operation of the mobile robot <b>100</b> may be generated and displayed responsive to receiving a user selection of a particular time or times via the user input element <b>3750</b>.

0241More particularly, in the example of <figref idref="DRAWINGS">FIG. 37</figref>, the time-slider bar <b>3750</b> is configured to receive a user input that manipulates slider element <b>3751</b> to respective positions along a length of bar element <b>3755</b>. The bar element <b>3755</b> visually represents a temporal range for which operating state information for the robot <b>100</b> been stored or is otherwise accessible. Markings indicating may be respective times may be provided along the length of the bar element <b>3755</b>. The user interface <b>3710</b> is configured to detect a position of the slider element <b>3751</b> along the length of the bar element <b>3750</b> responsive to manipulation of the slider element or other user input, from which the device <b>3700</b> is configured to determine the time specified by the user input based on correspondence of the position of the slider element <b>3751</b> along the temporal range represented by the bar element <b>3755</b>. The user interface <b>3710</b> is thereby configured to display status indications representing the operating states of the robot <b>100</b> that are associated with temporal data corresponding to the time specified by the user input. An animation <b>3720</b> illustrating the location and operation of the mobile robot <b>100</b> over a selected portion of its run-time may also be displayed responsive to user manipulation of the time-slider bar <b>3750</b>.

0242The animation <b>3720</b> may visually represent actual robot operating status information (based on actual past or present operation of the robot <b>100</b>, as previously logged or currently reported), and/or expected robot operating status information (based on scheduled future operation of the robot <b>100</b>, as specified by a user). The actual and expected/scheduled operating states (and the past, present or future temporal data associated therewith) may be continually logged and updated, and may be stored in a database. For example, in some embodiments, a computing device (such as the robot management node <b>200</b>) may receive data indicating actual operating states (and associated past or present temporal data) and/or expected operating states (and associated future temporal data) of the robot <b>100</b>. The data indicating the operating states and associated temporal data may be requested (or “pulled”) by the computing device, or may be reported (or “pushed”) by the robot <b>100</b> and/or network sources in communication therewith. The animation <b>3720</b> of cleaning progress may also be presented with various granularity. For example, the cleaning progress on a sub-region or room level may be selected and displayed in an enlarged view, rather than as small squares as shown in <figref idref="DRAWINGS">FIG. 37</figref>. Additionally or alternatively, the animation <b>3720</b> may illustrate the trajectory of the robot <b>100</b>, showing a previous- or real-time path of the robot <b>100</b> responsive to user selection. In some embodiments, the time selected via the input element <b>3750</b> is associated with a timestamp in a video feed collected and stored by the mobile robot <b>100</b>, and the user interface <b>3710</b> may display images/video captured at that selected time by a camera mounted to the mobile robot <b>100</b>. More generally, a viewable history <b>3720</b> of the instantaneous operation and operating patterns of the mobile robot <b>100</b> throughout the operating environment may be displayed at a user-specified time or over a user-specified time range.

0243In the above-description of various embodiments of the present disclosure, aspects of the present disclosure may be illustrated and described herein in any of a number of patentable classes or contexts including any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof. Accordingly, aspects of the present disclosure may be implemented in entirely hardware, entirely software (including firmware, resident software, micro-code, etc.) or combining software and hardware implementation that may all generally be referred to herein as a “circuit,” “module,” “component,” or “system.” Furthermore, aspects of the present disclosure may take the form of a computer program product comprising one or more computer readable media having computer readable program code embodied thereon.

0244Any combination of one or more computer readable media may be used. The computer readable media may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an appropriate optical fiber with a repeater, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

0245A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable signal medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

0246Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages. The program code may execute entirely on a user device, a mobile robot, or a remote server described herein, or partly on one or more of each. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider) or in a cloud computing environment or offered as a service such as a Software as a Service (SaaS).

0247Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable instruction execution apparatus, create a mechanism for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

0248These computer program instructions may also be stored in a computer readable medium that when executed can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions when stored in the computer readable medium produce an article of manufacture including instructions which when executed, cause a computer to implement the function/act specified in the flowchart and/or block diagram block or blocks. The computer program instructions may also be loaded onto a computer, other programmable instruction execution apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatuses or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

0249It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

0250The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various aspects of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

0251The foregoing is illustrative of the present invention and is not to be construed as limiting thereof. Although a few example embodiments of this invention have been described, those skilled in the art will readily appreciate that many modifications are possible in the example embodiments without materially departing from the novel teachings and advantages of this invention. Accordingly, all such modifications are intended to be included within the scope of this invention. Therefore, it is to be understood that the foregoing is illustrative of the present invention and is not to be construed as limited to the specific embodiments disclosed, and that modifications to the disclosed embodiments, as well as other embodiments, are intended to be included within the scope of the invention.