Object detection using location data and scale space representations of image data

0001This disclosure is generally related to object detection. More specifically, this disclosure relates to object detection in response to image data and location data.

I. DESCRIPTION OF RELATED ART

0002A device may perform object detection operations using an image-based technique. For example, a camera may be mounted on a vehicle or incorporated in an electronic device, and the camera may supply image data to a processor. The processor may execute a computer vision application to analyze (or “scan”) the image data to detect an object, such as an obstacle (e.g., a pedestrian, a tree, livestock, game, and/or another object) that is within or near the field of travel of the vehicle or mobile device.

0003In some circumstances, image-based object detection may be slow and ineffective. For example, an image “search” to recognize objects in an image may involve multiple time-consuming operations, such as image segmenting and windowing operations. Further, image-based object detection may be associated with a large range of uncertainty (or a large margin of error), which may cause detected object locations to be unreliable. These issues can frustrate the intended purpose for the object detection. As an example, in vehicular applications, image-based object detection may occur too slowly to enable the vehicle to slow, stop or to otherwise change course in time to avoid an obstacle.

0004Some devices may use location-based information, such as radar information, to detect objects. However, location-based information may be subject to a large range of uncertainty (e.g., such devices may have a large margin of error).

II. SUMMARY

0005Object detection may be performed using location data and scale space representations of image data. Each of the scale space representations may be a different resolution version of the image data. To illustrate, a device, for example, a vehicle or electronic device, may include an image sensor (e.g., a camera) configured to generate the image data. The device may also include or receive location data associated with an object to be detected and/or indicative of one or more areas in a scene where the object may be present. For example, location data may be received from a location sensor (e.g., a radar device, a global positioning system (GPS) device, etc.) that is configured to generate the location data. The location sensor can be coupled to the image sensing device or separate from the image sensing device. In another example, location data may correspond to one or more areas of a three-dimensional (3D) map indicative of where an object of interest (e.g., an object to be detected) may be present, such as an area of a scene or image that is above a road or ground surface. In some instances, more than one area of a scene may be identified by a 3D map and received as location data. For example, it may be desirable to detect objects on both left and right sides of a driving lane. In such devices, instead of searching all of the image data to detect an object (e.g., a pedestrian, a tree, livestock, game, and/or another object), a device (e.g., a processor) may search an overlap, intersection, or convergence of the location data and a scale space representation of the image data to detect the object. The device may be configured to search one or more scale space representations to detect the object, and different scale space representations may be searched to detect different objects. Searching the overlap or intersection of the location data and one or more scale space representations of the image data may be less computationally complex than other techniques (e.g., a “brute force” or complete search of all image data) and thus more efficient than other object detection techniques that do not consider location data.

0006As an illustrative, non-limiting example, a vehicle may perform pedestrian detection. To detect a pedestrian, the vehicle may utilize model-based object detection, where the model assumes that the height of objects of interest in image data provided by the camera will be less than or equal to a particular number of pixels (e.g., less than or equal to 100 pixels in height). When a pedestrian is close to the vehicle, the height of the pedestrian may be over 100 pixels in height, and as a result, the model may not be able to successfully detect the pedestrian. To detect the pedestrian, model-based object detection may be re-executed on a downsampled scale space representation of the image data. For example, if the height of the pedestrian is 180 pixels in the original (e.g., full-resolution) image, then the height of the pedestrian will be 90 pixels in a scale space representation that is downsampled by a factor of 2, and the model-based object detection may successfully detect the pedestrian in the downsampled image. Thus, to successfully detect objects (e.g., pedestrians) at varying distances, the vehicle may perform object detection on multiple scale space representations of the image data captured by the camera (e.g., a “high” or “full” resolution representation, a “medium” resolution representation, a “low” resolution representation, etc.). It should be noted that the present disclosure is not limited to object detection for vehicles. In various embodiments, object detection may be performed at and/or based on images captured by an image sensor of an electronic device, a mobile device, a personal mounted camera, a head mounted display, or an aerial vehicle, as illustrative non-limiting examples.

0007Iteratively searching each of multiple scale space representations in their entirety may be time-consuming. In accordance with the described techniques, an object detector may skip searching scale space representations that do not intersect or overlap with a probable location of an object (e.g., a pedestrian). Further, when a scale space representation overlaps the probable location of the object, the search may be restricted to an area of interest corresponding to the overlap. The probable location of the object may be provided by a location sensor that is part of the vehicle or that is separate from the vehicle. Additionally, the probable location of the object may be based at least in part on one or more areas of a 3D map where the object may be present. It is to be understood that as used herein, detecting an object can include detecting a representation of the object in image data (or in a scale space representation of the image data), detecting the actual physical object (e.g., based on ultrasound, radar, etc.), or both.

0008In a particular aspect, an apparatus includes an object detector configured to receive location data and image data of a scene from an image sensor. The scene is viewed from a vehicle and includes an object. The image data is associated with (e.g., can be used to generate) multiple scale space representations of the scene. The object detector is configured to detect the object responsive to the location data and a first scale space representation of the multiple scale space representations.

0009In another particular aspect, a method includes receiving from an image sensor, at a processor, image data of a scene viewed from an apparatus and including an object. The image data is associated with multiple scale space representations of the scene. The method also includes receiving, at the processor, location data associated with the object or indicative of an area where the object may be present. The method further includes identifying, at the processor, a search area of a first scale space representation of the multiple scale space representations based on an overlap between the first scale space representation and the location data. The method includes performing, at the processor, object detection in the search area of the first scale space representation.

0010In another particular aspect, an apparatus includes means for receiving image data of a scene viewed from the apparatus and including an object, where the image data is associated with multiple scale space representations of the scene. The apparatus further includes means for processing. The means for processing is configured to identify a search area of a first scale space representation of the multiple scale space representations based on an overlap between the first scale space representation and location data associated with the object or indicative of an area where the object may be present. The means for processing is also configured to perform object detection in the search area of the first scale space representation.

0011In another particular aspect, a computer-readable storage device stores instructions executable by a processor to cause the processor to receive, from an image sensor, image data of a scene viewed from an apparatus and including an object. The image data is associated with multiple scale space representations of the scene. The instructions are also executable to receive location data associated with the object or indicative of an area where the object may be present, and to identify a search area of a first scale space representation of the multiple scale space representations based on an overlap between the first scale space representation and the location data. The instructions are further executable to perform object detection in the search area of the first scale space representation.

0012One particular advantage provided by at least one of the disclosed embodiments is that a time of object detection that is performed on image data may be decreased by using location data that is associated with the object or that indicates an area where the object may be present. For example, object detection may not be performed on scale space representation(s) that do not intersect or overlap the location data, and object detection on scale space representation(s) that intersect or overlap the location data may be limited to the overlapping or intersecting area(s) of interest. Other aspects, advantages, and features of the present disclosure will become apparent after review of the entire application, including the following sections: Brief Description of the Drawings, Detailed Description, and the Claims.

III. BRIEF DESCRIPTION OF THE DRAWINGS

0013<figref idref="DRAWINGS">FIG. 1</figref> is a diagram that depicts an illustrative system that is operable to perform object detection using location data and scale space representations of image data;

0014<figref idref="DRAWINGS">FIGS. 2A, 2B, 2C, and 2D</figref> are diagrams that depict illustrative examples of operation of the system of <figref idref="DRAWINGS">FIG. 1</figref>;

0015<figref idref="DRAWINGS">FIG. 3</figref> is a diagram that depicts another illustrative example of a system that is operable to perform object detection using location data and scale space representations of image data;

0016<figref idref="DRAWINGS">FIG. 4</figref> is a diagram that depicts another illustrative example of a system that is operable to perform object detection using location data and scale space representations of image data;

0017<figref idref="DRAWINGS">FIG. 5A</figref> depicts an illustrative example of using three-dimensional (3D) map data to perform object detection;

0018<figref idref="DRAWINGS">FIG. 5B</figref> depicts an illustrative example of a mask that may be used during object detection;

0019<figref idref="DRAWINGS">FIG. 6</figref> is a flowchart that illustrates an illustrative example of a method of identifying area(s) of interest based on 3D map data;

0020<figref idref="DRAWINGS">FIG. 7</figref> is a flowchart that depicts an illustrative example of a method of object detection using location data and scale space representations of image data; and

0021<figref idref="DRAWINGS">FIG. 8</figref> is a block diagram of an illustrative example of an electronic device.

IV. DETAILED DESCRIPTION

0022<figref idref="DRAWINGS">FIG. 1</figref> depicts a particular illustrative embodiment of a system that is configured to perform object detection. In the example of <figref idref="DRAWINGS">FIG. 1</figref>, the object detection is performed at a vehicle <b>102</b>. However, it should be noted that the present disclosure is not to be limited as such. The object detection devices, methods, systems, etc. disclosed herein can be implemented in alternative environments to detect objects in traffic, in a field of view, etc. For example, one or more functions described herein may be implemented in electronic devices, mobile devices, gaming consoles, automotive system consoles (e.g., ADAS), wearable devices (e.g., personal mounted cameras), head mounted displays, etc. Additional examples include, but are not limited to, robots or robotic devices, unmanned aerial vehicles (UAVs), and drones. In the example of <figref idref="DRAWINGS">FIG. 1</figref>, the vehicle <b>102</b> may be a motor vehicle (e.g., a car, a truck, a motorcycle, a bus, or a train), a watercraft (e.g., a ship or a boat), an aircraft (e.g., an airplane or a helicopter), a spacecraft (e.g., a space shuttle), a bicycle, or another vehicle. The vehicle <b>102</b> may be a wheeled vehicle, a tracked vehicle, a railed vehicle, an airborne vehicle, or a skied vehicle, as illustrative examples. In some cases, the vehicle <b>102</b> may be operated by one or more drivers. For example, the vehicle <b>102</b> may include an advanced driving assistance system (ADAS) configured to assist a driver of the vehicle <b>102</b>. In other cases, the vehicle <b>102</b> may be a computer-controlled vehicle. Furthermore, although the objection detection in the example system of <figref idref="DRAWINGS">FIG. 1</figref> is performed at the vehicle <b>102</b>, it should be understood that in other examples the object detection disclosed herein can be performed in the “cloud” or outside of the vehicle <b>102</b>. For example, a vehicle or other electronic device could provide location data and/or image data to another device to perform the object detection.

0023The vehicle <b>102</b> (e.g., an ADAS of the vehicle <b>102</b>) may include one or more image sensors, such as an illustrative image sensor <b>104</b>. The image sensor <b>104</b> may include a camera, such as a charge-coupled device (CCD)-based camera and/or a complementary metal-oxide-semiconductor (CMOS)-based camera. In alternative embodiments, the image sensor <b>104</b> may include a different type of sensor (e.g., infrared).

0024In the example of <figref idref="DRAWINGS">FIG. 1</figref>, the vehicle <b>102</b> further includes one or more location sensors, such as an illustrative location sensor <b>106</b>. The location sensor <b>106</b> may include a radar device, a light detection and ranging (lidar) device, a global positioning system (GPS) device, an ultrasound device, and/or a communication device, such as a dedicated short-range communication (DSRC) device used in a vehicular communication network, as illustrative examples.

0025In the example of <figref idref="DRAWINGS">FIG. 1</figref>, a triangular field of view of the image sensor <b>104</b> is shown. It should be noted that the field of view of the image sensor <b>104</b> may be determined in various ways. As an illustrative non-limiting example, the image sensor <b>104</b> (or an apparatus including the image sensor <b>104</b>) may include a GPS transceiver, and the field of view may be determined based on a temporal difference between two GPS locations of the image sensor <b>104</b> (or apparatus). A difference between the two GPS locations may correspond to a center line of the triangular field of view or to a direction of travel of the image sensor <b>104</b>. As another illustrative non-limiting example, the direction of the image sensor <b>104</b> may be determined based on a motion sensor (e.g., accelerometer) that is coupled to the image sensor <b>104</b> or included in an apparatus that includes the image sensor <b>104</b>, such as a vehicle, computing device, or other apparatus. Thus, the field of view and direction of the image sensor <b>104</b> may be determined even if not known a priori.

0026The vehicle <b>102</b> may further include a processor <b>108</b> and a memory <b>110</b>. The memory <b>110</b> may store instructions and data accessible by the processor <b>108</b>. The processor <b>108</b> may include a central processor unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an electronic control unit (ECU), another processing device, or a combination thereof. The processor <b>108</b> may include an object detector <b>128</b>, such as a location data-based and scale space-based object detector. The object detector <b>128</b> may correspond to hardware components of the vehicle <b>102</b>, software (e.g., instructions) executable by the processor <b>108</b>, or a combination thereof.

0027During operation, the location sensor <b>106</b> may provide location data <b>116</b> to the processor <b>108</b>. In some implementations, the location data <b>116</b> may include radar data, lidar data, GPS data, etc. associated with one or more objects, such as an illustrative object <b>112</b>. The object <b>112</b> may be stationary or may be in motion. For example, the object <b>112</b> may correspond to a pedestrian, another vehicle, a traffic signal, a roadway obstacle (e.g., a fallen traffic signal, a tree limb, or debris), livestock (e.g., cattle, bison, horses, sheep, or goats), game (e.g., elk, moose, bear, or deer), or a roadside object (e.g., a sign, an advertising billboard, or a road side unit (RSU)), as illustrative non-limiting examples. The object <b>112</b> may be proximate to the vehicle <b>102</b> or disposed at a significant distance from the vehicle (e.g., not proximate to the vehicle). For example, the object <b>112</b> may be within a particular range of the vehicle <b>102</b>, within a field or direction of travel of the vehicle <b>102</b>, within a field of view of the image sensor <b>104</b>, etc. In some implementations, the location sensor <b>106</b> includes one or more sensors configured to scan the field of travel of the vehicle <b>102</b> for objects, such as the object <b>112</b>. For example, the location sensor <b>106</b> may include a radar device, an ultrasound device, and/or a lidar device configured to generate the location data <b>116</b> using signals reflected from the object <b>112</b>. In some implementations, the location sensor <b>106</b> is configured to receive location data associated with one or more objects. For example, the location sensor <b>106</b> may include dedicated short-range communication device, an RFID device, a personal network device, or another communication device.

0028The image sensor <b>104</b> may generate image data <b>114</b> (e.g., by capturing one or more images) of a scene that includes the object <b>112</b>. The image sensor <b>104</b> may provide the image data <b>114</b> to the processor <b>108</b>. In some implementations, the image sensor <b>104</b> may generate the image data <b>114</b> in response to a command from the location sensor <b>106</b>. In a particular illustrative example, if the location sensor <b>106</b> detects the object <b>112</b> (e.g., using a radar, ultrasound, or lidar technique, or by receiving GPS and/or DSRC information from another device), the location sensor <b>106</b> may assert a control signal at a bus that connects the image sensor <b>104</b> and the location sensor <b>106</b>. Alternatively, or in addition, operation of the image sensor <b>104</b> may be controlled by the processor <b>108</b>. For example, the processor <b>108</b> may cause the image sensor <b>104</b> to generate the image data <b>114</b> in response to receiving the location data <b>116</b> from the location sensor <b>106</b>. In other cases, the image sensor <b>104</b> may operate independently of the location sensor <b>106</b>. For example, the image sensor <b>104</b> may constantly, periodically, or occasionally capture images and may load image data of the images (e.g., the image data <b>114</b>) to a buffer, cache, or other memory (e.g., the memory <b>110</b>). In this example, in response to receiving the location data <b>116</b> from the location sensor <b>106</b>, the processor <b>108</b> may retrieve the image data <b>114</b> from the buffer, cache, or other memory. To illustrate, the memory <b>110</b>, or a portion thereof, may function as a circular buffer that is configured to store image data received from the image sensor <b>104</b>.

0029The image data <b>114</b> may be associated with multiple scale space representations of the scene, such as a first scale space representation <b>120</b>, a second scale space representation <b>122</b>, and a third scale space representation <b>124</b>, for example. Each of the scale space representations <b>120</b>, <b>122</b>, and <b>124</b> may be associated with a respective image resolution. For example, the first scale space representation <b>120</b> may be associated with a first image resolution, the second scale space representation <b>122</b> may be associated with a second image resolution that is less than the first image resolution, and the third scale space representation <b>124</b> may be associated with a third image resolution that is less than the second image resolution. In the example of <figref idref="DRAWINGS">FIG. 1</figref>, the first scale space representation <b>120</b> may correspond to the scene projected onto a “high,” “full,” and/or “complete” resolution image plane represented in the field of view of the image sensor <b>104</b> by a line segment <b>111</b>. The second scale space representation <b>122</b> may correspond to the scene projected onto a “medium” resolution image plane represented by a line segment <b>113</b> and the third scale space representation <b>124</b> may correspond to the scene projected onto a “low” resolution image plane represented by line segment <b>115</b>. Thus, as shown in <figref idref="DRAWINGS">FIG. 1</figref>, the line segments <b>111</b>, <b>113</b>, and <b>115</b> (and the corresponding respective “high,” “medium,” and “low” resolution image planes) are different distances from the image sensor <b>104</b>.

0030Although <figref idref="DRAWINGS">FIG. 1</figref> illustrates three scale space representations, it should be appreciated that the disclosure is also applicable to different numbers of scale space representations (e.g., two scale space representations, four scale space representations, or another number of scale space representations). In an illustrative example, the first scale space representation <b>120</b> corresponds to the image data <b>114</b> at a full or complete resolution of the image sensor <b>104</b>, the second scale space representation <b>122</b> corresponds to the image data <b>114</b> downsampled by a first factor (e.g., 2), and the third scale space representation <b>124</b> corresponds to the image data <b>114</b> downsampled by a second factor (e.g., 5) that is larger than the first factor. The processor <b>108</b> may be configured to downsample a higher resolution scale space representation to generate one or more lower resolution scale space representations using filtering, decimation, subsampling, interpolation, and/or other image processing techniques. For example, the processor <b>108</b> may downsample the first scale space representation <b>120</b> to generate the second scale space representation <b>122</b> and may downsample the second scale space representation <b>122</b> to generate the third scale space representation <b>124</b>. In another illustrative implementation, the vehicle <b>102</b> includes multiple image sensors, where each image sensor generates a different resolution image or scale space representation of a scene.

0031The object detector <b>128</b> may be responsive to the location data <b>116</b> and one or more of the scale space representations <b>120</b>-<b>124</b>. For example, the object detector <b>128</b> may select a scale space representation based on the location data <b>116</b> and may perform object detection on the selected scale space representation to detect the object <b>112</b>. To illustrate, the object detector <b>128</b> may use the location data <b>116</b> to determine an estimated region <b>118</b> in which the object <b>112</b> is located. The estimated region <b>118</b> may correspond to probable location(s) of the object <b>112</b> in view of a margin of error associated with the location data <b>116</b> (e.g., due to noise, sensor delay, speed/direction of motion of the object <b>112</b> and/or the vehicle <b>102</b>, etc.). The object detector <b>128</b> may determine whether any of the image planes corresponding to the scale space representations <b>120</b>-<b>124</b> overlaps or intersects the estimated region <b>118</b>. In the example of <figref idref="DRAWINGS">FIG. 1</figref>, the first scale space representation <b>120</b> (corresponding to the line segment <b>111</b>) and the third scale space representation <b>124</b> (corresponding to the line segment <b>115</b>) do not overlap the estimated region <b>118</b>. In response to detecting the lack of overlap, the object detector <b>128</b> may skip performing object detection on the first scale space representation <b>120</b> and the third scale space representation <b>124</b> (e.g., to more quickly perform object detection and/or save computational resources).

0032In the example of <figref idref="DRAWINGS">FIG. 1</figref>, the object detector <b>128</b> may determine that the second scale space representation <b>122</b> (corresponding to the line segment <b>113</b>) overlaps the estimated region <b>118</b>. For example, the object detector <b>128</b> may determine that the second scale space representation <b>122</b> overlaps the estimated region <b>118</b> based on the distance of the resolution image plane of the second scale space <b>122</b> and the received location data <b>116</b>. In response, the object detector <b>128</b> may further process the second scale space representation <b>122</b>. For example, the object detector <b>128</b> may identify a search area within the second scale space representation <b>122</b>, where the search area corresponds to an overlap, intersection, or convergence between a set of locations or area associated with the scale space representation <b>122</b> and a set of locations or area associated with the location data <b>116</b>. To illustrate, <figref idref="DRAWINGS">FIG. 1</figref> shows a top-down view in which the image sensor <b>104</b> has a horizontal field of view bound by field of view lines <b>117</b> and <b>119</b>. Thus, the horizontal boundaries of the second scale space representation <b>122</b> are denoted in <figref idref="DRAWINGS">FIG. 1</figref> by points L<b>0</b> and L<b>1</b>, and the overlap between the estimated region <b>118</b> and the second scale space representation <b>122</b> is represented by a line segment between points C<b>0</b> and C<b>1</b> (which may alternatively be denoted as “line segment C<b>0</b>C<b>1</b>”). The object detector <b>128</b> may perform computer vision operations (e.g., image segmenting operations) in the search area between C<b>0</b> and C<b>1</b>, instead of in the larger area between L<b>0</b> and L<b>1</b>, to “pinpoint” the location of the object <b>112</b>. For example, the object detector <b>128</b> may determine a pedestrian location or recognize the object <b>112</b> (e.g., to recognize text on a sign). Searching the area between C<b>0</b> and C<b>1</b> may include searching parts of the image that extend beyond C<b>0</b> and C<b>1</b>. For example, C<b>0</b> and C<b>1</b> may define center points that are used by different object detection search windows. If a search window to be used by the object detector <b>128</b> to identify the object <b>112</b> has a width of W, the search area “between” C<b>0</b> and C<b>1</b> may range from C<b>0</b>-0.5 W (when the search window is centered on C<b>0</b>) to C<b>1</b>+0.5 W (when the search window is centered on C<b>1</b>). A search area may similarly extend in other dimensions (e.g., in the vertical dimension beyond a height of a search window). The search window may correspond to an object model, bounding box, etc. Thus, although a search area may be described herein in terms of a line segment or set of points, it is to be understood that the actual area of the image being searched may extend beyond the line segment or set of points.

0033Although <figref idref="DRAWINGS">FIG. 1</figref> illustrates a two-dimensional (2D) view, the object detection operations described herein can be applied in three-dimensional (3D) scenarios. To illustrate, the points C<b>0</b>, C<b>1</b>, L<b>0</b>, and L<b>1</b> may all be in 3D space and may have (x, y, z) coordinates. In <figref idref="DRAWINGS">FIG. 1</figref>, object detection and reduction is performed horizontally, along the x-axis in the x-y plane. Alternatively, or in addition (e.g., in parallel with the horizontal operations), object detection and reduction may be performed vertically, along the z-axis in the x-z plane, by selecting an appropriate vertical scale space representation. It will be appreciated that by performing operations in both horizontal and vertical scale spaces, the location of the object <b>112</b> may be more accurately determined than when only one direction is used.

0034In other examples, the object detector <b>128</b> may determine that multiple scale space representations overlap the estimated region <b>118</b> associated with the location data <b>116</b>. To illustrate, in <figref idref="DRAWINGS">FIG. 1</figref>, if the third scale space representation <b>124</b> corresponds to a line segment <b>115</b>′ instead of the line segment <b>115</b>, then the object detector <b>128</b> would determine that both the second scale space representation <b>122</b> and the third scale space representation <b>124</b> overlap the estimated region <b>118</b>. In such a scenario, the object detector <b>128</b> may select which of the overlapping scale space representations to perform object detection on. To illustrate, a smaller (e.g., lower resolution) scale space representation may be preferable for object detection in some circumstances, such as when a pedestrian is very close and appears to be very tall due to scale. In this situation, the lower resolution scale space representation may be searched instead of a higher resolution scale space representation. Alternatively, if the higher resolution scale space representation was searched and the object was not found (e.g., because the pedestrian was “too tall” due to scale), the lower resolution scale space representation may also be searched. As another example, when the object being detected is relatively small or has an intricate shape, a larger (e.g., higher resolution) scale space representation may be preferred.

0035An example process performed by the object detector <b>128</b> is described for illustration purposes with reference to <figref idref="DRAWINGS">FIGS. 2A, 2B, 2C and 2D</figref>. The process may include assigning the image sensor <b>104</b> a particular location, such as the point (x0, y0) in a coordinate space. The measured or reported position of the object <b>112</b> may be assigned a position (x1, y1), which may be at the center of the estimated region <b>118</b>. It is to be understood that the image sensor <b>104</b> and the center of the estimated region <b>118</b> may have coordinates (x0, y0, z0) and (x1, y1, z1), respectively, in 3D space, though only the x and y coordinates are described for ease of explanation.

0036The line segment L<b>0</b>L<b>1</b> may be associated with a particular distance (d) from the image sensor <b>104</b> (e.g., based on camera geometry, resolution, and search area size). Based on the value of d and the position (x0, y0) of the image sensor <b>104</b>, the line segment L<b>0</b>L<b>1</b> may be mapped, fitted, or transposed to a location that shares the same coordinate system as the points (x0, y0) and (x1, y1). The points of intersection between the field of view lines <b>117</b>, <b>119</b> and the estimated region <b>118</b> (i.e., the points C<b>0</b> and C<b>1</b>) may be determined. To enable calculation of distances between locations in image data, the image data may be “fused” with the coordinate system of the location data. In some embodiments, a projection of the 3D world space of the image data may be performed offline based on a known location of a device (e.g., vehicle, mobile device, etc.), a field of view, and a pose of the image sensor <b>104</b>. The projection may then be fused with the location data to identify points to be scanned during object detection.

0037It is appreciated that at the distance d from the image sensor <b>104</b>, the position of the object <b>112</b> (e.g., the pedestrian) will be between C<b>0</b> and C<b>1</b>. A line segment I<b>0</b>I<b>1</b> may represent center points for a search area for the object <b>112</b>, and may be defined as the intersection of the line segment L<b>0</b>L<b>1</b> and the line segment C<b>0</b>C<b>1</b>. It is noted that in the example of <figref idref="DRAWINGS">FIG. 2A</figref>, I<b>0</b>=C<b>0</b> and I<b>1</b>=C<b>1</b>, and therefore the line segment I<b>0</b>I<b>1</b> is identical to the line segment C<b>0</b>C<b>1</b>. However, when an estimated region for an object is not entirely located within a field of view of the image sensor, I<b>0</b> may not be equal to C<b>0</b> and/or I<b>1</b> may not be equal to C<b>1</b>. For example, in <figref idref="DRAWINGS">FIG. 2C</figref>, I<b>0</b>=L<b>0</b> and I<b>1</b>=C<b>1</b>.

0038The object detector <b>128</b> may perform a linear mapping to map I<b>0</b> and I<b>1</b> to image coordinates. In the example of <figref idref="DRAWINGS">FIG. 2A</figref>, L<b>0</b> corresponds to the left border of the image (having an x-coordinate of 0, i.e., x==0) and L<b>1</b> corresponds to a right side of the image (having an x-coordinate equal to a width of the image, i.e., x==image_width). Thus, a mapping function M may be defined as:

0039<maths id="MATH-US-00001" num="00001"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><mi>M</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>I</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><mn>0</mn></mrow><mo>)</mo></mrow></mrow><mo>=</mo><mrow><mfrac><mrow><mi>distance</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mi>L</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><mn>0</mn></mrow><mo>,</mo><mrow><mi>I</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><mn>0</mn></mrow></mrow><mo>)</mo></mrow></mrow><mrow><mi>distance</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mi>L</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><mn>0</mn></mrow><mo>,</mo><mrow><mi>L</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><mn>1</mn></mrow></mrow><mo>)</mo></mrow></mrow></mfrac><mo>×</mo><mi>image_width</mi></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mrow><mi>Equation</mi><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mn>1</mn></mrow><mo>)</mo></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mi>M</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>I</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><mn>1</mn></mrow><mo>)</mo></mrow></mrow><mo>=</mo><mrow><mfrac><mrow><mi>distance</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mi>L</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><mn>0</mn></mrow><mo>,</mo><mrow><mi>I</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><mn>1</mn></mrow></mrow><mo>)</mo></mrow></mrow><mrow><mi>distance</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mi>L</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><mn>0</mn></mrow><mo>,</mo><mrow><mi>L</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><mn>1</mn></mrow></mrow><mo>)</mo></mrow></mrow></mfrac><mo>×</mo><mi>image_width</mi></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mrow><mi>Equation</mi><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mn>2</mn></mrow><mo>)</mo></mrow></mtd></mtr></mtable></math></maths>

0040The search area for the object may be restricted to locations (e.g., centers of search windows) in the image having x-coordinates between x==M(I<b>0</b>) and x==M(I<b>1</b>). To illustrate, assume that a horizontal field of view of the image sensor <b>104</b> is 54°, as shown in <figref idref="DRAWINGS">FIG. 2A</figref>, and that a vertical field of view of the image sensor <b>104</b> is 34°, as shown in <figref idref="DRAWINGS">FIG. 2B</figref>. It is to be understood that the horizontal and vertical fields of view may be larger or smaller in alternative embodiments. It is also assumed that the line segment L<b>0</b>L<b>1</b> corresponds to 1080p resolution (1920×1080 pixels), and the height of an average pedestrian is assumed to be six feet (about 2 meters), which corresponds to a search area that is 80 pixels in height. It is further assumed that the distance from the image sensor <b>104</b> being searched is p feet and an upper half of the image corresponds to q feet at that distance, as shown in <figref idref="DRAWINGS">FIG. 2B</figref>. In view of the foregoing assumptions the following can be established:

0041<maths id="MATH-US-00002" num="00002"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><mi>tan</mi><mo>⁡</mo><mrow><mo>(</mo><mfrac><mrow><mn>34</mn><mo>⁢</mo><mi>°</mi></mrow><mn>2</mn></mfrac><mo>)</mo></mrow></mrow><mo>=</mo><mfrac><mi>q</mi><mi>p</mi></mfrac></mrow></mtd><mtd><mrow><mo>(</mo><mrow><mi>Equation</mi><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mn>3</mn></mrow><mo>)</mo></mrow></mtd></mtr><mtr><mtd><mrow><mfrac><mi>q</mi><mrow><mn>1080</mn><mo>⁢</mo><mstyle><mtext>/</mtext></mstyle><mo>⁢</mo><mn>2</mn></mrow></mfrac><mo>=</mo><mfrac><mn>6</mn><mn>80</mn></mfrac></mrow></mtd><mtd><mrow><mo>(</mo><mrow><mi>Equation</mi><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mn>4</mn></mrow><mo>)</mo></mrow></mtd></mtr></mtable></math></maths>

0042Solving Equations 3 and 4 for p:

0043<maths id="MATH-US-00003" num="00003"><math overflow="scroll"><mrow><mi>p</mi><mo>=</mo><mrow><mfrac><mi>q</mi><mrow><mi>tan</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mn>17</mn><mo>⁢</mo><mi>°</mi></mrow><mo>)</mo></mrow></mrow></mfrac><mo>=</mo><mrow><mfrac><mfrac><mrow><mn>6</mn><mo>×</mo><mn>540</mn></mrow><mn>80</mn></mfrac><mrow><mi>tan</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mn>17</mn><mo>⁢</mo><mi>°</mi></mrow><mo>)</mo></mrow></mrow></mfrac><mo>=</mo><mrow><mn>132</mn><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mi>feet</mi></mrow></mrow></mrow></mrow></math></maths>

0044If the image sensor <b>104</b> is assumed to be (0,0), the reported location of the pedestrian (i.e., center of the circle corresponding to the object <b>112</b>) is (x1,y1)==(−5,120), and the possible error of the reported location is within 30 feet (i.e., the radius of the estimated region <b>118</b> is 30):

0045<maths id="MATH-US-00004" num="00004"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><mrow><mrow><mo>(</mo><mrow><mrow><mo>(</mo><mrow><mi>x</mi><mo>-</mo><mrow><mo>(</mo><mrow><mo>-</mo><mn>5</mn></mrow><mo>)</mo></mrow></mrow><mo>)</mo></mrow><mo>×</mo><mrow><mo>(</mo><mrow><mi>x</mi><mo>-</mo><mrow><mo>(</mo><mrow><mo>-</mo><mn>5</mn></mrow><mo>)</mo></mrow></mrow><mo>)</mo></mrow></mrow><mo>)</mo></mrow><mo>+</mo><mrow><mo>(</mo><mrow><mrow><mo>(</mo><mrow><mi>y</mi><mo>-</mo><mn>120</mn></mrow><mo>)</mo></mrow><mo>×</mo><mrow><mo>(</mo><mrow><mi>y</mi><mo>-</mo><mn>120</mn></mrow><mo>)</mo></mrow></mrow><mo>)</mo></mrow></mrow><mo>=</mo><mrow><mn>30</mn><mo>×</mo><mn>30</mn></mrow></mrow><mo>⁢</mo><mstyle><mtext></mtext></mstyle><mo>⁢</mo><mrow><mi>L</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><mn>0</mn><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mi>is</mi><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mi>at</mi><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mi>the</mi><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mrow><mi>point</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mrow><mo>-</mo><mrow><mi>tan</mi><mo>⁡</mo><mrow><mo>(</mo><mfrac><mrow><mn>54</mn><mo>⁢</mo><mi>°</mi></mrow><mn>2</mn></mfrac><mo>)</mo></mrow></mrow></mrow><mo>×</mo><mn>132</mn></mrow><mo>,</mo><mn>132</mn></mrow><mo>)</mo></mrow></mrow></mrow><mo>⁢</mo><mstyle><mtext></mtext></mstyle><mo>⁢</mo><mrow><mi>L</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><mn>1</mn><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mi>is</mi><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mi>at</mi><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mi>the</mi><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mrow><mi>point</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mrow><mi>tan</mi><mo>⁡</mo><mrow><mo>(</mo><mfrac><mrow><mn>54</mn><mo>⁢</mo><mi>°</mi></mrow><mn>2</mn></mfrac><mo>)</mo></mrow></mrow><mo>×</mo><mn>132</mn></mrow><mo>,</mo><mn>132</mn></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mrow><mi>Equation</mi><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mn>5</mn></mrow><mo>)</mo></mrow></mtd></mtr></mtable></math></maths>

0046Because L<b>0</b>, L<b>1</b>, C<b>0</b>, and C<b>1</b> are collinear, all four points have the same y-coordinate, i.e., <b>132</b> in the current example. Substituting y==132 into Equation 5 results in the solutions (x+5)=27.5 and (x+5)=−27.5. Thus, C<b>0</b> is located at the point (−32.5, 132) and C<b>1</b> is located at the point (22.5, 132). Using the previously defined mapping M:

0047<maths id="MATH-US-00005" num="00005"><math overflow="scroll"><mrow><mrow><mi>M</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>C</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><msub><mn>0</mn><mi>x</mi></msub></mrow><mo>)</mo></mrow></mrow><mo>=</mo><mrow><mrow><mfrac><mrow><mrow><mo>-</mo><mn>32.5</mn></mrow><mo>+</mo><mrow><mrow><mi>tan</mi><mo>⁡</mo><mrow><mo>(</mo><mn>27</mn><mo>)</mo></mrow></mrow><mo>×</mo><mn>132</mn></mrow></mrow><mrow><mn>2</mn><mo>×</mo><mrow><mi>tan</mi><mo>⁡</mo><mrow><mo>(</mo><mn>27</mn><mo>)</mo></mrow></mrow><mo>×</mo><mn>132</mn></mrow></mfrac><mo>×</mo><mn>1920</mn></mrow><mo>=</mo><mn>496</mn></mrow></mrow></math></maths><maths id="MATH-US-00005-2" num="00005.2"><math overflow="scroll"><mrow><mrow><mi>M</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>C</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><msub><mn>1</mn><mi>x</mi></msub></mrow><mo>)</mo></mrow></mrow><mo>=</mo><mrow><mrow><mfrac><mrow><mn>22.5</mn><mo>+</mo><mrow><mrow><mi>tan</mi><mo>⁡</mo><mrow><mo>(</mo><mn>27</mn><mo>)</mo></mrow></mrow><mo>×</mo><mn>132</mn></mrow></mrow><mrow><mn>2</mn><mo>×</mo><mrow><mi>tan</mi><mo>⁡</mo><mrow><mo>(</mo><mn>27</mn><mo>)</mo></mrow></mrow><mo>×</mo><mn>132</mn></mrow></mfrac><mo>×</mo><mn>1920</mn></mrow><mo>=</mo><mn>1281</mn></mrow></mrow></math></maths>

0048Thus, for the object <b>112</b>, the search area can bound to the left by x==496 and to the right by x==1281, which provides approximately a 60% savings compared to searching from x==0 to x==1920. Moreover, it is to be understood that although the foregoing example illustrates calculations and savings in the horizontal direction, computational savings may also be achieved by similarly restricting the search area in the vertical direction based on the location data <b>116</b>.

0049<figref idref="DRAWINGS">FIGS. 2C and 2D</figref> illustrate additional examples of operation. In the example of <figref idref="DRAWINGS">FIG. 2C</figref>, the estimated region of the object partially intersects the field of view of the image sensor. In the example of <figref idref="DRAWINGS">FIG. 2D</figref>, two object detection operations are performed: a first object detection operation is performed on a first scale space representation (corresponding to line segment <b>202</b>) to detect the object on the right, and a second object detection operation is performed on a second scale space representation (corresponding to line segment <b>204</b>) to detect the object on the left.

0050<figref idref="DRAWINGS">FIGS. 1-2</figref> thus illustrate embodiments of a system that is configured to refine object detection operations, which are to be performed on image data or scale space representation(s) generated therefrom, based on location data associated with an object to be detected. For example, the location data may be used to narrow a search area for the object to less than the entire scene, which may enable the object to be detected faster. To illustrate, a pedestrian may be detected quickly enough for a vehicle (or a driver of the vehicle) to react to the pedestrian, such as, by slowing the vehicle, redirecting the vehicle, and/or speeding up the vehicle while redirecting the vehicle away from the pedestrian if, for example, applying brakes in icy weather would result in the vehicle skidding into the pedestrian. As another illustrative example, the object detection may be performed quickly enough for an ADAS of the vehicle to change a speed of the vehicle and/or redirect the vehicle to avoid colliding with a moving or stationary object.

0051<figref idref="DRAWINGS">FIG. 3</figref> illustrates an alternative embodiment of the vehicle <b>102</b>. <figref idref="DRAWINGS">FIG. 3</figref> differs from <figref idref="DRAWINGS">FIG. 1</figref> in that a location sensor that generates the location data <b>116</b> is external to the vehicle <b>102</b>. For example, in <figref idref="DRAWINGS">FIG. 3</figref>, the object <b>112</b> includes or is associated with a location sensor <b>306</b>. In some implementations, the location sensor <b>306</b> may be worn by a pedestrian, attached or coupled to another vehicle, or integrated within another vehicle. As discussed above, in some implementations the location sensor <b>306</b> may be separate from the object <b>112</b> and the vehicle <b>102</b>. For example, the location sensor <b>306</b> can be fixed to a roadside unit (RSU), fixed to a street sign, or fixed or associated with another vehicle that is separate from the vehicle <b>102</b>. In some examples, a stationary roadside sensor may communicate with a RSU and/or a vehicle. Thus, although <figref idref="DRAWINGS">FIG. 3</figref> illustrates direct communication between the location sensor <b>306</b> and the vehicle <b>102</b>, data from the location sensor <b>306</b> may alternatively be transmitted through one or more intermediary devices before reaching the vehicle <b>102</b>. To illustrate, data from a location sensor carried by a pedestrian may travel from the location sensor to an RSU to a first vehicle (e.g., that is closest to the pedestrian) to a second vehicle (e.g., that is trailing the first vehicle).

0052The vehicle <b>102</b> may include a receiver <b>340</b> configured to receive the location data <b>116</b> via a connection <b>350</b>, such as a wireless network connection. In illustrative examples, the location sensor <b>306</b> and the receiver <b>340</b> communicate via a cellular connection, a wide area network, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 connection, an ad-hoc network connection, a dedicated short-range communication (DSRC) network connection, or another type of connection.

0053In an illustrative example, the location data <b>116</b> received from the location sensor <b>306</b> includes location coordinates, for example, GPS coordinates. When the object <b>112</b> is another vehicle (e.g., the location sensor <b>306</b> is part of a device within or coupled to the other vehicle), the location data <b>116</b> may be received from the other vehicle (e.g., via a DRSC network) using vehicle-to-vehicle (V2V) communication. As another example, a roadside unit (RSU) may transmit traffic information to vehicles. To illustrate, the object <b>112</b> may include a vehicle involved in an accident within the field of travel of the vehicle <b>102</b>, and the location data <b>116</b> may include location coordinates of the vehicle involved in the accident. In this case, the receiver <b>340</b> may receive the location data <b>116</b> from the RSU using vehicle-to-infrastructure (V2I) communication. It should be understood that in other examples location data can be received from multiple sources. For example, location data associated with a common object could be received from more than one vehicle, RSU, or location sensor. When multiple sources of location data are used, the sources (and/or data therefrom) can be aggregated, prioritized, or used to refine one another. For example, the estimated region <b>118</b> of <figref idref="DRAWINGS">FIGS. 1-2</figref> may encompass all of the location data provided by the multiple sources, may correspond to location data from a most reliable source, may be based on an intersection of the location data from each (or a subset) of the multiple sources, or may otherwise be determined based on the location data from some or all of the multiple sources.

0054It should be noted that although the foregoing embodiments describe refining object detection in image data based on location data, the present disclosure is not so limited. In accordance with the present disclosure, data from a first sensor may be used to refine operation of a second sensor. As an illustrative non-limiting example, a property of an image sensor (e.g., resolution, capture frequency, area of interest, field of view (in the case of a movable camera), etc.) may be adjusted based on location data. As another example, image data may be used to speed up location determination by a location sensor, such as by determining a “coarse” location of the object based on the image data. Thus, the present disclosure enables different types of sensors to communicate with each other to refine their respective operations.

0055In a particular embodiment, the processor <b>108</b> may perform an object detection operation using location data from a three-dimensional (3D) map application <b>130</b>. To illustrate, the 3D map application <b>130</b> may indicate or provide a portion or area of a 3D map that can be accessed by the object detector <b>128</b> to narrow a search area based on a “known” object class (e.g., pedestrian, vehicle, traffic sign, etc.) being searched. Although <figref idref="DRAWINGS">FIG. 1</figref> illustrates the location data <b>116</b> as being determined by the location sensor <b>106</b>, in alternative embodiments the location data <b>116</b> may be determined based on the 3D map application <b>130</b> instead of or in addition to the location sensor <b>106</b> (or an external location sensor, such as the location sensor <b>306</b> of <figref idref="DRAWINGS">FIG. 3</figref>). The object detection techniques described herein may thus be used in conjunction with embodiments where 3D map data is used during object detection, a location sensor is used during object detection, or both 3D map data and a location sensor are used during object detection.

0056<figref idref="DRAWINGS">FIG. 4</figref> illustrates an example of a system in which the object detector <b>128</b> receives location data <b>116</b> from the 3D map application <b>130</b> and does not receive any additional location data from a location sensor. The location data <b>116</b> provided by the 3D map application <b>130</b> may be indicative of where an object class may be present within the image data <b>114</b>. For example, the system shown in <figref idref="DRAWINGS">FIG. 4</figref> may enable a vehicle to perform detection for pedestrians, tree limbs, other vehicles, etc., based on location data <b>116</b> received from the 3D map application <b>130</b> and indicative of where these objects may be present in the image data <b>114</b>.

0057For example, if the object detector <b>128</b> is configured to detect a pedestrian within a scale space representation of a scene, the 3D map application <b>130</b> may indicate one or more areas of the scene where a pedestrian could possibly be present (e.g., from 0 meters to 3 meters above a road or ground surface), and the object detector <b>128</b> may initiate a search of the indicated area(s) without receiving data from a separate location sensor. In another example, the 3D map application <b>130</b> may indicate one or more areas of the scene that correspond to a road surface, such that the object detector <b>128</b> can efficiently detect objects that are on the road. In some implementations, the object class may be determined by the processor <b>108</b> based on the image data <b>114</b> and/or the location data <b>116</b>. Alternatively, the object class may be indicated by the object <b>112</b> (e.g., transmitted by the object <b>112</b> to a receiver as described with reference to <figref idref="DRAWINGS">FIG. 3</figref>).

0058During operation, a 3D map may be dynamically generated at the vehicle <b>102</b>, downloaded or pre-downloaded to the vehicle <b>102</b>, etc. After determining a location of the road surface and receiving the location data <b>116</b>, the object detector <b>128</b> may focus the search area to the road surface based on the 3D map (e.g., non-road surface areas may be excluded from the image search area). As an illustrative example, if a portion of the line segment L<b>0</b>L<b>1</b> of <figref idref="DRAWINGS">FIG. 1</figref> (or the line segment C<b>0</b>C<b>1</b> of <figref idref="DRAWINGS">FIG. 1</figref>) corresponds to a non-road surface area, then the object detector <b>128</b> may exclude the portion from the search area. Examples of non-road surface areas include regions above or below the road surface and regions left or right of the road surface. It should be understood, however, that when the image sensor <b>104</b> is in motion as part of a vehicle or other device, the area(s) of interest may vary from frame to frame based on the location of a vehicle or device relative to the 3D map. For example, when topographical changes occur (e.g., a vehicle approaches an incline or decline in the road), the search area of interest may change. The location data <b>116</b> provided by the 3D map application <b>130</b> may be dynamic and may change depending on the known location of the vehicle or device.

0059In some implementations, the object detector <b>128</b> is configured to generate a search area mask based on the 3D map indicated by the 3D map application <b>130</b>. A mask image may be generated either offline or online (based on, for example, a tradeoff between storage and computational burden) using the 3D map. An example process is further described with reference to <figref idref="DRAWINGS">FIG. 6</figref>.

0060<figref idref="DRAWINGS">FIG. 5A</figref> depicts two scenarios in which a pedestrian is within a field of view of a vehicle, where the pedestrian is located on terrain represented by 3D map data available to the vehicle. In the first scenario, at <b>510</b>, the pedestrian is closer to the vehicle and, as a result, the vehicle uses a smaller scale space representation to perform object detection. In the second scenario, at <b>520</b>, the pedestrian is further from the vehicle and, as a result, the vehicle uses a larger scale space representation to perform object detection. When multiple scale space representations overlap a possible location of an object some scale space representations may be processed while other scale space representations are excluded from further processing. As an illustrative non-limiting example, a smaller (e.g., lower resolution) scale space representation may be preferable to detect a pedestrian that is close to a vehicle, because the height and/or width of the pedestrian in a larger scale space representation may be too large to detect using a pedestrian object class or model. Prior to performing the object detection, the scale space representation being searched may be masked based on the location data provided by a 3D map application. An illustrative example of a mask is shown in <figref idref="DRAWINGS">FIG. 5B</figref>, at <b>530</b>. The mask may be generated by marking different regions of the scale space representation with zeros or non-zeros (e.g., “1” or a non-zero fractional value). Areas marked with non-zeros may be searched and areas marked with zeros may be ignored during object detection. When non-zero fractional values are used, a higher non-zero fraction value may represent a greater probability of locating the object in the corresponding area. For example, as shown in <figref idref="DRAWINGS">FIG. 5B</figref>, the area marked with “0.5” may be searched prior to searching the area marked “0.3.”

0061In a particular embodiment, the probabilities for the areas of the mask (e.g., 0.5 and 0.3 in the mask of <figref idref="DRAWINGS">FIG. 5B</figref>) may be generated based on a set of training samples. The set of training samples may contain 3D position(s) of place(s) where a pedestrian can appear. The set of training samples may also include positions that have been marked as “dangerous” by an expert. For example, such “dangerous” positions may be positions in which pedestrians have a higher likelihood of being obscured, being difficult for a driver to notice, being involved in an accident, etc. In some embodiments, the 3D positions of the set of training samples may be associated with probabilities. While generating the mask image, the 3D positions may be projected onto an image plane, and different portions of the mask may be assigned different probabilities based on the 3D positions corresponding to underlying pixels of the image plane.

0062In a particular embodiment, a probabilistic model and mask can be used with respect to sensor data instead of or in addition to 3D map data. For example, the estimated region <b>118</b> of <figref idref="DRAWINGS">FIGS. 1-2</figref> may correspond to a mask, where different sub-regions have different probability values between zero and one. The probability values may be based on location data from a location sensor (e.g., the location sensor <b>106</b>), location data from a 3D map application (e.g., the 3D map application <b>130</b>), 3D map data, a sensor or application margin of error, sensor calibration, and/or sensor sensitivity, as illustrative non-limiting examples.

0063Thus, the processor <b>108</b> may determine a search area or areas based on 3D map data and the object class of the object. The object detector may perform object detection in the search area, as described with reference to <figref idref="DRAWINGS">FIG. 1</figref>. It is noted that when using 3D map data to perform object detection, projection may be performed in view of possible inaccuracy of localization result(s). That is, a range of camera locations may be enumerated due to localization inaccuracy. It should also be noted that in particular embodiments, masks/search areas may be constructed offline and used (e.g., to skip searching of masked off areas) in a real-time (or near-real-time) stream received from a camera (e.g., the image sensor <b>104</b>). Moreover, instead of dynamically constructing a 3D map, 3D conditions may be approximated with a plane/surface estimator, which may provide a rough real-time (or near-real-time) plane. For example, in an area where most roads are on planar surfaces, the 3D map may be estimated by using a plane with some slope variation to estimate the projected masks. In a particular embodiment, the foregoing method can also be used to find an object of interest in a particular area. For example, the object detector <b>128</b> may find pedestrians (or other objects) in an area that include or are close to a “current” driving lane of the vehicle <b>102</b> but may ignore pedestrians in sidewalk areas. As another example, the object detector <b>128</b> may be part of an object tracking framework in which tracking areas are limited based on masks.

0064Although <figref idref="DRAWINGS">FIGS. 1-3</figref> describe performing object detection using location data from a location sensor and <figref idref="DRAWINGS">FIG. 4</figref> illustrates performing object detection using location data provided by a 3D map application instead of a location sensor, it should be noted that the aforementioned embodiments are not mutually exclusive. In some examples, 3D map data can be used along with data from a location sensor, and may enable faster object detection and further reduction of scan area(s).

0065Referring to <figref idref="DRAWINGS">FIG. 6</figref>, a particular embodiment of a method of identifying area(s) of interest based on 3D map data is shown and generally designated <b>600</b>. The method <b>600</b> includes receiving 3D map data, at block <b>602</b>. For example, the 3D map application <b>130</b> may receive 3D map data, such as from an external data source, a local storage device, etc. The method <b>600</b> also includes determining whether a location of the 3D map is likely to include an object of interest, at block <b>604</b>. If so, the method <b>600</b> includes identifying the location as location data to be provided to an object detector, at block <b>606</b>. If the location is not likely to include the object of interest, or after identifying the location data, the method <b>600</b> includes determining whether additional locations are to be examined, at block <b>608</b>. If so, the method <b>600</b> returns to block <b>604</b>. To illustrate, referring to <figref idref="DRAWINGS">FIGS. 4-5</figref>, the location data <b>116</b> may indicate one or more areas of the terrain that are likely to include a pedestrian, such as the pedestrian that is close to the vehicle, at <b>510</b>, or the pedestrian that is far from the vehicle, at <b>520</b>.

0066Continuing to block <b>610</b>, the method <b>600</b> includes projecting the object of interest in the location data with a size that is based on an object class of the object of interest. In the example of FIG. SA, a pedestrian may be projected in the identified location data corresponding to the roadway being traveled by the vehicle. The method <b>600</b> further includes determining a scale space representation that overlaps the projected object of interest, at block <b>612</b>, and marking the overlapping area(s) of the scale space representation with non-zero value(s), at block <b>614</b>. To illustrate, for the pedestrian closer to the vehicle in FIG. SA at <b>510</b>, areas in the small scale space representation may be masked with non-zero values. For the pedestrian farther from the vehicle in FIG. SA at <b>520</b>, areas in the large scale space representation may be masked with non-zero values. In an illustrative example, the scale space representation may be masked as shown in <figref idref="DRAWINGS">FIG. 5B</figref> at <b>530</b>.

0067In an example, 3D map data may be used to alter an orientation or field of view of an image sensor, and/or the 3D map data may be used to select a portion of image data that is generated by multiple image sensors. As another example, when a vehicle turns left or right, the image sensor of the vehicle may automatically be actuated left or right so that the image sensor continues to capture image data that is “ahead of” the vehicle. As another example, an image sensor of a vehicle may be actuated automatically to “look” left and right to identify pedestrians, other approaching vehicles, road hazards, etc. To illustrate, a 360° stitched view of a vehicle's surroundings may be generated and a driver of the vehicle may be notified ahead of time if there is a pedestrian crossing a street that the driver may later turn on. In a particular embodiment, when the driver is using turn-by-turn navigation directions (e.g., provided by a navigation application of the vehicle or a mobile device within the vehicle), the navigation directions may be used to anticipate turns that will be made by the vehicle, lane changes that will be made, etc., such as to prioritize search areas, determine probability values for search masks, etc.

0068The method <b>600</b> includes searching area(s) of the scale space representation marked with non-zero values in decreasing order to identify the object of interest, at block <b>616</b>. For example, referring to the mask shown in <figref idref="DRAWINGS">FIG. 5B</figref>, the areas marked “0.5” may be searched prior to searching the areas marked “0.3.” The method <b>600</b> ends at block <b>618</b>.

0069The method <b>600</b> of <figref idref="DRAWINGS">FIG. 6</figref> thus illustrates an example of performing object detection on one or more scale space representations based on 3D map data. The 3D map data may be used in lieu of, or in addition to, location data provided by a location sensor.

0070Referring to <figref idref="DRAWINGS">FIG. 7</figref>, an illustrative method of object detection is shown and generally designated <b>700</b>. The method <b>700</b> includes receiving, from an image sensor, image data of a scene that includes an object, at block <b>702</b>. The image data is associated with multiple scale space representations of the scene. For example, the processor <b>108</b> may receive the image data <b>114</b> from the image sensor <b>104</b>, where the image data <b>114</b> is associated with (e.g., can be used to generate) the scale space representations <b>120</b>-<b>124</b>.

0071The method <b>700</b> also includes receiving location data associated with the object or indicative of an area where the object may be present, at block <b>704</b>. For example, the processor <b>108</b> may receive the location data <b>116</b> from the on-board location sensor <b>106</b> in the vehicle <b>102</b>, as described with reference to <figref idref="DRAWINGS">FIG. 1</figref>. As another example, the processor <b>108</b> may receive the location data <b>116</b> from the location sensor <b>306</b>, which is external to the vehicle <b>102</b>, via the receiver <b>340</b>, as described with reference to <figref idref="DRAWINGS">FIG. 3</figref>. In another example, the object detector <b>128</b> may receive location data <b>116</b> from a 3D map application <b>130</b> with the location data being indicative of an area where an object may be present (e.g., on a road, in a lane, etc.), as described with reference to <figref idref="DRAWINGS">FIG. 4</figref>. In yet another example, location data may be received from multiple sources, such as from multiple location sensors, a location sensor and a 3D map application, etc.

0072The method <b>700</b> further includes determining whether a first scale space representation overlaps the location data, at block <b>706</b>. When the first scale space representation overlaps the location data, a search area in the first scale space representation is identified, at block <b>710</b>. Advancing to block <b>712</b>, object detection is performed in the search area. To illustrate, in the example of <figref idref="DRAWINGS">FIG. 1</figref>, the object detector <b>128</b> may determine that the scale space representation <b>122</b> overlaps the estimated region <b>118</b> corresponding to the location data <b>116</b>. In response, the object detector <b>128</b> may determine a search area of the scale space representation <b>122</b>, as described with reference to Equations 1-5, and may perform object detection in the search area.

0073Alternatively, when the first scale space representation does not overlap the location data, the method <b>700</b> includes avoiding the performance of object detection on the scale space representation, at block <b>708</b>. To illustrate, in the example of <figref idref="DRAWINGS">FIG. 1</figref>, the object detector may avoid performing object detection on the scale space representations <b>120</b> and <b>124</b> due to the lack of overlap between the estimated region <b>118</b> and the scale space representations <b>120</b> and <b>124</b>. Continuing to block <b>714</b>, the method <b>700</b> includes determining whether additional scale space representations are to be examined (e.g., when multiple scale space representations overlap the location data). If so, the method <b>700</b> returns to block <b>706</b>. If not, or after the object detection is performed at block <b>712</b>, the method <b>700</b> includes determining whether additional objects are to be detected, at block <b>716</b>. If so, the method <b>700</b> returns to block <b>704</b> to process location data for the additional objects. If not, the method <b>700</b> ends, at block <b>718</b>.

0074When multiple scale space representations are found to overlap location data, object detection may be performed on one, some, or all over the overlapping scale space representations (e.g., until the object of interest is detected). For example, as described with reference to <figref idref="DRAWINGS">FIG. 5A</figref>, a lowest resolution scale space representation may be examined first to detect a pedestrian that is close to a vehicle.

0075One or more operations of the methods <b>600</b> and/or <b>700</b> may be initiated, controlled, or performed by a hardware device, such as a processing unit. For example, depending on the particular implementation, the processing unit may include a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a processing unit such as a central processing unit (CPU), a digital signal processor (DSP), a controller, another hardware device, a firmware device, or a combination thereof.

0076Referring to <figref idref="DRAWINGS">FIG. 8</figref>, a block diagram of a particular illustrative embodiment of an electronic device is depicted and generally designated <b>800</b>. In a particular embodiment, the electronic device <b>800</b>, or components thereof, may be worn by or carried by a pedestrian (e.g., as part of a mobile phone, a tablet computer, a smartwatch, etc.). In another particular embodiment, the electronic device <b>800</b>, or components thereof, may be included in or may be attached/coupled to a vehicle. In another particular embodiment, the electronic device <b>800</b>, or components thereof, may be included in or may be attached/coupled to a roadside unit (RSU), a street sign, a traffic light, or another roadside object or device. In further embodiments, the electronic device <b>800</b> may correspond to a computer (e.g., a laptop computer, a tablet computer, or a desktop computer), a set top box, an entertainment unit, a navigation device, a personal digital assistant (PDA), a television, a tuner, a radio (e.g., a satellite radio), a music player (e.g., a digital music player and/or a portable music player), a video player (e.g., a digital video player, such as a digital video disc (DVD) player and/or a portable digital video player), another electronic device, or a combination thereof.

0077The electronic device <b>800</b> includes a memory <b>832</b> and a processor <b>810</b>, such as a digital signal processor (DSP), a central processing unit (CPU), and/or a graphics processing unit (GPU), as illustrative examples. The processor <b>810</b> may execute instructions <b>868</b>. In an illustrative example, the instructions <b>868</b> are executable by the processor <b>810</b> to perform one or more functions or methods described herein, including but not limited to the method <b>600</b> of <figref idref="DRAWINGS">FIG. 6</figref> and/or the method <b>700</b> of <figref idref="DRAWINGS">FIG. 7</figref>. The processor <b>810</b> may also include hardware corresponding to, and/or execute software instructions corresponding to, an object detector <b>869</b> (e.g., the object detector <b>128</b> of <figref idref="DRAWINGS">FIGS. 1, 3, and 4</figref>) and a 3D map application <b>870</b> (e.g., the 3D map application <b>130</b> of <figref idref="DRAWINGS">FIGS. 1 and 4</figref>).

0078<figref idref="DRAWINGS">FIG. 8</figref> also shows a display controller <b>826</b> that is coupled to the processor <b>810</b> and to a display <b>828</b>. A coder/decoder (CODEC) <b>834</b>, such as an analog audio processing front-end, can also be coupled to the processor <b>810</b>. A speaker <b>836</b> and a microphone <b>838</b> can be coupled to the CODEC <b>834</b>. <figref idref="DRAWINGS">FIG. 8</figref> also indicates that a wireless interface <b>840</b>, such as a wireless controller and/or a transceiver, can be coupled to the processor <b>810</b> and to an antenna <b>842</b>. In particular embodiments, the electronic device <b>800</b> may include multiple wireless interfaces and antennas. Each wireless interface and/or antenna may correspond to a different communication technology or network (e.g., cellular, IEEE 802.11, DSRC, etc.).

0079In a particular embodiment, the processor <b>810</b> is further coupled to an image sensor <b>880</b> (e.g., the image sensor <b>104</b> of <figref idref="DRAWINGS">FIGS. 1, 3, and 4</figref>). The processor <b>810</b> may also be coupled to a location sensor <b>890</b> (e.g., the location sensor <b>106</b> of <figref idref="DRAWINGS">FIG. 1</figref>). Alternatively, the processor <b>810</b> may receive location data from an external location sensor via a receiver, such as the wireless interface <b>840</b> and/or the antenna <b>842</b>, and/or the processor <b>810</b> may receive location data from the 3D map application <b>870</b>.

0080In a particular embodiment, the processor <b>810</b>, the display controller <b>826</b>, the CODEC <b>834</b>, the wireless interface <b>840</b>, the image sensor <b>880</b>, and the location sensor <b>890</b> (when present) are included in a system-in-package or system-on-chip device <b>822</b>. Further, an input device <b>830</b> and a power supply <b>844</b> may be coupled to the system-on-chip device <b>822</b>. Moreover, in a particular embodiment, as illustrated in <figref idref="DRAWINGS">FIG. 8</figref>, the display <b>828</b>, the input device <b>830</b>, the speaker <b>836</b>, the microphone <b>838</b>, the antenna <b>842</b>, and the power supply <b>844</b> are external to the system-on-chip device <b>822</b>. However, each of the display <b>828</b>, the input device <b>830</b>, the speaker <b>836</b>, the microphone <b>838</b>, the antenna <b>842</b>, and the power supply <b>844</b> can be coupled to a component of the system-on-chip device <b>822</b>, such as to an interface or to a controller.

0081In connection with the described embodiments, an apparatus includes means for receiving image data of a scene viewed from the apparatus and including an object, the image data associated with multiple scale space representations of the scene. For example, the means for receiving may include the image sensor <b>104</b>, the image sensor <b>880</b>, a processor or controller coupled to an image sensor (e.g., the processor <b>108</b> or the processor <b>810</b>), another device configured to receive image data, or any combination thereof. The apparatus also includes means for processing. The means for processing is configured to identify a search area of a first scale space representation of the multiple scale space representations based on an overlap between the first scale space representation and location data associated with the object or indicative of an area where the object may be present. The means for processing is also configured to perform object detection in the search area of the first scale space representation. For example, the means for processing may include the processor <b>108</b>, the object detector <b>128</b>, the processor <b>810</b>, the object detector <b>869</b>, the 3D map application <b>130</b>, the 3D map application <b>870</b>, another device configured to process data, or any combination thereof. In a particular embodiment, the apparatus includes means for generating the location data. For example, the means for generating the location data may include the location sensor <b>106</b>, the location sensor <b>306</b>, the location sensor <b>890</b>, the 3D map application <b>130</b>, another device configured to generate location data, or any combination thereof. In a particular embodiment, the apparatus includes means for receiving the location data from an external location sensor. For example, the means for receiving may include the receiver <b>340</b>, the wireless interface <b>840</b>, the antenna <b>842</b>, another device configured to receive data from an external sensor, or any combination thereof.

0082The foregoing disclosed devices and functionalities may be designed and represented using computer files (e.g. RTL, GDSII, GERBER, etc.). The computer files may be stored on computer-readable media. Some or all such files may be provided to fabrication handlers who fabricate devices based on such files. Resulting products include wafers that are then cut into die and packaged into integrated circuits (or “chips”). The chips are then employed in electronic devices, such as a component of the vehicle <b>102</b>, the electronic device <b>800</b>, etc.

0083Although one or more of <figref idref="DRAWINGS">FIGS. 1-8</figref> may illustrate systems, apparatuses, and/or methods according to the teachings of the disclosure, the disclosure is not limited to these illustrated systems, apparatuses, and/or methods. One or more functions or components of any of <figref idref="DRAWINGS">FIGS. 1-8</figref> as illustrated or described herein may be combined with one or more other portions of another of <figref idref="DRAWINGS">FIGS. 1-8</figref>. Accordingly, no single example should be construed as limiting and embodiments of the disclosure may be suitably combined without departing from the teachings of the disclosure.

0084Those of skill would further appreciate that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or combinations of both. Various illustrative components, blocks, configurations, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or processor executable instructions depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

0085The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transitory computer-readable or processor-readable storage medium or storage device known in the art. An exemplary storage medium or device is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium or device may be integral to the processor. The processor and the storage medium or device may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. In the alternative, the processor and the storage medium or device may reside as discrete components in a computing device or user terminal. A computer-readable or processor-readable storage medium or storage device is not a signal.

0086The previous description of the disclosed embodiments is provided to enable a person skilled in the art to make or use the disclosed embodiments. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other embodiments without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope possible consistent with the principles and novel features as defined by the following claims.